This commit upgrades all dependencies in `aio/aio-builds-setup/scripts-js/`
to latest versions and also includes all necessary code changes to
ensure the tests are passing with the new dependency versions.
In particular:
- We ensure `nock`'s `Scope#done()` is not called before receiving a
response to account for a breaking change introduced in
nock/nock#1960.
- The use of `nock`'s `Scope#log()` method was removed, because the
method is no longer available since nock/nock#1966. See
https://github.com/nock/nock#debugging for more info on debugging
failed matches.
See also
e23ba31b13/migration_guides/migrating_to_13.md
for more info on migrating from `nock` v12 to v13.
PR Close#38736
Previously, some preview server tests were only running for public
builds. In the past, these tests were run for both public and non-public
builds. The non-public builds tests were disabled in #23576, probably
during debugging some failure.
This commit fixes it by ensuring the tests are run for both public and
non-public builds.
PR Close#36837
The test was introduced in #23576, but the behavior the test was
verifying does not match the actual behavior of
`BuildCleaner#getOpenPrNumbers()`.
The reason that the test did not fail is that the verification happened
asynchronously, but the test completed synchronously (by accident).
PR Close#36837
Previously, when the preview server `build-cleanup` script failed, the
error was logged but not reflected to the commands exit code. This seems
to have been accidentally broken in #23576.
This commit fixes it by ensuring the error is re-thrown from the
`BuildCleaner#cleanUp()` method to allow the process to exit with an
error exit code.
PR Close#36837
Previously, the `dev` npm script in `aio/aio-builds-setup/scripts-js/`
(the PR preview server implementation) would run both linting and unit
tests. This was slow and delayed the feedback loop on each change.
Since the `dev` script should be run during development and give
feedback as fast as possible, this commit removes the linting from the
`dev` script and only keeps the unit tests. Linting is still run in the
`test` npm script (which is more comprehensive). Also, in most cases the
developer's IDE will show linting errors in real time in the editor.
PR Close#36837
This commit upgrades all dependencies in `scripts-js/` to latest
versions and also includes all necessary code changes to ensure the
tests are passing with the new dependency versions.
PR Close#36837
Previously, the preview server docker image was based on Debian 8
(jessie). Recently, `jessie-updates` and `jessie-backborts` were removed
from the Debian mirrors ([more info][1]), thus breaking new builds of
the image.
Instead of updating `/etc/apt/sources.list` to remove the obsolete
sources, this commit upgrades to Debian 9 (stretch).
(The GCE VM running the preview server docker container was also
upgraded from Debian 8 to 9 this morning.)
---
Other changes:
- Removed dependency on `chkconfig`, which is not supported on Debian 9.
- Installing `nginx` from the regular repositories (instead of
`*-backports).
- Upgraded to `pm2` v3, which can handle hooking itself up to system
startup better (without `chkconfig` - see above).
- Updated tests to reflect the fact that `nginx` has dropped the reason
phrase in response status lines for HTTP/2 (in compliance with
[the spec][2]). (HTTP/1.1: `HTTP/1.1 200 OK` | HTTP/2: `HTTP/2 200`)
[1]: https://www.lucas-nussbaum.net/blog/?p=947
[2]: https://http2.github.io/http2-spec/#rfc.section.8.1.2.4
PR Close#29976
This can help with debugging issues, e.g. with the communication between
the preview server and CI, as it gives a better idea of exactly when was
the preview made available and how long it took.
PR Close#27436
Earlier versions may transitively depend on a malicious version of
`flatmap-stream` (see dominictarr/event-stream#116).
The `aio-builds-setup/` had an older version of `event-stream` (3.3.4),
which did not depend on `flatmap-stream`, but upgraded it anyway.
PR Close#27274
As it turns out, in GitHub API paginated requests, page numbering is
1-based. (https://developer.github.com/v3/#pagination)
Starting at page 0 (which returns the first page), results in making the
same request twice and logging incorrect numbers (since the first 100
items are listed twice).
PR Close#25671
Includes the following fixes:
- Fix cron entry format for clean-up script.
Crontabs in `/etc` should not have a user field. No idea why it used
to work before, but it started giving errors recently:
`/bin/sh: root: not found`.
- Set required env variable in clean-up script. (Broken in cc6f36a9d.)
This was producing the following error:
`ERROR: Missing required environment variable 'AIO_CIRCLE_CI_TOKEN'!`
- Use the correct path for downloads to be removed. (Broken in cc6f36a9d.)
PR Close#25671
There several reasons why PRs cannot have (public) previews:
- The PR did not affect any relevant files (e.g. non-spec files in
`aio/` or `packages/`).
- The PR cannot be automatically verified as "trusted" (based on its
author or labels).
Note:
The endpoint does not check whether there currently is a (public)
preview for the specified PR; only whether there can be one.
PR Close#25671
According to the docs, the response of GitHub's [PR files API][1]
_"includes a maximum of 300 files"_. This means that if a PR contains
more files, it is possible that not all files are retrieved (which
could, for example, give a false negative for the "significant files
touched" check - not likely but possible).
This commit fixes it by using paginated requests to retrieve all changed
files.
[1]: https://developer.github.com/v3/pulls/#list-pull-requests-files
PR Close#25671
Some tests where comparing actual with expected paths, without taking
into account that paths will be different on Windows.
This commit uses `path.resolve()` to convert expected paths to their
OS-specific form.
PR Close#25671
Previously, due to multiple scripts re-building during `yarn dev`
initialization, there could be race conditions that led to errors.
This commit fixes it by ensuring `yarn build` is run once (before
the main `yarn dev` script).
PR Close#25671
Previously, Travis pushed the build artitfacts to the preview server.
This required us to use JWT to secure the POST request from Travis, to
ensure we couldn't receive malicious builds.
JWT has been deprecated and we are moving our builds to CircleCI.
This commit rewrites the TypeScript part of the preview server that
handles converting build artifact into hosted previews of the docs.
closes#19352
Previously, the upload server (for PR previews) was run as root and
"downleveled" to a non-privileged user from inside the node script.
Now, with the latest version of `pm2` (which is used to run the upload server
scripts), we can get rid of that workaround and set the desired UID directly
through `pm2`.
This commit adds an API endpoint for notifying the preview server about PR
updates (`/pr-updated`). According to the update, the preview server can take
several actions. Currently, it will only check and (if necessary) update the
PR's preview visibility (but more actions could be supported in the future).
The API can be used with an automatic trigger (e.g. a GitHub webhook) to
instantly update a PR's preview visibility when it changes.
Fixes#16526
Previously, `BuildCreator#changePrVisibility()` would throw an error if the PR's
visibility was already up-to-date or if the PR directory did not exist (e.g. was
removed). This method was only used from inside `BuildCreator#create()`, which
had already checked for the existence of the directories.
This commit renames `changePrVisibility()` to `updatePrVisibility()` and makes
it more "forgiving" (i.e. it will only throw if both public and non-public
directories exist). This allows it to be used on events that may or may not have
caused the PR's visibility to change (e.g. a GitHub webhook triggered whenever a
PR's labels change).
Previously, there was a distinction between GET requests to invalid URLs and all
other requests. This was mainly because the upload-server only accepts GET
requests, but that is not a hard limitation and may change in the future.
Thus, it makes sense to return a 404 response for requests to invalid URLs
regardless of the method used.
Use the 7 first characters of the 40-chars long SHAs for shorter/cleaner URLs.
The collision probability is extremely low (since all SHAs are further
"namespaced" under the corresponding PR). In case of a collision, the second PR
will not be deployed, in order to avoid overwriting the original build.
(This is a design decision to keep the implementation simple. It can be changed
later if necessary.)
The previous clean-up code for PR directories on the preview server assumed that
all directories were named after the PR number. With the changes introduced
in #17640 it is possible to have PR directories that do not follow that naming
convention (e.g. "non-public" directories).
This PR ensures that both public and non-public directories are removed when
cleaning up.
This commit introduces the ability to show previews for PRs by any author. It works as follows:
- The build artifacts of all PRs are uploaded to the preview server.
- Automatically verified PRs (i.e. from trusted authors or having a specific label) are deployed and
publicly accessible as usual.
- PRs that could not be automatically verified are stored for later use (after re-verification).
- A PR can be marked as "trusted" and make its preview publicly accessible by adding the GitHub
label specified in the `AIO_TRUSTED_PR_LABEL` env var of the preview server.
At the moment, there is no automatic mechanism for notifying the preview server about changes to the
PR's verification status. The PR's "visibility" will be checked and updated every time a new build
is uploaded.
George Kalpakas
Igor Minar
Pete Bacon Darwin