Commit Graph

135 Commits

Author SHA1 Message Date
George Kalpakas 2d065a32ec docs(docs-infra): improve docs on creating/updating the preview server Docker image (#37015)
This commit includes the following improvements:

- Document that the `create-image.sh` script (and by extention the
  `update-preview-server.sh` script) need to have access to a `yarn`
  executable.

- Add a note on cron jobs running in non-interactive, non-login shells
  (which affects their execution context and have different behavior vs
  running the same commands in an interactive, login shell).

- Change the Node.js and `yarn` installation instructions to ensure the
  `yarn` executable will be available on the `PATH` and not require an
  interactive, login shell (as happens, for example, when installing it
  via [nvm](https://github.com/nvm-sh/nvm)). This makes it easier to set
  up a cron job that runs the `update-preview-server.sh` script.

NOTE: The equivalent updates have been made on the GCE VM that hosts the
      PR preview server.

PR Close #37015
2020-05-18 10:24:20 -07:00
George Kalpakas fcb68cc139 fix(docs-infra): better detect failed attempts to update the preview server Docker container (#37015)
In order to avoid unnecessary operations, the `update-preview-server.sh`
script, that is used to update the PR preview server Docker container,
will only try to update the Docker container if either any files in the
`aio/aio-builds-setup/` directory have changed since the last update or
if a previous update failed. A failed previous update is detected by
checking whether the temporary `aio-builds:provisional` Docker image
still exists. This temporary image is created during the update
operation and is renamed to `aio-builds:latest` if the update goes well.

Previously, the update script was not able to detect a previous failed
attempt if the operation failed before creating the
`aio-builds:provisional` Docker image, such as if the `create-image.sh`
script failed. This could lead to a situation where the preview server
Docker container would not be updated after a failed attempt.

This commit improves the logic for detecting failed attempts to account
for this type of failures. It does this by not removing an older
`aio-builds:provisional` Docker image until a new one is successfully
created.

NOTE: While this is not full-proof, it is an improvement as it
      eliminates a certain kind of failures.

PR Close #37015
2020-05-18 10:24:20 -07:00
George Kalpakas 1b8752e595 refactor(docs-infra): take advantage of latest Jasmine features in preview server tests (#36837)
This commit updates the preview server tests to take advantage of
features supported in the latest version of Jasmine that were not
supported when the rests were first written.

Changes include:

- Use [async/await] in tests.
- Use the new [toBeInstanceOf()] and [toHaveBeenCalledBefore()] matchers.
- Use the new [toBeResolved()] and [toBeRejected()] async matchers (and
  their variants).
- Use the new [withArgs()] method of `Spy` to simplify "trained"
  responses.
- Use the new [resolveTo()]/[rejectWith()] methods of `SpyStrategy` (and
  their variants) to simplify promise-based spies.
- Implement custom async matchers (via [addAsyncMatchers()]) to simplify
  certain tests.

[addAsyncMatchers()]: https://jasmine.github.io/api/3.5/jasmine.html#.addAsyncMatchers
[async/await]: https://jasmine.github.io/tutorials/async
[rejectWith()]: https://jasmine.github.io/api/3.5/SpyStrategy.html#rejectWith
[resolveTo()]: https://jasmine.github.io/api/3.5/SpyStrategy.html#resolveTo
[toBeInstanceOf()]: https://jasmine.github.io/api/3.5/matchers.html#toBeInstanceOf
[toBeRejected()]: https://jasmine.github.io/api/3.5/async-matchers.html#toBeRejected
[toBeResolved()]: https://jasmine.github.io/api/3.5/async-matchers.html#toBeResolved
[toHaveBeenCalledBefore()]: https://jasmine.github.io/api/3.5/matchers.html#toHaveBeenCalledBefore
[withArgs()]: https://jasmine.github.io/api/3.5/Spy.html#withArgs

PR Close #36837
2020-05-05 17:46:46 -07:00
George Kalpakas b3cf5b246f test(docs-infra): enable accidentally disabled preview server tests (#36837)
Previously, some preview server tests were only running for public
builds. In the past, these tests were run for both public and non-public
builds. The non-public builds tests were disabled in #23576, probably
during debugging some failure.

This commit fixes it by ensuring the tests are run for both public and
non-public builds.

PR Close #36837
2020-05-05 17:46:46 -07:00
George Kalpakas 0a695e181e test(docs-infra): remove incorrect preview server test (#36837)
The test was introduced in #23576, but the behavior the test was
verifying does not match the actual behavior of
`BuildCleaner#getOpenPrNumbers()`.

The reason that the test did not fail is that the verification happened
asynchronously, but the test completed synchronously (by accident).

PR Close #36837
2020-05-05 17:46:46 -07:00
George Kalpakas a5d1d1161b fix(docs-infra): exit with an error when cleaning up the preview server fails (#36837)
Previously, when the preview server `build-cleanup` script failed, the
error was logged but not reflected to the commands exit code. This seems
to have been accidentally broken in #23576.

This commit fixes it by ensuring the error is re-thrown from the
`BuildCleaner#cleanUp()` method to allow the process to exit with an
error exit code.

PR Close #36837
2020-05-05 17:46:46 -07:00
George Kalpakas a25c3c5c3b build(docs-infra): remove linting from the preview server `dev` npm script (#36837)
Previously, the `dev` npm script in `aio/aio-builds-setup/scripts-js/`
(the PR preview server implementation) would run both linting and unit
tests. This was slow and delayed the feedback loop on each change.

Since the `dev` script should be run during development and give
feedback as fast as possible, this commit removes the linting from the
`dev` script and only keeps the unit tests. Linting is still run in the
`test` npm script (which is more comprehensive). Also, in most cases the
developer's IDE will show linting errors in real time in the editor.

PR Close #36837
2020-05-05 17:46:46 -07:00
George Kalpakas d1a8afbbd0 refactor(docs-infra): update argument order for `update-preview-server.sh` script (#36837)
Update the order in which the `update-preview-server.sh` script expects
its arguments (and the associated docs) to be consistent with the order
of arguments in other commands/docs (such as
`vm-setup--start-docker-container.md`).

PR Close #36837
2020-05-05 17:46:46 -07:00
George Kalpakas f4fcb0c54c docs(docs-infra): update preview server setup instructions (#36837)
I recently went through the process of setting up a preview server VM
again and updated the instructions and references based on the latest
docs for Debian, Docker, Google Compute Engine, etc.

PR Close #36837
2020-05-05 17:46:46 -07:00
George Kalpakas e92b81351b build(docs-infra): unpin certain dependencies of the preview server (#36837)
Previously, in order to remain as deterministic as possible, the
Dockerfile for the preview server Docker image had all dependencies
pinned to specific versions. It turns out that some packages (such as
`nginx`, `nodejs`, and `openssl` - potentially others too) make older
versions unavailable on the repositories once a newer version is
available.

See for example:
- https://github.com/nodesource/distributions/issues/33
- https://askubuntu.com/questions/715104/how-can-i-downgrade-openssl-via-apt-get

This commit, therefore, removes the exact versions for these packages.
The latest versions will be installed everytime the Docker image is
built (subject to Docker caching).

PR Close #36837
2020-05-05 17:46:45 -07:00
George Kalpakas 53805f07ba test(docs-infra): check TLS certificates as part of preview server's health check (#36837)
In order to ease local development, self-signed SSL/TLS certificates are
created when building the preview server Docker image. These
certificates are valid for 365 days. Thus, it is possible for an old
certificate to be re-used past its expiration date due to Docker's
caching intermediate layers.

Previously, this would lead to hard-to-debug failures in the
`aio-health-check` and `aio-verify-setup` checks. Even after finding out
that the failures were caused by an expired certificate, it was not
obvious why that would be the case.

This commit adds an additional check to the `aio-health-check` command
that checks the certificates' expiration dates. This helps surface such
errors. It also prints a more helpful message, prompting the user to
build the Docker image with the `--no-cache` option to fix the problem
with self-signed certificates.

PR Close #36837
2020-05-05 17:46:45 -07:00
George Kalpakas e73daa3736 build(docs-infra): upgrade preview server Docker image to Debian 10 (buster) (#36837)
Previously, the preview server Docker image was based on Debian 9
(stretch).

This commit upgrades the preview server Docker image to Debian 10
(buster) and also upgrades all dependencies to latest versions
(including upgrading Node.js from v10 to v12).

(The GCE VM running the preview server Docker container was also
upgraded from Debian 9 to 10 on 2020-04-27.)

---
Other changes:
- Pinned the installed version of `curl` to make the `aio-health-check`
  and `aio-verify-setup` checks (which use `curl`) more deterministic.
- Dropped the `*-backports` Debian repositories, since they are no
  longer needed. The `*-backports` repositories were introduced in
  593fe5ed25 to install `nginx` from, but
  became obsolete in 2f1a862b83, which
  switched to installing `nginx` from the regular repositories again.
- Added `vim` to the list of installed dependencies (for convenience
  during debugging).

PR Close #36837
2020-05-05 17:46:45 -07:00
George Kalpakas eef01160f4 build(docs-infra): upgrade all JS dependencies to latest versions (#36837)
This commit upgrades all dependencies in `scripts-js/` to latest
versions and also includes all necessary code changes to ensure the
tests are passing with the new dependency versions.

PR Close #36837
2020-05-05 17:46:45 -07:00
Igor Minar 86e1e6c082 feat: typescript 3.6 support (#32946)
BREAKING CHANGE: typescript 3.4 and 3.5 are no longer supported, please update to typescript 3.6

Fixes #32380

PR Close #32946
2019-10-18 13:15:16 -04:00
George Kalpakas f84c4b066a build(docs-infra): pin versions of packages installed in preview server docker image (#29976)
This minimises the risk of unexpected failures due to breaking changes,
when building a new image (e.g. as a result of an unrelated config
change in Dockerfile).

PR Close #29976
2019-04-23 08:33:27 -07:00
George Kalpakas 2f1a862b83 build(docs-infra): upgrade preview server docker image to Debian 9 (#29976)
Previously, the preview server docker image was based on Debian 8
(jessie). Recently, `jessie-updates` and `jessie-backborts` were removed
from the Debian mirrors ([more info][1]), thus breaking new builds of
the image.

Instead of updating `/etc/apt/sources.list` to remove the obsolete
sources, this commit upgrades to Debian 9 (stretch).

(The GCE VM running the preview server docker container was also
upgraded from Debian 8 to 9 this morning.)

---
Other changes:
- Removed dependency on `chkconfig`, which is not supported on Debian 9.
- Installing `nginx` from the regular repositories (instead of
  `*-backports).
- Upgraded to `pm2` v3, which can handle hooking itself up to system
  startup better (without `chkconfig` - see above).
- Updated tests to reflect the fact that `nginx` has dropped the reason
  phrase in response status lines for HTTP/2 (in compliance with
  [the spec][2]). (HTTP/1.1: `HTTP/1.1 200 OK` | HTTP/2: `HTTP/2 200`)

[1]: https://www.lucas-nussbaum.net/blog/?p=947
[2]: https://http2.github.io/http2-spec/#rfc.section.8.1.2.4

PR Close #29976
2019-04-23 08:33:27 -07:00
George Kalpakas 24c06091f2 build(docs-infra): increase build artifact size limit for preview server (#29976)
In #29926, the size of the build artifacts has increased due to turning
on differential loading (which generates an es2015/es5 pair for each JS
resource).

To avoid the preview server's rejecting the build artifacts (as in
[288181][1]), this commit increases the max allowed artifact size from
20MB to 25MB (current artifact size after #29926 is ~22MB).

[1]: https://circleci.com/gh/angular/angular/288181

PR Close #29976
2019-04-23 08:33:27 -07:00
George Kalpakas 0c59342cd0 build(docs-infra): automatically create previews for members of `aio-auto-previews` (#29293)
PR Close #29293
2019-03-14 10:51:18 -04:00
George Kalpakas a0cdefb5fb ci(docs-infra): do not automatically post preview comments on PRs for team (#28211)
Right now, we post such comments whenever a file has been touched that
could potentially have affected the docs. Since the API docs are built
from comments in the source code, almost all non-docs changes are
generating such preview comments, even though most of the time they are
irrelevant to the author and create unnecessary noise on the PR
(especially for actively worked-on PRs).

This commit removes the `team` GitHub team from the list of teams whose
members will automatically get preview comments.
(Adding the `aio: preview` label would still work on any PR.)

Jira: FW-967

PR Close #28211
2019-01-17 14:11:42 -08:00
George Kalpakas 7524c99be2 fix(docs-infra): log the successful creation of previews (#27436)
This can help with debugging issues, e.g. with the communication between
the preview server and CI, as it gives a better idea of exactly when was
the preview made available and how long it took.

PR Close #27436
2018-12-04 19:59:24 -08:00
George Kalpakas f45aedcbf0 build(docs-infra): upgrade `npm-run-all` to latest version for security (#27274)
Earlier versions may transitively depend on a malicious version of
`flatmap-stream` (see dominictarr/event-stream#116).

The `aio-builds-setup/` had an older version of `event-stream` (3.3.4),
which did not depend on `flatmap-stream`, but upgraded it anyway.

PR Close #27274
2018-11-26 12:20:48 -08:00
George Kalpakas ce6948fc1b ci(docs-infra): remove jobs from Travis config (#26377)
PR Close #26377
2018-10-23 14:35:38 -07:00
George Kalpakas ab6f055479 ci(docs-infra): show custom 404 page on preview server (for consistency) (#26199)
PR Close #26199
2018-10-05 15:39:02 -07:00
George Kalpakas a01acec7fe fix(docs-infra): use correct parameters for paginated requests to GitHub (#25671)
As it turns out, in GitHub API paginated requests, page numbering is
1-based. (https://developer.github.com/v3/#pagination)

Starting at page 0 (which returns the first page), results in making the
same request twice and logging incorrect numbers (since the first 100
items are listed twice).

PR Close #25671
2018-09-26 15:26:19 -07:00
George Kalpakas 021f4344b1 fix(docs-infra): fix preview server periodic clean-up (#25671)
Includes the following fixes:

- Fix cron entry format for clean-up script.
  Crontabs in `/etc` should not have a user field. No idea why it used
  to work before, but it started giving errors recently:
  `/bin/sh: root: not found`.

- Set required env variable in clean-up script. (Broken in cc6f36a9d.)
  This was producing the following error:
  `ERROR: Missing required environment variable 'AIO_CIRCLE_CI_TOKEN'!`

- Use the correct path for downloads to be removed. (Broken in cc6f36a9d.)

PR Close #25671
2018-09-26 15:26:19 -07:00
George Kalpakas f113b49909 test(docs-infra): remove unnecessary test helpers (#25671)
`supertest.Request` extends `Promise` and can be used directly without
"promisifying".

PR Close #25671
2018-09-26 15:26:19 -07:00
George Kalpakas d8d276c245 docs(docs-infra): update preview server docs to account for recent changes (#25671)
Mostly (but not exclusively) a follow-up to #23576.

PR Close #25671
2018-09-26 15:26:19 -07:00
George Kalpakas 6d6b0ff1ad feat(docs-infra): add API endpoint for checking if PR can have preview (#25671)
There several reasons why PRs cannot have (public) previews:
- The PR did not affect any relevant files (e.g. non-spec files in
  `aio/` or `packages/`).
- The PR cannot be automatically verified as "trusted" (based on its
  author or labels).

Note:
The endpoint does not check whether there currently is a (public)
preview for the specified PR; only whether there can be one.

PR Close #25671
2018-09-26 15:26:19 -07:00
George Kalpakas f378454c92 fix(docs-infra): correctly check PR files on preview server (#25671)
According to the docs, the response of GitHub's [PR files API][1]
_"includes a maximum of 300 files"_. This means that if a PR contains
more files, it is possible that not all files are retrieved (which
could, for example, give a false negative for the "significant files
touched" check - not likely but possible).

This commit fixes it by using paginated requests to retrieve all changed
files.

[1]: https://developer.github.com/v3/pulls/#list-pull-requests-files

PR Close #25671
2018-09-26 15:26:19 -07:00
George Kalpakas c8c8436e58 test(docs-infra): fix test for preview server's `GithubPullRequests` (#25671)
PR Close #25671
2018-09-26 15:26:19 -07:00
George Kalpakas b31c8b6063 test(docs-infra): fix test for preview server's `BuildCleaner` completing prematurely (#25671)
PR Close #25671
2018-09-26 15:26:19 -07:00
George Kalpakas 897261efdc test(docs-infra): fix preview server unit tests on Windows (#25671)
Some tests where comparing actual with expected paths, without taking
into account that paths will be different on Windows.

This commit uses `path.resolve()` to convert expected paths to their
OS-specific form.

PR Close #25671
2018-09-26 15:26:19 -07:00
George Kalpakas 35d70ff265 test(docs-infra): add support for source-maps in preview server tests (#25671)
PR Close #25671
2018-09-26 15:26:19 -07:00
George Kalpakas fc0a7959a4 refactor(docs-infra): use mockable logger (#25671)
Related discussion:
https://github.com/angular/angular/pull/23576#discussion_r187925949.

PR Close #25671
2018-09-26 15:26:19 -07:00
George Kalpakas 182c08bee1 refactor(docs-infra): fix method name (getPrfromBranch --> getPrFromBranch) (#25671)
PR Close #25671
2018-09-26 15:26:19 -07:00
George Kalpakas e2bc0ad6c2 build(docs-infra): upgrade preview server dependencies (#25671)
PR Close #25671
2018-09-26 15:26:19 -07:00
George Kalpakas 73333ee3e5 build(docs-infra): replace `concurrently` with `npm-run-all` for preview server dev (#25671)
`npm-run-all` works just as well, but is better at handling termination on Windows.

PR Close #25671
2018-09-26 15:26:19 -07:00
George Kalpakas c819859598 build(docs-infra): do not exit preview server `dev` script when `build` fails (#25671)
PR Close #25671
2018-09-26 15:26:19 -07:00
George Kalpakas 79aefa7659 build(docs-infra): avoid race condition in `aio-builds-setup/` npm scripts (#25671)
Previously, due to multiple scripts re-building during `yarn dev`
initialization, there could be race conditions that led to errors.

This commit fixes it by ensuring `yarn build` is run once (before
the main `yarn dev` script).

PR Close #25671
2018-09-26 15:26:19 -07:00
Pete Bacon Darwin 92c8752d0a docs(docs-infra): the `build.sh` script was renamed to `create-image.sh` 2018-08-16 10:26:13 +01:00
Pete Bacon Darwin 68bfe686d8 ci(docs-infra): rename 'upload-server' to 'preview-server'
The server no longer has files uploaded to it. Instead it is more
accurate to refer to it as dealing with "previews" of PRs.
2018-08-16 10:26:13 +01:00
Pete Bacon Darwin d604ef7cf0 ci(docs-infra): add explicit return types to methods 2018-08-16 10:26:13 +01:00
Pete Bacon Darwin 36c4c8daa9 ci(docs-infra): improve preview-server logging 2018-08-16 10:26:13 +01:00
Pete Bacon Darwin cc6f36a9d7 ci(docs-infra): change AIO preview server stuff to pull builds from CircleCI
Previously, Travis pushed the build artitfacts to the preview server.
This required us to use JWT to secure the POST request from Travis, to
ensure we couldn't receive malicious builds.

JWT has been deprecated and we are moving our builds to CircleCI.

This commit rewrites the TypeScript part of the preview server that
handles converting build artifact into hosted previews of the docs.
2018-08-16 10:26:13 +01:00
Pete Bacon Darwin 643766637e ci(docs-infra): factor out the aio-builds-setup environment variables 2018-08-16 10:26:12 +01:00
Pete Bacon Darwin 3b8b7f4087 ci(docs-infra): add helper scripts for running TDD in Docker 2018-08-16 10:26:12 +01:00
Pete Bacon Darwin 9b820555a3 docs(docs-infra): update the preview server documentation 2018-08-16 10:26:12 +01:00
Pete Bacon Darwin 364459c576 ci(docs-infra): move AIO preview deployment to CircleCI
Now instead of pushing the AIO build artifacts to the preview server
from inside a Travis job, the artifacts are built and hosted on the
CircleCI infrastructure. The preview server will then pull these
down after being triggered by a CircleCI build webhook.
2018-08-16 10:26:11 +01:00
Pete Bacon Darwin 8347bb0d2d ci(docs-infra): update upload-server to run on node.js v10 2018-08-16 10:21:24 +01:00
George Kalpakas 5e8bf2f88d build(docs-infra): ensure `dist/` directory is cleaned before running `tsc --watch` (#24372)
PR Close #24372
2018-06-11 09:18:46 -07:00