docs(docs-infra): update the preview server documentation

aio/aio-builds-setup/docs/image-config--environment-variables.md

@@ -10,18 +10,26 @@ environment variables and their default values can be found in the
 Each variable has a `TEST_` prefixed counterpart, which is used for testing purposes. In most cases
 you don't need to specify values for those.
 
+- `AIO_ARTIFACT_PATH`:
+  The path used to identify the AIO build artifact on the CircleCI servers. This should be equal to
+  the path given in the `.circleci/config.yml` file for the
+  `aio_preview->steps->store_artifacts->destination` key.
+
 - `AIO_BUILDS_DIR`:
-  The directory (inside the container) where the uploaded build artifacts are kept.
+  The directory (inside the container) where the hosted build artifacts are kept.
 
 - `AIO_DOMAIN_NAME`:
   The domain name of the server.
 
 - `AIO_GITHUB_ORGANIZATION`:
-  The GitHub organization whose teams are whitelisted for accepting uploads.
+  The GitHub organization whose teams are whitelisted for accepting build artifacts.
   See also `AIO_GITHUB_TEAM_SLUGS`.
 
+- `AIO_GITHUB_REPO`:
+  The Github repository for which PRs will be hosted.
+
 - `AIO_GITHUB_TEAM_SLUGS`:
-  A comma-separated list of teams, whose authors are allowed to upload PRs.
+  A comma-separated list of teams, whose authors are allowed to preview PRs.
   See also `AIO_GITHUB_ORGANIZATION`.
 
 - `AIO_NGINX_HOSTNAME`:
@@ -36,8 +44,10 @@ you don't need to specify values for those.
   The port number on which nginx listens for HTTPS connections. This should be mapped to the
   corresponding port on the host VM (as described [here](vm-setup--start-docker-container.md)).
 
-- `AIO_REPO_SLUG`:
+- `AIO_SIGNIFICANT_FILES_PATTERN`:
-  The repository slug (in the form `<user>/<repo>`) for which PRs will be uploaded.
+  The RegExp that determines whether a changed file indicates that a new preview needs to
+  be deployed. For example, if there is a changed file in the `/packages` directory then
+  some of the API docs might have changed, so we need to create a new preview.
 
 - `AIO_TRUSTED_PR_LABEL`:
   The PR whose presence indicates the PR has been manually verified and is allowed to have its

aio/aio-builds-setup/docs/misc--debug-docker-container.md

@@ -10,3 +10,39 @@ TODO (gkalpak): Add docs. Mention:
 - Test upload-server accessible at:
   - `http://$TEST_AIO_UPLOAD_HOTNAME:$TEST_AIO_UPLOAD_PORT`
 - Local DNS (via dnsmasq) maps the above hostnames to 127.0.0.1
+
+
+## Developing the upload server TypeScript files
+
+If you are running Docker on OS/X then you can benefit from linking the built TypeScript
+files (i.e. `script-js/dist`) to the JavaScript files inside the Docker container.
+
+First start watching and building the TypeScript files (in the host):
+
+```bash
+yarn build-watch
+```
+
+Now build, start and attach to the Docker container. See "Setting up the VM"
+section in [TOC](_TOC.md). Then link the JavaScript folders (in the container):
+
+```bash
+aio-dev-mode
+```
+
+Now whenever you make changes to the TypeScript, it will be automatically built
+in the host, and the changes are automatically available in the container.
+You can then run the unit tests (in the container):
+
+```bash
+aio-verify-setup
+```
+
+Sometimes, the errors in the unit test log are not enough to tell you what went wrong.
+In that case you can also look at the log of the upload-server itself.
+A helper script that runs the unit tests (i.e. `aio-verify-setup`) and displays the
+last relevant test-upload-server log is:
+
+```bash
+aio-verify-setup-and-log
+```

aio/aio-builds-setup/docs/misc--integrate-with-ci.md

@@ -2,10 +2,7 @@
 
 
 TODO (gkalpak): Add docs. Mention:
-- Travis' JWT addon (+ limitations).
-  Relevant files: `.travis.yml`, `scripts/ci/env.sh`
 - Testing on CI.
   Relevant files: `scripts/ci/test-aio.sh`, `aio/aio-builds-setup/scripts/test.sh`
 - Deploying from CI.
-  Relevant files: `scripts/ci/deploy.sh`, `aio/scripts/deploy-preview.sh`,
+  Relevant files: `scripts/ci/deploy.sh`, `aio/scripts/deploy-to-firebase.sh`
-                  `aio/scripts/deploy-to-firebase.sh`

aio/aio-builds-setup/docs/overview--general.md

@@ -2,9 +2,10 @@
 
 
 ## Objective
-Whenever a PR job is run on Travis, we want to build `angular.io` and upload the build artifacts to
+Whenever a PR job is run on the CI infrastructure (e.g. CircleCI), we want to build `angular.io`
-a publicly accessible server so that collaborators (developers, designers, authors, etc) can preview
+and upload the build artifacts to a publicly accessible server so that collaborators (developers,
-the changes without having to checkout and build the app locally.
+designers, authors, etc) can preview the changes without having to checkout and build the app
+locally.
 
 
 ## Source code
@@ -32,34 +33,24 @@ This section gives a brief summary of the several operations performed on CI and
 container:
 
 
-### On CI (Travis)
+### On CI (CircleCI)
 - Build job completes successfully.
 - The CI script checks whether the build job was initiated by a PR against the angular/angular
   master branch.
 - The CI script checks whether the PR has touched any files that might affect the angular.io app
   (currently the `aio/` or `packages/` directories, ignoring spec files).
-- Optionally, the CI script can check whether the PR can be automatically verified (i.e. if the
+- The CI script gzips and stores the build artifacts in the CI infrastructure.
-  author of the PR is a member of one of the whitelisted GitHub teams or the PR has the specified
+- When the build completes CircleCI triggers a webhook on the upload-server.
-  "trusted PR" label).
-  **Note:**
-  For security reasons, the same checks will be performed on the server as well. This is an optional
-  step that can be used in case one wants to apply special logic depending on the outcome of the
-  pre-verification. For example:
-  1. One might want to deploy automatically verified PRs only. In that case, the pre-verification
-     helps avoid the wasted overhead associated with uploads that are going to be rejected (e.g.
-     building the artifacts, sending them to the server, running checks on the server, detecting the
-     reasons of deployment failure and whether to fail the build, etc).
-  2. One might want to apply additional logic (e.g. different tests) depending on whether the PR is
-     automatically verified or not).
-- The CI script gzips and uploads the build artifacts to the server.
 
 More info on how to set things up on CI can be found [here](misc--integrate-with-ci.md).
 
 
-### Uploading build artifacts
+### Hosting build artifacts
-- nginx receives the upload request.
+
-- nginx checks that the uploaded gzip archive does not exceed the specified max file size, stores it
+- nginx receives the webhook trigger and passes it through to the upload server.
-  in a temporary location and passes the filepath to the Node.js upload-server.
+- The upload-server makes a request to CircleCI for the URL of the AIO build artifacts.
+- The upload-server makes a request to this URL to receive the artifact - failing if the size
+  exceeds the specified max file size - and stores it in a temporary location.
 - The upload-server runs several checks to determine whether the request should be accepted and
   whether it should be publicly accessible or stored for later verification (more details can be
   found [here](overview--security-model.md)).

aio/aio-builds-setup/docs/overview--http-status-codes.md

@@ -25,7 +25,7 @@ with a bried explanation of what they mean:
   File not found.
 
 
-## `https://ngbuilds.io/create-build/<pr>/<sha>`
+## `https://ngbuilds.io/circle-build`
 
 - **201 (Created)**:
   Build deployed successfully and is publicly available.
@@ -33,14 +33,14 @@ with a bried explanation of what they mean:
 - **202 (Accepted)**:
   Build not automatically verifiable. Stored for later deployment (after re-verification).
 
-- **400 (Bad Request)**:
+- **204 (No Content)**:
-  No payload.
+  Build was not successful, so no further action is being taken.
 
-- **401 (Unauthorized)**:
+- **400 (Bad Request)**:
-  No `AUTHORIZATION` header.
+  Invalid payload.
 
 - **403 (Forbidden)**:
-  Unable to verify build (e.g. invalid JWT token, or unable to talk to 3rd-party APIs, etc).
+  Unable to talk to 3rd-party APIs.
 
 - **405 (Method Not Allowed)**:
   Request method other than POST.
@@ -49,9 +49,6 @@ with a bried explanation of what they mean:
   Request to overwrite existing (public or non-public) directory (e.g. deploy existing build or
   change PR visibility when the destination directory does already exist).
 
-- **413 (Payload Too Large)**:
-  Payload larger than size specified in `AIO_UPLOAD_MAX_SIZE`.
-
 
 ## `https://ngbuilds.io/health-check`
 

aio/aio-builds-setup/docs/overview--scripts-and-commands.md

@@ -21,7 +21,7 @@ available:
   from a git repository. See [here](vm-setup--update-docker-container.md) for more info.
 
 
-## Commands
+## Production Commands
 The following commands are available globally from inside the docker container. They are either used
 by the container to perform its various operations or can be used ad-hoc, mainly for testing
 purposes. Each command is backed by a corresponding script inside
@@ -44,6 +44,9 @@ purposes. Each command is backed by a corresponding script inside
   Spins up a Node.js upload-server instance.
   _It is used in `aio-init` (see above) during initialization._
 
+
+## Developer Commands
+
 - `aio-upload-server-test`:
   Spins up a Node.js upload-server instance for tests.
   _It is used in `aio-verify-setup` (see below) for running tests._
@@ -51,3 +54,13 @@ purposes. Each command is backed by a corresponding script inside
 - `aio-verify-setup`:
   Runs a suite of e2e-like tests, mainly verifying the correct (inter)operation of nginx and the
   Node.js upload-server.
+
+- `aio-verify-setup-and-log`:
+  Runs the `aio-verify-setup` command but also then dumps the logs from the upload server, which
+  gives additional useful debugging information. See the [debugging docs](misc--debug-docker-container.md)
+  for more info.
+
+- `aio-dev-mode`:
+  Links external source files (from the Docker host) to interal source files (in the Docker
+  container). This makes it easier to use an IDE to edit files in the host that are then
+  tested in the container. See the [debugging docs](misc--debug-docker-container.md) for more info.

aio/aio-builds-setup/docs/overview--security-model.md

@@ -1,27 +1,27 @@
 # Overview - Security model
 
 
-Whenever a PR job is run on Travis, we want to build `angular.io` and upload the build artifacts to
+Whenever a PR job is run on CircleCI, we want to build `angular.io` and host the build artifacts on
 a publicly accessible server so that collaborators (developers, designers, authors, etc) can preview
 the changes without having to checkout and build the app locally.
 
-This document discusses the security considerations associated with uploading build artifacts as
+This document discusses the security considerations associated with moving build artifacts as
-part of the CI setup and serving them publicly.
+part of the CI process and serving them publicly.
 
 
 ## Security objectives
 
-- **Prevent uploading arbitrary content to our servers.**
+- **Prevent hosting arbitrary content to on servers.**
-  Since there is no restriction on who can submit a PR, we cannot allow any PR's build artifacts to
+  Since there is no restriction on who can submit a PR, we cannot allow arbitrary untrusted PRs'
-  be uploaded.
+  build artifacts to be hosted.
 
-- **Prevent overwriting other peoples uploaded content.**
+- **Prevent overwriting other people's hosted build artifacts.**
-  There needs to be a mechanism in place to ensure that the uploaded content does indeed correspond
+  There needs to be a mechanism in place to ensure that the hosted content does indeed correspond
   to the PR indicated by its URL.
 
 - **Prevent arbitrary access on the server.**
-  Since the PR author has full access over the build artifacts that would be uploaded, we must
+  Since the PR author has full access over the build artifacts that would be hosted, we must
-  ensure that the uploaded files will not enable arbitrary access to the server or expose sensitive
+  ensure that the build artifacts will not have arbitrary access to the server or expose sensitive
   info.
 
 
@@ -30,7 +30,7 @@ part of the CI setup and serving them publicly.
 - Because the PR author can change the scripts run on CI, any security mechanisms must be immune to
   such changes.
 
-- For security reasons, encrypted Travis variables are not available to PRs, so we can't rely on
+- For security reasons, encrypted CircleCI variables are not available to PRs, so we can't rely on
   them to implement security.
 
 
@@ -40,7 +40,8 @@ part of the CI setup and serving them publicly.
 ### In a nutshell
 The implemented approach can be broken up to the following sub-tasks:
 
-1. Verify which PR the uploaded artifacts correspond to.
+0. Receive notification from CircleCI of a completed build.
+1. Verify that the build is valid and download the artifact.
 2. Fetch the PR's metadata, including author and labels.
 3. Check whether the PR can be automatically verified as "trusted" (based on its author or labels).
 4. If necessary, update the corresponding PR's verification status.
@@ -53,22 +54,28 @@ The implemented approach can be broken up to the following sub-tasks:
 ### Implementation details
 This section describes how each of the aforementioned sub-tasks is accomplished:
 
-1. **Verify which PR the uploaded artifacts correspond to.**
+0. **Receive notification from CircleCI of a completed build**
 
-   We are taking advantage of Travis' [JWT addon](https://docs.travis-ci.com/user/jwt). By sharing
+  CircleCI is configured to trigger a webhook on our upload-server whenever a build completes.
-   a secret between Travis (which keeps it private but uses it to sign a JWT) and the server (which
+  The payload contains the number of the build that completed.
-   uses it to verify the authenticity of the JWT), we can accomplish the following:
-   a. Verify that the upload request comes from Travis.
-   b. Determine the PR that these artifacts correspond to (since Travis puts that information into
-      the JWT, without the PR author being able to modify it).
 
-   _Note:_
+1. **Verify that the build is valid and download the artifact.**
-   _There are currently certain limitation in the implementation of the JWT addon._
+
-   _See the next section for more details._
+   We cannot trust that the data in the webhook trigger is authentic, so we only extract the build
+   number and then run a direct query against the CircleCI API to get hold of the real data for
+   the given build number.
+
+   If the build was not successful then we ignore this trigger. Otherwise we check that the
+   associated github organisation and repository are what we expect (e.g. angular/angular).
+
+   Next we make another call to the CircleCI API to get a list of the URLS for artifacts of that
+   build. If there is one that matches the configured artifact path, we download the contents of the
+   build artifact and store it in a local folder. This download has a maximum size limit to prevent
+   PRs from producing artifacts that are so large they would cause the upload server to crash.
 
 2. **Fetch the PR's metadata, including author and labels**.
 
-   Once we have securely associated the uploaded artifacts to a PR, we retrieve the PR's metadata -
+   Once we have securely downloaded the artifact for a build, we retrieve the PR's metadata -
    including the author's username and the labels - using the
    [GitHub API](https://developer.github.com/v3/).
    To avoid rate-limit restrictions, we use a Personal Access Token (issued by
@@ -96,19 +103,19 @@ This section describes how each of the aforementioned sub-tasks is accomplished:
 
 5. **Deploy the artifacts to the corresponding PR's directory.**
 
-   With the preceding steps, we have verified that the uploaded artifacts have been uploaded by
+   With the preceding steps, we have verified that the build artifacts are valid.
-   Travis. Additionally, we have determined whether the PR can be trusted to have its previews
+   Additionally, we have determined whether the PR can be trusted to have its previews
    publicly accessible or whether further verification is necessary. The artifacts will be stored to
    the PR's directory, but will not be publicly accessible unless the PR has been verified.
    Essentially, as long as sub-tasks 1, 2 and 3 can be securely accomplished, it is possible to
-   "project" the trust we have in a team's members through the PR and Travis to the build artifacts.
+   "project" the trust we have in a team's members through the PR to the build artifacts.
 
 6. **Prevent overwriting previously deployed artifacts**.
 
    In order to enforce this restriction (and ensure that the deployed artifacts' validity is
-   preserved throughout their "lifetime"), the server that handles the upload (currently a Node.js
+   preserved throughout their "lifetime"), the server that handles the artifacts (currently a Node.js
-   Express server) rejects uploads that target an existing directory.
+   Express server) rejects builds that have already been handled.
-   _Note: A PR can contain multiple uploads; one for each SHA that was built on Travis._
+   _Note: A PR can contain multiple builds; one for each SHA that was built on CircleCI._
 
 7. **Prevent uploaded files from accessing anything outside their directory.**
 
@@ -118,6 +125,11 @@ This section describes how each of the aforementioned sub-tasks is accomplished:
 
 ## Assumptions / Things to keep in mind
 
+- Other than the initial webhook trigger, which provides a build number, all requests for data come
+  from the upload-server making requests to well defined API endpoints (e.g. CircleCI and Github).
+  This means that any secret access keys need only be stored on the upload-server and not on any of
+  the CI build infrastructure (e.g. CircleCI).
+
 - Each trusted PR author has full control over the content that is uploaded for their PRs. Part of
   the security model relies on the trustworthiness of these authors.
 
@@ -125,14 +137,3 @@ This section describes how each of the aforementioned sub-tasks is accomplished:
   the content that is uploaded for the specific PR (e.g. by pushing more commits to it). The user
   adding the label is responsible for ensuring that this control is not abused and that the PR is
   either closed (one way of another) or the access is revoked.
-
-- If anyone gets access to the `PREVIEW_DEPLOYMENT_TOKEN` (a.k.a. `NGBUILDS_IO_KEY` on
-  angular/angular) variable generated for each Travis job, they will be able to impersonate the
-  corresponding PR's author on the preview server for as long as the token is valid (currently 90
-  mins). Because of this, the value of the `PREVIEW_DEPLOYMENT_TOKEN` should not be made publicly
-  accessible (e.g. by printing it on the Travis job log).
-
-- Travis does only allow specific whitelisted property names to be used with the JWT addon. The only
-  known such property at the time is `SAUCE_ACCESS_KEY` (used for integration with SauceLabs). In
-  order to be able to actually use the JWT addon we had to name the encrypted variable
-  `SAUCE_ACCESS_KEY` (which we later re-assign to `NGBUILDS_IO_KEY`).

aio/aio-builds-setup/docs/vm-setup--create-docker-image.md

@@ -1,6 +1,12 @@
 # VM setup - Create docker image
 
 
+## Install node and yarn
+- Install [nvm](https://github.com/creationix/nvm#installation).
+- Install node.js: `nvm install 8`
+- Install yarn: `npm -g install yarn`
+
+
 ## Checkout repository
 - `git clone <repo-url>`
 

aio/aio-builds-setup/docs/vm-setup--set-up-secrets.md

@@ -12,14 +12,10 @@ Necessary secrets:
      - Retrieving members of the trusted GitHub teams.
      - Posting comments with preview links on PRs.
 
-2. `PREVIEW_DEPLOYMENT_TOKEN`
+2. `CIRCLE_CI_TOKEN`
    - Used for:
-     - Decoding the JWT tokens received with `/create-build` requests.
+     - Retrieving build information.
-
+     - Downloading build artifacts.
-**Note:**
-`TEST_GITHUB_TOKEN` and `TEST_PREVIEW_DEPLOYMENT_TOKEN` can also be created similar to their
-non-TEST counterparts and they will be loaded when running `aio-verify-setup`, but it is currently
-not clear if/how they can be used in tests.
 
 
 ## Create secrets
@@ -28,25 +24,15 @@ not clear if/how they can be used in tests.
    - Visit https://github.com/settings/tokens.
    - Generate new token with the `public_repo` scope.
 
-2. `PREVIEW_DEPLOYMENT_TOKEN`
+2. `CIRCLE_CI_TOKEN`
-   - Just generate a hard-to-guess character sequence.
+   - Visit https://circleci.com/gh/angular/angular/edit#api
-   - Add it to `.travis.yml` under `addons -> jwt -> secure`.
+   - Create an API token with `Build Artifacts` scope
-     Can be added automatically with: `travis encrypt --add addons.jwt PREVIEW_DEPLOYMENT_TOKEN=<access-key>`
-
-**Note:**
-Due to [travis-ci/travis-ci#7223](https://github.com/travis-ci/travis-ci/issues/7223) it is not
-currently possible to use the JWT addon (as described above) for anything other than the
-`SAUCE_ACCESS_KEY` variable. You can get creative, though...
-
-**WARNING**
-TO avoid arbitrary uploads, make sure the `PREVIEW_DEPLOYMENT_TOKEN` is NOT printed in the Travis log.
-
 
 ## Save secrets on the VM
 
 - `sudo mkdir /aio-secrets`
 - `sudo touch /aio-secrets/GITHUB_TOKEN`
 - Insert `<github-token>` into `/aio-secrets/GITHUB_TOKEN`.
-- `sudo touch /aio-secrets/PREVIEW_DEPLOYMENT_TOKEN`
+- `sudo touch /aio-secrets/CIRCLE_CI_TOKEN`
-- Insert `<access-token>` into `/aio-secrets/PREVIEW_DEPLOYMENT_TOKEN`.
+- Insert `<access-token>` into `/aio-secrets/CIRCLE_CI_TOKEN`.
 - `sudo chmod 400 /aio-secrets/*`

aio/aio-builds-setup/docs/vm-setup--start-docker-container.md

@@ -13,14 +13,15 @@ sudo docker run \
   --publish 80:80 \
   --publish 443:443 \
   --restart unless-stopped \
- [--volume <host-cert-dir>:/etc/ssl/localcerts:ro] \
   --volume <host-secrets-dir>:/aio-secrets:ro \
   --volume <host-builds-dir>:/var/www/aio-builds \
+ [--volume <host-cert-dir>:/etc/ssl/localcerts:ro] \
  [--volume <host-logs-dir>:/var/log/aio] \
+ [--volume <host-dockerbuild-dir>:/dockerbuild] \
   <name>[:<tag>]
 ```
 
-Below is the same command with inline comments explaining each option. The aPI docs for `docker run`
+Below is the same command with inline comments explaining each option. The API docs for `docker run`
 can be found [here](https://docs.docker.com/engine/reference/run/).
 
 ```
@@ -45,11 +46,6 @@ sudo docker run \
   # (This ensures that the container will be automatically started on boot.)
   --restart unless-stopped \
 
-  # The directory the contains the SSL certificates.
-  # (See [here](vm-setup--create-host-dirs-and-files.md) for more info.)
-  # If not provided, the container will use self-signed certificates.
- [--volume <host-cert-dir>:/etc/ssl/localcerts:ro] \
-
   # The directory the contains the secrets (e.g. GitHub token, JWT secret, etc).
   # (See [here](vm-setup--set-up-secrets.md) for more info.)
   --volume <host-secrets-dir>:/aio-secrets:ro \
@@ -59,14 +55,23 @@ sudo docker run \
   #  this will be a directory inside the disk.)
   --volume <host-builds-dir>:/var/www/aio-builds \
 
+  # The directory the contains the SSL certificates.
+  # (See [here](vm-setup--create-host-dirs-and-files.md) for more info.)
+  # If not provided, the container will use self-signed certificates.
+ [--volume <host-cert-dir>:/etc/ssl/localcerts:ro] \
+
   # The directory where the logs are being kept.
   # (See [here](vm-setup--create-host-dirs-and-files.md) for more info.)
   # If not provided, the logs will be kept inside the container, which means they will be lost
   # whenever a new container is created.
  [--volume <host-logs-dir>:/var/log/aio] \
 
+  # This directory allows you to share the source scripts between the host and the container when
+  # debugging. (See [here](misc--debug-docker-container.md) for how to set this up.)
+ [--volume <host-dockerbuild-dir>:/dockerbuild] \
+
   # The name of the docker image to use (and an optional tag; defaults to `latest`).
-  # (See [here](vm-setup--create-docker-image.md) for instructions on how to create the iamge.)
+  # (See [here](vm-setup--create-docker-image.md) for instructions on how to create the image.)
   <name>[:<tag>]
 ```
 
@@ -74,7 +79,8 @@ sudo docker run \
 ## Example
 The following command would start a docker container based on the previously created `foobar-builds`
 docker image, alias it as 'foobar-builds-1' and map predefined directories on the host VM to be used
-by the container for accessing secrets and SSL certificates and keeping the build artifacts and logs.
+by the container for accessing secrets and SSL certificates and keeping the build artifacts and logs;
+and will map the source scripts from the host to the container.
 
 ```
 sudo docker run \
@@ -84,9 +90,10 @@ sudo docker run \
   --publish 80:80 \
   --publish 443:443 \
   --restart unless-stopped \
-  --volume /etc/ssl/localcerts:/etc/ssl/localcerts:ro \
   --volume /foobar-secrets:/aio-secrets:ro \
   --volume /mnt/disks/foobar-builds:/var/www/aio-builds \
+  --volume /etc/ssl/localcerts:/etc/ssl/localcerts:ro \
   --volume /foobar-logs:/var/log/aio \
+  --volume ~/angular/aio/aio-builds-setup/dockerbuild:/dockerbuild \
   foobar-builds
 ```