honeymoose
/
angular-cn
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

docs(docs-infra): update the preview server documentation

This commit is contained in:
Pete Bacon Darwin 2018-05-09 18:54:53 +01:00
parent 364459c576
commit 9b820555a3
10 changed files with 157 additions and 113 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

20
aio/aio-builds-setup/docs/image-config--environment-variables.md
View File

@ -10,18 +10,26 @@ environment variables and their default values can be found in the
Each variable has a `TEST_` prefixed counterpart, which is used for testing purposes. In most cases
you don't need to specify values for those.
- `AIO_ARTIFACT_PATH`:
The path used to identify the AIO build artifact on the CircleCI servers. This should be equal to
the path given in the `.circleci/config.yml` file for the
`aio_preview->steps->store_artifacts->destination` key.
- `AIO_BUILDS_DIR`:
The directory (inside the container) where the uploaded build artifacts are kept.
The directory (inside the container) where the hosted build artifacts are kept.
- `AIO_DOMAIN_NAME`:
The domain name of the server.
- `AIO_GITHUB_ORGANIZATION`:
The GitHub organization whose teams are whitelisted for accepting uploads.
The GitHub organization whose teams are whitelisted for accepting build artifacts.
See also `AIO_GITHUB_TEAM_SLUGS`.
- `AIO_GITHUB_REPO`:
The Github repository for which PRs will be hosted.
- `AIO_GITHUB_TEAM_SLUGS`:
A comma-separated list of teams, whose authors are allowed to upload PRs.
A comma-separated list of teams, whose authors are allowed to preview PRs.
See also `AIO_GITHUB_ORGANIZATION`.
- `AIO_NGINX_HOSTNAME`:
@ -36,8 +44,10 @@ you don't need to specify values for those.
The port number on which nginx listens for HTTPS connections. This should be mapped to the
corresponding port on the host VM (as described [here](vm-setup--start-docker-container.md)).
- `AIO_REPO_SLUG`:
The repository slug (in the form `<user>/<repo>`) for which PRs will be uploaded.
- `AIO_SIGNIFICANT_FILES_PATTERN`:
The RegExp that determines whether a changed file indicates that a new preview needs to
be deployed. For example, if there is a changed file in the `/packages` directory then
some of the API docs might have changed, so we need to create a new preview.
- `AIO_TRUSTED_PR_LABEL`:
The PR whose presence indicates the PR has been manually verified and is allowed to have its

36
aio/aio-builds-setup/docs/misc--debug-docker-container.md
View File

@ -10,3 +10,39 @@ TODO (gkalpak): Add docs. Mention:
- Test upload-server accessible at:
- `http://$TEST_AIO_UPLOAD_HOTNAME:$TEST_AIO_UPLOAD_PORT`
- Local DNS (via dnsmasq) maps the above hostnames to 127.0.0.1
## Developing the upload server TypeScript files
If you are running Docker on OS/X then you can benefit from linking the built TypeScript
files (i.e. `script-js/dist`) to the JavaScript files inside the Docker container.
First start watching and building the TypeScript files (in the host):
```bash
yarn build-watch
```
Now build, start and attach to the Docker container. See "Setting up the VM"
section in [TOC](_TOC.md). Then link the JavaScript folders (in the container):
```bash
aio-dev-mode
```
Now whenever you make changes to the TypeScript, it will be automatically built
in the host, and the changes are automatically available in the container.
You can then run the unit tests (in the container):
```bash
aio-verify-setup
```
Sometimes, the errors in the unit test log are not enough to tell you what went wrong.
In that case you can also look at the log of the upload-server itself.
A helper script that runs the unit tests (i.e. `aio-verify-setup`) and displays the
last relevant test-upload-server log is:
```bash
aio-verify-setup-and-log
```

5
aio/aio-builds-setup/docs/misc--integrate-with-ci.md
View File

@ -2,10 +2,7 @@
TODO (gkalpak): Add docs. Mention:
- Travis' JWT addon (+ limitations).
Relevant files: `.travis.yml`, `scripts/ci/env.sh`
- Testing on CI.
Relevant files: `scripts/ci/test-aio.sh`, `aio/aio-builds-setup/scripts/test.sh`
- Deploying from CI.
Relevant files: `scripts/ci/deploy.sh`, `aio/scripts/deploy-preview.sh`,
`aio/scripts/deploy-to-firebase.sh`
Relevant files: `scripts/ci/deploy.sh`, `aio/scripts/deploy-to-firebase.sh`

35
aio/aio-builds-setup/docs/overview--general.md
View File

@ -2,9 +2,10 @@
## Objective
Whenever a PR job is run on Travis, we want to build `angular.io` and upload the build artifacts to
a publicly accessible server so that collaborators (developers, designers, authors, etc) can preview
the changes without having to checkout and build the app locally.
Whenever a PR job is run on the CI infrastructure (e.g. CircleCI), we want to build `angular.io`
and upload the build artifacts to a publicly accessible server so that collaborators (developers,
designers, authors, etc) can preview the changes without having to checkout and build the app
locally.
## Source code
@ -32,34 +33,24 @@ This section gives a brief summary of the several operations performed on CI and
container:
### On CI (Travis)
### On CI (CircleCI)
- Build job completes successfully.
- The CI script checks whether the build job was initiated by a PR against the angular/angular
master branch.
- The CI script checks whether the PR has touched any files that might affect the angular.io app
(currently the `aio/` or `packages/` directories, ignoring spec files).
- Optionally, the CI script can check whether the PR can be automatically verified (i.e. if the
author of the PR is a member of one of the whitelisted GitHub teams or the PR has the specified
"trusted PR" label).
**Note:**
For security reasons, the same checks will be performed on the server as well. This is an optional
step that can be used in case one wants to apply special logic depending on the outcome of the
pre-verification. For example:
1. One might want to deploy automatically verified PRs only. In that case, the pre-verification
helps avoid the wasted overhead associated with uploads that are going to be rejected (e.g.
building the artifacts, sending them to the server, running checks on the server, detecting the
reasons of deployment failure and whether to fail the build, etc).
2. One might want to apply additional logic (e.g. different tests) depending on whether the PR is
automatically verified or not).
- The CI script gzips and uploads the build artifacts to the server.
- The CI script gzips and stores the build artifacts in the CI infrastructure.
- When the build completes CircleCI triggers a webhook on the upload-server.
More info on how to set things up on CI can be found [here](misc--integrate-with-ci.md).
### Uploading build artifacts
- nginx receives the upload request.
- nginx checks that the uploaded gzip archive does not exceed the specified max file size, stores it
in a temporary location and passes the filepath to the Node.js upload-server.
### Hosting build artifacts
- nginx receives the webhook trigger and passes it through to the upload server.
- The upload-server makes a request to CircleCI for the URL of the AIO build artifacts.
- The upload-server makes a request to this URL to receive the artifact - failing if the size
exceeds the specified max file size - and stores it in a temporary location.
- The upload-server runs several checks to determine whether the request should be accepted and
whether it should be publicly accessible or stored for later verification (more details can be
found [here](overview--security-model.md)).

15
aio/aio-builds-setup/docs/overview--http-status-codes.md
View File

@ -25,7 +25,7 @@ with a bried explanation of what they mean:
File not found.
## `https://ngbuilds.io/create-build/<pr>/<sha>`
## `https://ngbuilds.io/circle-build`
- **201 (Created)**:
Build deployed successfully and is publicly available.
@ -33,14 +33,14 @@ with a bried explanation of what they mean:
- **202 (Accepted)**:
Build not automatically verifiable. Stored for later deployment (after re-verification).
- **400 (Bad Request)**:
No payload.
- **204 (No Content)**:
Build was not successful, so no further action is being taken.
- **401 (Unauthorized)**:
No `AUTHORIZATION` header.
- **400 (Bad Request)**:
Invalid payload.
- **403 (Forbidden)**:
Unable to verify build (e.g. invalid JWT token, or unable to talk to 3rd-party APIs, etc).
Unable to talk to 3rd-party APIs.
- **405 (Method Not Allowed)**:
Request method other than POST.
@ -49,9 +49,6 @@ with a bried explanation of what they mean:
Request to overwrite existing (public or non-public) directory (e.g. deploy existing build or
change PR visibility when the destination directory does already exist).
- **413 (Payload Too Large)**:
Payload larger than size specified in `AIO_UPLOAD_MAX_SIZE`.
## `https://ngbuilds.io/health-check`

15
aio/aio-builds-setup/docs/overview--scripts-and-commands.md
View File

@ -21,7 +21,7 @@ available:
from a git repository. See [here](vm-setup--update-docker-container.md) for more info.
## Commands
## Production Commands
The following commands are available globally from inside the docker container. They are either used
by the container to perform its various operations or can be used ad-hoc, mainly for testing
purposes. Each command is backed by a corresponding script inside
@ -44,6 +44,9 @@ purposes. Each command is backed by a corresponding script inside
Spins up a Node.js upload-server instance.
_It is used in `aio-init` (see above) during initialization._
## Developer Commands
- `aio-upload-server-test`:
Spins up a Node.js upload-server instance for tests.
_It is used in `aio-verify-setup` (see below) for running tests._
@ -51,3 +54,13 @@ purposes. Each command is backed by a corresponding script inside
- `aio-verify-setup`:
Runs a suite of e2e-like tests, mainly verifying the correct (inter)operation of nginx and the
Node.js upload-server.
- `aio-verify-setup-and-log`:
Runs the `aio-verify-setup` command but also then dumps the logs from the upload server, which
gives additional useful debugging information. See the [debugging docs](misc--debug-docker-container.md)
for more info.
- `aio-dev-mode`:
Links external source files (from the Docker host) to interal source files (in the Docker
container). This makes it easier to use an IDE to edit files in the host that are then
tested in the container. See the [debugging docs](misc--debug-docker-container.md) for more info.

81
aio/aio-builds-setup/docs/overview--security-model.md
View File

@ -1,27 +1,27 @@
# Overview - Security model
Whenever a PR job is run on Travis, we want to build `angular.io` and upload the build artifacts to
Whenever a PR job is run on CircleCI, we want to build `angular.io` and host the build artifacts on
a publicly accessible server so that collaborators (developers, designers, authors, etc) can preview
the changes without having to checkout and build the app locally.
This document discusses the security considerations associated with uploading build artifacts as
part of the CI setup and serving them publicly.
This document discusses the security considerations associated with moving build artifacts as
part of the CI process and serving them publicly.
## Security objectives
- **Prevent uploading arbitrary content to our servers.**
Since there is no restriction on who can submit a PR, we cannot allow any PR's build artifacts to
be uploaded.
- **Prevent hosting arbitrary content to on servers.**
Since there is no restriction on who can submit a PR, we cannot allow arbitrary untrusted PRs'
build artifacts to be hosted.
- **Prevent overwriting other peoples uploaded content.**
There needs to be a mechanism in place to ensure that the uploaded content does indeed correspond
- **Prevent overwriting other people's hosted build artifacts.**
There needs to be a mechanism in place to ensure that the hosted content does indeed correspond
to the PR indicated by its URL.
- **Prevent arbitrary access on the server.**
Since the PR author has full access over the build artifacts that would be uploaded, we must
ensure that the uploaded files will not enable arbitrary access to the server or expose sensitive
Since the PR author has full access over the build artifacts that would be hosted, we must
ensure that the build artifacts will not have arbitrary access to the server or expose sensitive
info.
@ -30,7 +30,7 @@ part of the CI setup and serving them publicly.
- Because the PR author can change the scripts run on CI, any security mechanisms must be immune to
such changes.
- For security reasons, encrypted Travis variables are not available to PRs, so we can't rely on
- For security reasons, encrypted CircleCI variables are not available to PRs, so we can't rely on
them to implement security.
@ -40,7 +40,8 @@ part of the CI setup and serving them publicly.
### In a nutshell
The implemented approach can be broken up to the following sub-tasks:
1. Verify which PR the uploaded artifacts correspond to.
0. Receive notification from CircleCI of a completed build.
1. Verify that the build is valid and download the artifact.
2. Fetch the PR's metadata, including author and labels.
3. Check whether the PR can be automatically verified as "trusted" (based on its author or labels).
4. If necessary, update the corresponding PR's verification status.
@ -53,22 +54,28 @@ The implemented approach can be broken up to the following sub-tasks:
### Implementation details
This section describes how each of the aforementioned sub-tasks is accomplished:
1. **Verify which PR the uploaded artifacts correspond to.**
0. **Receive notification from CircleCI of a completed build**
We are taking advantage of Travis' [JWT addon](https://docs.travis-ci.com/user/jwt). By sharing
a secret between Travis (which keeps it private but uses it to sign a JWT) and the server (which
uses it to verify the authenticity of the JWT), we can accomplish the following:
a. Verify that the upload request comes from Travis.
b. Determine the PR that these artifacts correspond to (since Travis puts that information into
the JWT, without the PR author being able to modify it).
CircleCI is configured to trigger a webhook on our upload-server whenever a build completes.
The payload contains the number of the build that completed.
_Note:_
_There are currently certain limitation in the implementation of the JWT addon._
_See the next section for more details._
1. **Verify that the build is valid and download the artifact.**
We cannot trust that the data in the webhook trigger is authentic, so we only extract the build
number and then run a direct query against the CircleCI API to get hold of the real data for
the given build number.
If the build was not successful then we ignore this trigger. Otherwise we check that the
associated github organisation and repository are what we expect (e.g. angular/angular).
Next we make another call to the CircleCI API to get a list of the URLS for artifacts of that
build. If there is one that matches the configured artifact path, we download the contents of the
build artifact and store it in a local folder. This download has a maximum size limit to prevent
PRs from producing artifacts that are so large they would cause the upload server to crash.
2. **Fetch the PR's metadata, including author and labels**.
Once we have securely associated the uploaded artifacts to a PR, we retrieve the PR's metadata -
Once we have securely downloaded the artifact for a build, we retrieve the PR's metadata -
including the author's username and the labels - using the
[GitHub API](https://developer.github.com/v3/).
To avoid rate-limit restrictions, we use a Personal Access Token (issued by
@ -96,19 +103,19 @@ This section describes how each of the aforementioned sub-tasks is accomplished:
5. **Deploy the artifacts to the corresponding PR's directory.**
With the preceding steps, we have verified that the uploaded artifacts have been uploaded by
Travis. Additionally, we have determined whether the PR can be trusted to have its previews
With the preceding steps, we have verified that the build artifacts are valid.
Additionally, we have determined whether the PR can be trusted to have its previews
publicly accessible or whether further verification is necessary. The artifacts will be stored to
the PR's directory, but will not be publicly accessible unless the PR has been verified.
Essentially, as long as sub-tasks 1, 2 and 3 can be securely accomplished, it is possible to
"project" the trust we have in a team's members through the PR and Travis to the build artifacts.
"project" the trust we have in a team's members through the PR to the build artifacts.
6. **Prevent overwriting previously deployed artifacts**.
In order to enforce this restriction (and ensure that the deployed artifacts' validity is
preserved throughout their "lifetime"), the server that handles the upload (currently a Node.js
Express server) rejects uploads that target an existing directory.
_Note: A PR can contain multiple uploads; one for each SHA that was built on Travis._
preserved throughout their "lifetime"), the server that handles the artifacts (currently a Node.js
Express server) rejects builds that have already been handled.
_Note: A PR can contain multiple builds; one for each SHA that was built on CircleCI._
7. **Prevent uploaded files from accessing anything outside their directory.**
@ -118,6 +125,11 @@ This section describes how each of the aforementioned sub-tasks is accomplished:
## Assumptions / Things to keep in mind
- Other than the initial webhook trigger, which provides a build number, all requests for data come
from the upload-server making requests to well defined API endpoints (e.g. CircleCI and Github).
This means that any secret access keys need only be stored on the upload-server and not on any of
the CI build infrastructure (e.g. CircleCI).
- Each trusted PR author has full control over the content that is uploaded for their PRs. Part of
the security model relies on the trustworthiness of these authors.
@ -125,14 +137,3 @@ This section describes how each of the aforementioned sub-tasks is accomplished:
the content that is uploaded for the specific PR (e.g. by pushing more commits to it). The user
adding the label is responsible for ensuring that this control is not abused and that the PR is
either closed (one way of another) or the access is revoked.
- If anyone gets access to the `PREVIEW_DEPLOYMENT_TOKEN` (a.k.a. `NGBUILDS_IO_KEY` on
angular/angular) variable generated for each Travis job, they will be able to impersonate the
corresponding PR's author on the preview server for as long as the token is valid (currently 90
mins). Because of this, the value of the `PREVIEW_DEPLOYMENT_TOKEN` should not be made publicly
accessible (e.g. by printing it on the Travis job log).
- Travis does only allow specific whitelisted property names to be used with the JWT addon. The only
known such property at the time is `SAUCE_ACCESS_KEY` (used for integration with SauceLabs). In
order to be able to actually use the JWT addon we had to name the encrypted variable
`SAUCE_ACCESS_KEY` (which we later re-assign to `NGBUILDS_IO_KEY`).

6
aio/aio-builds-setup/docs/vm-setup--create-docker-image.md
View File

@ -1,6 +1,12 @@
# VM setup - Create docker image
## Install node and yarn
- Install [nvm](https://github.com/creationix/nvm#installation).
- Install node.js: `nvm install 8`
- Install yarn: `npm -g install yarn`
## Checkout repository
- `git clone <repo-url>`

30
aio/aio-builds-setup/docs/vm-setup--set-up-secrets.md
View File

@ -12,14 +12,10 @@ Necessary secrets:
- Retrieving members of the trusted GitHub teams.
- Posting comments with preview links on PRs.
2. `PREVIEW_DEPLOYMENT_TOKEN`
2. `CIRCLE_CI_TOKEN`
- Used for:
- Decoding the JWT tokens received with `/create-build` requests.
**Note:**
`TEST_GITHUB_TOKEN` and `TEST_PREVIEW_DEPLOYMENT_TOKEN` can also be created similar to their
non-TEST counterparts and they will be loaded when running `aio-verify-setup`, but it is currently
not clear if/how they can be used in tests.
- Retrieving build information.
- Downloading build artifacts.
## Create secrets
@ -28,25 +24,15 @@ not clear if/how they can be used in tests.
- Visit https://github.com/settings/tokens.
- Generate new token with the `public_repo` scope.
2. `PREVIEW_DEPLOYMENT_TOKEN`
- Just generate a hard-to-guess character sequence.
- Add it to `.travis.yml` under `addons -> jwt -> secure`.
Can be added automatically with: `travis encrypt --add addons.jwt PREVIEW_DEPLOYMENT_TOKEN=<access-key>`
**Note:**
Due to [travis-ci/travis-ci#7223](https://github.com/travis-ci/travis-ci/issues/7223) it is not
currently possible to use the JWT addon (as described above) for anything other than the
`SAUCE_ACCESS_KEY` variable. You can get creative, though...
**WARNING**
TO avoid arbitrary uploads, make sure the `PREVIEW_DEPLOYMENT_TOKEN` is NOT printed in the Travis log.
2. `CIRCLE_CI_TOKEN`
- Visit https://circleci.com/gh/angular/angular/edit#api
- Create an API token with `Build Artifacts` scope
## Save secrets on the VM
- `sudo mkdir /aio-secrets`
- `sudo touch /aio-secrets/GITHUB_TOKEN`
- Insert `<github-token>` into `/aio-secrets/GITHUB_TOKEN`.
- `sudo touch /aio-secrets/PREVIEW_DEPLOYMENT_TOKEN`
- Insert `<access-token>` into `/aio-secrets/PREVIEW_DEPLOYMENT_TOKEN`.
- `sudo touch /aio-secrets/CIRCLE_CI_TOKEN`
- Insert `<access-token>` into `/aio-secrets/CIRCLE_CI_TOKEN`.
- `sudo chmod 400 /aio-secrets/*`

27
aio/aio-builds-setup/docs/vm-setup--start-docker-container.md
View File

@ -13,14 +13,15 @@ sudo docker run \
--publish 80:80 \
--publish 443:443 \
--restart unless-stopped \
[--volume <host-cert-dir>:/etc/ssl/localcerts:ro] \
--volume <host-secrets-dir>:/aio-secrets:ro \
--volume <host-builds-dir>:/var/www/aio-builds \
[--volume <host-cert-dir>:/etc/ssl/localcerts:ro] \
[--volume <host-logs-dir>:/var/log/aio] \
[--volume <host-dockerbuild-dir>:/dockerbuild] \
<name>[:<tag>]
```
Below is the same command with inline comments explaining each option. The aPI docs for `docker run`
Below is the same command with inline comments explaining each option. The API docs for `docker run`
can be found [here](https://docs.docker.com/engine/reference/run/).
```
@ -45,11 +46,6 @@ sudo docker run \
# (This ensures that the container will be automatically started on boot.)
--restart unless-stopped \
# The directory the contains the SSL certificates.
# (See [here](vm-setup--create-host-dirs-and-files.md) for more info.)
# If not provided, the container will use self-signed certificates.
[--volume <host-cert-dir>:/etc/ssl/localcerts:ro] \
# The directory the contains the secrets (e.g. GitHub token, JWT secret, etc).
# (See [here](vm-setup--set-up-secrets.md) for more info.)
--volume <host-secrets-dir>:/aio-secrets:ro \
@ -59,14 +55,23 @@ sudo docker run \
# this will be a directory inside the disk.)
--volume <host-builds-dir>:/var/www/aio-builds \
# The directory the contains the SSL certificates.
# (See [here](vm-setup--create-host-dirs-and-files.md) for more info.)
# If not provided, the container will use self-signed certificates.
[--volume <host-cert-dir>:/etc/ssl/localcerts:ro] \
# The directory where the logs are being kept.
# (See [here](vm-setup--create-host-dirs-and-files.md) for more info.)
# If not provided, the logs will be kept inside the container, which means they will be lost
# whenever a new container is created.
[--volume <host-logs-dir>:/var/log/aio] \
# This directory allows you to share the source scripts between the host and the container when
# debugging. (See [here](misc--debug-docker-container.md) for how to set this up.)
[--volume <host-dockerbuild-dir>:/dockerbuild] \
# The name of the docker image to use (and an optional tag; defaults to `latest`).
# (See [here](vm-setup--create-docker-image.md) for instructions on how to create the iamge.)
# (See [here](vm-setup--create-docker-image.md) for instructions on how to create the image.)
<name>[:<tag>]
```
@ -74,7 +79,8 @@ sudo docker run \
## Example
The following command would start a docker container based on the previously created `foobar-builds`
docker image, alias it as 'foobar-builds-1' and map predefined directories on the host VM to be used
by the container for accessing secrets and SSL certificates and keeping the build artifacts and logs.
by the container for accessing secrets and SSL certificates and keeping the build artifacts and logs;
and will map the source scripts from the host to the container.
```
sudo docker run \
@ -84,9 +90,10 @@ sudo docker run \
--publish 80:80 \
--publish 443:443 \
--restart unless-stopped \
--volume /etc/ssl/localcerts:/etc/ssl/localcerts:ro \
--volume /foobar-secrets:/aio-secrets:ro \
--volume /mnt/disks/foobar-builds:/var/www/aio-builds \
--volume /etc/ssl/localcerts:/etc/ssl/localcerts:ro \
--volume /foobar-logs:/var/log/aio \
--volume ~/angular/aio/aio-builds-setup/dockerbuild:/dockerbuild \
foobar-builds
```