2017-02-09 12:06:19 -05:00
# Integration tests for Angular
build: update lockfiles for integration projects (#33968)
In the `integration_test` CircleCI job, we run `yarn install` on all
projects in the `integration/` directory. If a project has no lockfile
or if the lockfile is out-of-sync with the corresponding `package.json`
file, then the installed dependency versions are no longer pinned, which
can result in different versions being installed between different runs
of the same job (if, for example, a new version is released for a
package) and breaks hermeticity.
This could be prevented by using the `--frozen-lockfile` option with
`yarn install`, but this is not possible with the current setup, because
yarn needs to be able to install the locally built Angular packages,
whose checksums will be different from the ones in the lockfile.
Therefore, we have to manually ensure that the lockfiles remain in-sync
with the corresponding `package.json` files for the rest of the
dependencies.
For example, previously, [cli-hello-world-lazy/yarn.lock][1] had an
entry for `@angular-devkit/build-angular@0.900.0-next.9` (pinned to
`0.900.0-next.9`), but [cli-hello-world-lazy/package.json][2] specified
the `@angular-devkit/build-angular` version as `^0.900.0-rc.0` (note the
leading caret). As a result, since the version in the lock file does not
much the one in `package.json`, the lockfile is ignored and the latest
available version that matches `^0.900.0-rc.0` is installed.
This, for example, started causing unrelated CI failures ([example][3]),
when `@angular-devkit/build-angular@9.0.0-rc.3` was released with a size
improvement.
This commit ensures that all integration projects have a lockfile and
that lockfiles are up-to-date (with the current `package.json` files).
[1]: https://github.com/angular/angular/blob/fc2f6b845/integration/cli-hello-world-lazy/yarn.lock#L13
[2]: https://github.com/angular/angular/blob/fc2f6b845/integration/cli-hello-world-lazy/package.json#L26
[3]: https://circleci.com/gh/angular/angular/535959#tests/containers/2
PR Close #33968
2019-11-21 14:09:33 -05:00
This directory contains end-to-end tests for Angular. Each directory is a self-contained application
that exactly mimics how a user might expect Angular to work, so they allow high-fidelity
reproductions of real-world issues.
2018-01-16 10:09:59 -05:00
2019-11-22 19:31:40 -05:00
For this to work, we first build the Angular distribution via `./scripts/build-packages-dist.js` , then
build: update lockfiles for integration projects (#33968)
In the `integration_test` CircleCI job, we run `yarn install` on all
projects in the `integration/` directory. If a project has no lockfile
or if the lockfile is out-of-sync with the corresponding `package.json`
file, then the installed dependency versions are no longer pinned, which
can result in different versions being installed between different runs
of the same job (if, for example, a new version is released for a
package) and breaks hermeticity.
This could be prevented by using the `--frozen-lockfile` option with
`yarn install`, but this is not possible with the current setup, because
yarn needs to be able to install the locally built Angular packages,
whose checksums will be different from the ones in the lockfile.
Therefore, we have to manually ensure that the lockfiles remain in-sync
with the corresponding `package.json` files for the rest of the
dependencies.
For example, previously, [cli-hello-world-lazy/yarn.lock][1] had an
entry for `@angular-devkit/build-angular@0.900.0-next.9` (pinned to
`0.900.0-next.9`), but [cli-hello-world-lazy/package.json][2] specified
the `@angular-devkit/build-angular` version as `^0.900.0-rc.0` (note the
leading caret). As a result, since the version in the lock file does not
much the one in `package.json`, the lockfile is ignored and the latest
available version that matches `^0.900.0-rc.0` is installed.
This, for example, started causing unrelated CI failures ([example][3]),
when `@angular-devkit/build-angular@9.0.0-rc.3` was released with a size
improvement.
This commit ensures that all integration projects have a lockfile and
that lockfiles are up-to-date (with the current `package.json` files).
[1]: https://github.com/angular/angular/blob/fc2f6b845/integration/cli-hello-world-lazy/yarn.lock#L13
[2]: https://github.com/angular/angular/blob/fc2f6b845/integration/cli-hello-world-lazy/package.json#L26
[3]: https://circleci.com/gh/angular/angular/535959#tests/containers/2
PR Close #33968
2019-11-21 14:09:33 -05:00
install the distribution into each app.
2018-01-16 10:09:59 -05:00
build: update lockfiles for integration projects (#33968)
In the `integration_test` CircleCI job, we run `yarn install` on all
projects in the `integration/` directory. If a project has no lockfile
or if the lockfile is out-of-sync with the corresponding `package.json`
file, then the installed dependency versions are no longer pinned, which
can result in different versions being installed between different runs
of the same job (if, for example, a new version is released for a
package) and breaks hermeticity.
This could be prevented by using the `--frozen-lockfile` option with
`yarn install`, but this is not possible with the current setup, because
yarn needs to be able to install the locally built Angular packages,
whose checksums will be different from the ones in the lockfile.
Therefore, we have to manually ensure that the lockfiles remain in-sync
with the corresponding `package.json` files for the rest of the
dependencies.
For example, previously, [cli-hello-world-lazy/yarn.lock][1] had an
entry for `@angular-devkit/build-angular@0.900.0-next.9` (pinned to
`0.900.0-next.9`), but [cli-hello-world-lazy/package.json][2] specified
the `@angular-devkit/build-angular` version as `^0.900.0-rc.0` (note the
leading caret). As a result, since the version in the lock file does not
much the one in `package.json`, the lockfile is ignored and the latest
available version that matches `^0.900.0-rc.0` is installed.
This, for example, started causing unrelated CI failures ([example][3]),
when `@angular-devkit/build-angular@9.0.0-rc.3` was released with a size
improvement.
This commit ensures that all integration projects have a lockfile and
that lockfiles are up-to-date (with the current `package.json` files).
[1]: https://github.com/angular/angular/blob/fc2f6b845/integration/cli-hello-world-lazy/yarn.lock#L13
[2]: https://github.com/angular/angular/blob/fc2f6b845/integration/cli-hello-world-lazy/package.json#L26
[3]: https://circleci.com/gh/angular/angular/535959#tests/containers/2
PR Close #33968
2019-11-21 14:09:33 -05:00
To test Angular CLI applications, we use the `cli-hello-world-*` integration tests.
When a significant change is released in the CLI, the applications should be updated with
`ng update` :
2018-01-16 10:09:59 -05:00
```bash
build: update lockfiles for integration projects (#33968)
In the `integration_test` CircleCI job, we run `yarn install` on all
projects in the `integration/` directory. If a project has no lockfile
or if the lockfile is out-of-sync with the corresponding `package.json`
file, then the installed dependency versions are no longer pinned, which
can result in different versions being installed between different runs
of the same job (if, for example, a new version is released for a
package) and breaks hermeticity.
This could be prevented by using the `--frozen-lockfile` option with
`yarn install`, but this is not possible with the current setup, because
yarn needs to be able to install the locally built Angular packages,
whose checksums will be different from the ones in the lockfile.
Therefore, we have to manually ensure that the lockfiles remain in-sync
with the corresponding `package.json` files for the rest of the
dependencies.
For example, previously, [cli-hello-world-lazy/yarn.lock][1] had an
entry for `@angular-devkit/build-angular@0.900.0-next.9` (pinned to
`0.900.0-next.9`), but [cli-hello-world-lazy/package.json][2] specified
the `@angular-devkit/build-angular` version as `^0.900.0-rc.0` (note the
leading caret). As a result, since the version in the lock file does not
much the one in `package.json`, the lockfile is ignored and the latest
available version that matches `^0.900.0-rc.0` is installed.
This, for example, started causing unrelated CI failures ([example][3]),
when `@angular-devkit/build-angular@9.0.0-rc.3` was released with a size
improvement.
This commit ensures that all integration projects have a lockfile and
that lockfiles are up-to-date (with the current `package.json` files).
[1]: https://github.com/angular/angular/blob/fc2f6b845/integration/cli-hello-world-lazy/yarn.lock#L13
[2]: https://github.com/angular/angular/blob/fc2f6b845/integration/cli-hello-world-lazy/package.json#L26
[3]: https://circleci.com/gh/angular/angular/535959#tests/containers/2
PR Close #33968
2019-11-21 14:09:33 -05:00
$ cd integration/cli-hello-world[-*]
2018-10-31 11:25:57 -04:00
$ yarn install
$ yarn ng update @angular/cli @angular -devkit/build-angular
2019-11-22 19:31:40 -05:00
$ yarn build
$ yarn test
2018-01-16 10:09:59 -05:00
```
2017-02-09 12:06:19 -05:00
2019-11-22 19:31:40 -05:00
Afterwards the `@angular/cli` and `@angular-devkit/build-angular` should be reverted to the `file:../` urls
and the main `package.json` should be updated with the new versions.
2018-01-25 19:43:29 -05:00
## Render3 tests
2019-01-30 09:02:53 -05:00
The directory `cli-hello-world-ivy-compat` contains a test for render3 used with the angular cli.
2018-01-25 19:43:29 -05:00
build: update lockfiles for integration projects (#33968)
In the `integration_test` CircleCI job, we run `yarn install` on all
projects in the `integration/` directory. If a project has no lockfile
or if the lockfile is out-of-sync with the corresponding `package.json`
file, then the installed dependency versions are no longer pinned, which
can result in different versions being installed between different runs
of the same job (if, for example, a new version is released for a
package) and breaks hermeticity.
This could be prevented by using the `--frozen-lockfile` option with
`yarn install`, but this is not possible with the current setup, because
yarn needs to be able to install the locally built Angular packages,
whose checksums will be different from the ones in the lockfile.
Therefore, we have to manually ensure that the lockfiles remain in-sync
with the corresponding `package.json` files for the rest of the
dependencies.
For example, previously, [cli-hello-world-lazy/yarn.lock][1] had an
entry for `@angular-devkit/build-angular@0.900.0-next.9` (pinned to
`0.900.0-next.9`), but [cli-hello-world-lazy/package.json][2] specified
the `@angular-devkit/build-angular` version as `^0.900.0-rc.0` (note the
leading caret). As a result, since the version in the lock file does not
much the one in `package.json`, the lockfile is ignored and the latest
available version that matches `^0.900.0-rc.0` is installed.
This, for example, started causing unrelated CI failures ([example][3]),
when `@angular-devkit/build-angular@9.0.0-rc.3` was released with a size
improvement.
This commit ensures that all integration projects have a lockfile and
that lockfiles are up-to-date (with the current `package.json` files).
[1]: https://github.com/angular/angular/blob/fc2f6b845/integration/cli-hello-world-lazy/yarn.lock#L13
[2]: https://github.com/angular/angular/blob/fc2f6b845/integration/cli-hello-world-lazy/package.json#L26
[3]: https://circleci.com/gh/angular/angular/535959#tests/containers/2
PR Close #33968
2019-11-21 14:09:33 -05:00
The `cli-hello-world-ivy-minimal` contains a minimal ivy app that is meant to mimic the bazel
2019-01-28 10:04:38 -05:00
equivalent in `packages/core/test/bundling/hello_world` , and should be kept similar.
2017-02-09 12:06:19 -05:00
## Writing an integration test
The API for each test is:
- Each sub-directory here is an integration test
- Each test should have a `package.json` file
- The test runner will run `yarn` and `yarn test` on the package
This means that the test should be started by test script, like
```
2018-10-31 11:25:57 -04:00
"scripts": {"test": "runProgramA & & assertResultIsGood"}
2017-02-09 12:06:19 -05:00
```
build: update lockfiles for integration projects (#33968)
In the `integration_test` CircleCI job, we run `yarn install` on all
projects in the `integration/` directory. If a project has no lockfile
or if the lockfile is out-of-sync with the corresponding `package.json`
file, then the installed dependency versions are no longer pinned, which
can result in different versions being installed between different runs
of the same job (if, for example, a new version is released for a
package) and breaks hermeticity.
This could be prevented by using the `--frozen-lockfile` option with
`yarn install`, but this is not possible with the current setup, because
yarn needs to be able to install the locally built Angular packages,
whose checksums will be different from the ones in the lockfile.
Therefore, we have to manually ensure that the lockfiles remain in-sync
with the corresponding `package.json` files for the rest of the
dependencies.
For example, previously, [cli-hello-world-lazy/yarn.lock][1] had an
entry for `@angular-devkit/build-angular@0.900.0-next.9` (pinned to
`0.900.0-next.9`), but [cli-hello-world-lazy/package.json][2] specified
the `@angular-devkit/build-angular` version as `^0.900.0-rc.0` (note the
leading caret). As a result, since the version in the lock file does not
much the one in `package.json`, the lockfile is ignored and the latest
available version that matches `^0.900.0-rc.0` is installed.
This, for example, started causing unrelated CI failures ([example][3]),
when `@angular-devkit/build-angular@9.0.0-rc.3` was released with a size
improvement.
This commit ensures that all integration projects have a lockfile and
that lockfiles are up-to-date (with the current `package.json` files).
[1]: https://github.com/angular/angular/blob/fc2f6b845/integration/cli-hello-world-lazy/yarn.lock#L13
[2]: https://github.com/angular/angular/blob/fc2f6b845/integration/cli-hello-world-lazy/package.json#L26
[3]: https://circleci.com/gh/angular/angular/535959#tests/containers/2
PR Close #33968
2019-11-21 14:09:33 -05:00
Note that the `package.json` file uses a special `file:../../dist` scheme to reference the Angular
packages, so that the locally-built Angular is installed into the test app.
Also, beware of floating (non-locked) dependencies. If in doubt, you can install the package
directly from `file:../../node_modules` .
> WARNING
>
> Always ensure that `yarn.lock` files are up-to-date with the corresponding `package.json` files
> (wrt the non-local dependencies - i.e. dependencies whose versions do not start with `file:`).
>
> You can update a `yarn.lock` file by running `yarn install` in the project subdirectory.
2017-02-09 12:06:19 -05:00
## Running integration tests
```
$ ./integration/run_tests.sh
```
2018-03-15 18:36:11 -04:00
build: update lockfiles for integration projects (#33968)
In the `integration_test` CircleCI job, we run `yarn install` on all
projects in the `integration/` directory. If a project has no lockfile
or if the lockfile is out-of-sync with the corresponding `package.json`
file, then the installed dependency versions are no longer pinned, which
can result in different versions being installed between different runs
of the same job (if, for example, a new version is released for a
package) and breaks hermeticity.
This could be prevented by using the `--frozen-lockfile` option with
`yarn install`, but this is not possible with the current setup, because
yarn needs to be able to install the locally built Angular packages,
whose checksums will be different from the ones in the lockfile.
Therefore, we have to manually ensure that the lockfiles remain in-sync
with the corresponding `package.json` files for the rest of the
dependencies.
For example, previously, [cli-hello-world-lazy/yarn.lock][1] had an
entry for `@angular-devkit/build-angular@0.900.0-next.9` (pinned to
`0.900.0-next.9`), but [cli-hello-world-lazy/package.json][2] specified
the `@angular-devkit/build-angular` version as `^0.900.0-rc.0` (note the
leading caret). As a result, since the version in the lock file does not
much the one in `package.json`, the lockfile is ignored and the latest
available version that matches `^0.900.0-rc.0` is installed.
This, for example, started causing unrelated CI failures ([example][3]),
when `@angular-devkit/build-angular@9.0.0-rc.3` was released with a size
improvement.
This commit ensures that all integration projects have a lockfile and
that lockfiles are up-to-date (with the current `package.json` files).
[1]: https://github.com/angular/angular/blob/fc2f6b845/integration/cli-hello-world-lazy/yarn.lock#L13
[2]: https://github.com/angular/angular/blob/fc2f6b845/integration/cli-hello-world-lazy/package.json#L26
[3]: https://circleci.com/gh/angular/angular/535959#tests/containers/2
PR Close #33968
2019-11-21 14:09:33 -05:00
The test runner will first re-build any stale npm packages, then `cd` into each subdirectory to
execute the test.
2020-01-31 18:50:44 -05:00
## Browser tests
For integration tests we use the puppeteer provisioned version of Chrome. For both Karma and Protractor tests we set a number of browser testing flags. To avoid duplication, they will be listed and explained here and the code will reference this file for more information.
### No Sandbox: --no-sandbox
The sandbox needs to be disabled with the `--no-sandbox` flag for both Karma and Protractor tests, because it causes Chrome to crash on some environments.
See: http://chromedriver.chromium.org/help/chrome-doesn-t-start
See: https://github.com/puppeteer/puppeteer/blob/v1.0.0/docs/troubleshooting.md#chrome-headless-fails-due-to-sandbox-issues
### Headless: --headless
So that browsers are not popping up and tearing down when running these tests we run Chrome in headless mode. The `--headless` flag puts Chrome in headless mode and a number of other flags are recommended in this mode as well:
* `--headless`
* `--disable-gpu`
* `--disable-dev-shm-usage`
* `--hide-scrollbars`
* `--mute-audio`
These come from the flags that puppeteer passes to chrome when it launches it in headless mode: https://github.com/puppeteer/puppeteer/blob/18f2ecdffdfc70e891750b570bfe8bea5b5ca8c2/lib/Launcher.js#L91
And from the flags that the Karma `ChromeHeadless` browser passes to Chrome: https://github.com/karma-runner/karma-chrome-launcher/blob/5f70a76de87ecbb57f3f3cb556aa6a2a1a4f643f/index.js#L171
#### Disable shared memory space: --disable-dev-shm-usage
The `--disable-dev-shm-usage` flag disables the usage of `/dev/shm` because it causes Chrome to crash on some environments.
On CircleCI, the puppeteer provisioned Chrome crashes with `CI we get Root cause: org.openqa.selenium.WebDriverException: unknown error: DevToolsActivePort file doesn't exist which resolves` without this flag.
See: https://github.com/puppeteer/puppeteer/blob/v1.0.0/docs/troubleshooting.md#tips
See: https://stackoverflow.com/questions/50642308/webdriverexception-unknown-error-devtoolsactiveport-file-doesnt-exist-while-t
