angular-cn/integration/README.md

142 lines
6.6 KiB
Markdown
Raw Normal View History

# Integration tests for Angular
build: update lockfiles for integration projects (#33968) In the `integration_test` CircleCI job, we run `yarn install` on all projects in the `integration/` directory. If a project has no lockfile or if the lockfile is out-of-sync with the corresponding `package.json` file, then the installed dependency versions are no longer pinned, which can result in different versions being installed between different runs of the same job (if, for example, a new version is released for a package) and breaks hermeticity. This could be prevented by using the `--frozen-lockfile` option with `yarn install`, but this is not possible with the current setup, because yarn needs to be able to install the locally built Angular packages, whose checksums will be different from the ones in the lockfile. Therefore, we have to manually ensure that the lockfiles remain in-sync with the corresponding `package.json` files for the rest of the dependencies. For example, previously, [cli-hello-world-lazy/yarn.lock][1] had an entry for `@angular-devkit/build-angular@0.900.0-next.9` (pinned to `0.900.0-next.9`), but [cli-hello-world-lazy/package.json][2] specified the `@angular-devkit/build-angular` version as `^0.900.0-rc.0` (note the leading caret). As a result, since the version in the lock file does not much the one in `package.json`, the lockfile is ignored and the latest available version that matches `^0.900.0-rc.0` is installed. This, for example, started causing unrelated CI failures ([example][3]), when `@angular-devkit/build-angular@9.0.0-rc.3` was released with a size improvement. This commit ensures that all integration projects have a lockfile and that lockfiles are up-to-date (with the current `package.json` files). [1]: https://github.com/angular/angular/blob/fc2f6b845/integration/cli-hello-world-lazy/yarn.lock#L13 [2]: https://github.com/angular/angular/blob/fc2f6b845/integration/cli-hello-world-lazy/package.json#L26 [3]: https://circleci.com/gh/angular/angular/535959#tests/containers/2 PR Close #33968
2019-11-21 14:09:33 -05:00
This directory contains end-to-end tests for Angular. Each directory is a self-contained application
that exactly mimics how a user might expect Angular to work, so they allow high-fidelity
reproductions of real-world issues.
For this to work, we first build the Angular distribution via `./scripts/build/build-packages-dist.js`, then
build: update lockfiles for integration projects (#33968) In the `integration_test` CircleCI job, we run `yarn install` on all projects in the `integration/` directory. If a project has no lockfile or if the lockfile is out-of-sync with the corresponding `package.json` file, then the installed dependency versions are no longer pinned, which can result in different versions being installed between different runs of the same job (if, for example, a new version is released for a package) and breaks hermeticity. This could be prevented by using the `--frozen-lockfile` option with `yarn install`, but this is not possible with the current setup, because yarn needs to be able to install the locally built Angular packages, whose checksums will be different from the ones in the lockfile. Therefore, we have to manually ensure that the lockfiles remain in-sync with the corresponding `package.json` files for the rest of the dependencies. For example, previously, [cli-hello-world-lazy/yarn.lock][1] had an entry for `@angular-devkit/build-angular@0.900.0-next.9` (pinned to `0.900.0-next.9`), but [cli-hello-world-lazy/package.json][2] specified the `@angular-devkit/build-angular` version as `^0.900.0-rc.0` (note the leading caret). As a result, since the version in the lock file does not much the one in `package.json`, the lockfile is ignored and the latest available version that matches `^0.900.0-rc.0` is installed. This, for example, started causing unrelated CI failures ([example][3]), when `@angular-devkit/build-angular@9.0.0-rc.3` was released with a size improvement. This commit ensures that all integration projects have a lockfile and that lockfiles are up-to-date (with the current `package.json` files). [1]: https://github.com/angular/angular/blob/fc2f6b845/integration/cli-hello-world-lazy/yarn.lock#L13 [2]: https://github.com/angular/angular/blob/fc2f6b845/integration/cli-hello-world-lazy/package.json#L26 [3]: https://circleci.com/gh/angular/angular/535959#tests/containers/2 PR Close #33968
2019-11-21 14:09:33 -05:00
install the distribution into each app.
build: update lockfiles for integration projects (#33968) In the `integration_test` CircleCI job, we run `yarn install` on all projects in the `integration/` directory. If a project has no lockfile or if the lockfile is out-of-sync with the corresponding `package.json` file, then the installed dependency versions are no longer pinned, which can result in different versions being installed between different runs of the same job (if, for example, a new version is released for a package) and breaks hermeticity. This could be prevented by using the `--frozen-lockfile` option with `yarn install`, but this is not possible with the current setup, because yarn needs to be able to install the locally built Angular packages, whose checksums will be different from the ones in the lockfile. Therefore, we have to manually ensure that the lockfiles remain in-sync with the corresponding `package.json` files for the rest of the dependencies. For example, previously, [cli-hello-world-lazy/yarn.lock][1] had an entry for `@angular-devkit/build-angular@0.900.0-next.9` (pinned to `0.900.0-next.9`), but [cli-hello-world-lazy/package.json][2] specified the `@angular-devkit/build-angular` version as `^0.900.0-rc.0` (note the leading caret). As a result, since the version in the lock file does not much the one in `package.json`, the lockfile is ignored and the latest available version that matches `^0.900.0-rc.0` is installed. This, for example, started causing unrelated CI failures ([example][3]), when `@angular-devkit/build-angular@9.0.0-rc.3` was released with a size improvement. This commit ensures that all integration projects have a lockfile and that lockfiles are up-to-date (with the current `package.json` files). [1]: https://github.com/angular/angular/blob/fc2f6b845/integration/cli-hello-world-lazy/yarn.lock#L13 [2]: https://github.com/angular/angular/blob/fc2f6b845/integration/cli-hello-world-lazy/package.json#L26 [3]: https://circleci.com/gh/angular/angular/535959#tests/containers/2 PR Close #33968
2019-11-21 14:09:33 -05:00
To test Angular CLI applications, we use the `cli-hello-world-*` integration tests.
When a significant change is released in the CLI, the applications should be updated with
`ng update`:
```bash
build: update lockfiles for integration projects (#33968) In the `integration_test` CircleCI job, we run `yarn install` on all projects in the `integration/` directory. If a project has no lockfile or if the lockfile is out-of-sync with the corresponding `package.json` file, then the installed dependency versions are no longer pinned, which can result in different versions being installed between different runs of the same job (if, for example, a new version is released for a package) and breaks hermeticity. This could be prevented by using the `--frozen-lockfile` option with `yarn install`, but this is not possible with the current setup, because yarn needs to be able to install the locally built Angular packages, whose checksums will be different from the ones in the lockfile. Therefore, we have to manually ensure that the lockfiles remain in-sync with the corresponding `package.json` files for the rest of the dependencies. For example, previously, [cli-hello-world-lazy/yarn.lock][1] had an entry for `@angular-devkit/build-angular@0.900.0-next.9` (pinned to `0.900.0-next.9`), but [cli-hello-world-lazy/package.json][2] specified the `@angular-devkit/build-angular` version as `^0.900.0-rc.0` (note the leading caret). As a result, since the version in the lock file does not much the one in `package.json`, the lockfile is ignored and the latest available version that matches `^0.900.0-rc.0` is installed. This, for example, started causing unrelated CI failures ([example][3]), when `@angular-devkit/build-angular@9.0.0-rc.3` was released with a size improvement. This commit ensures that all integration projects have a lockfile and that lockfiles are up-to-date (with the current `package.json` files). [1]: https://github.com/angular/angular/blob/fc2f6b845/integration/cli-hello-world-lazy/yarn.lock#L13 [2]: https://github.com/angular/angular/blob/fc2f6b845/integration/cli-hello-world-lazy/package.json#L26 [3]: https://circleci.com/gh/angular/angular/535959#tests/containers/2 PR Close #33968
2019-11-21 14:09:33 -05:00
$ cd integration/cli-hello-world[-*]
$ yarn install
$ yarn ng update @angular/cli @angular-devkit/build-angular
$ yarn build
$ yarn test
```
Afterwards the `@angular/cli` and `@angular-devkit/build-angular` should be reverted to the `file:../` urls
and the main `package.json` should be updated with the new versions.
## Render3 tests
The directory `cli-hello-world-ivy-compat` contains a test for render3 used with the angular cli.
build: update lockfiles for integration projects (#33968) In the `integration_test` CircleCI job, we run `yarn install` on all projects in the `integration/` directory. If a project has no lockfile or if the lockfile is out-of-sync with the corresponding `package.json` file, then the installed dependency versions are no longer pinned, which can result in different versions being installed between different runs of the same job (if, for example, a new version is released for a package) and breaks hermeticity. This could be prevented by using the `--frozen-lockfile` option with `yarn install`, but this is not possible with the current setup, because yarn needs to be able to install the locally built Angular packages, whose checksums will be different from the ones in the lockfile. Therefore, we have to manually ensure that the lockfiles remain in-sync with the corresponding `package.json` files for the rest of the dependencies. For example, previously, [cli-hello-world-lazy/yarn.lock][1] had an entry for `@angular-devkit/build-angular@0.900.0-next.9` (pinned to `0.900.0-next.9`), but [cli-hello-world-lazy/package.json][2] specified the `@angular-devkit/build-angular` version as `^0.900.0-rc.0` (note the leading caret). As a result, since the version in the lock file does not much the one in `package.json`, the lockfile is ignored and the latest available version that matches `^0.900.0-rc.0` is installed. This, for example, started causing unrelated CI failures ([example][3]), when `@angular-devkit/build-angular@9.0.0-rc.3` was released with a size improvement. This commit ensures that all integration projects have a lockfile and that lockfiles are up-to-date (with the current `package.json` files). [1]: https://github.com/angular/angular/blob/fc2f6b845/integration/cli-hello-world-lazy/yarn.lock#L13 [2]: https://github.com/angular/angular/blob/fc2f6b845/integration/cli-hello-world-lazy/package.json#L26 [3]: https://circleci.com/gh/angular/angular/535959#tests/containers/2 PR Close #33968
2019-11-21 14:09:33 -05:00
The `cli-hello-world-ivy-minimal` contains a minimal ivy app that is meant to mimic the bazel
equivalent in `packages/core/test/bundling/hello_world`, and should be kept similar.
## Writing an integration test
The API for each test is:
- Each sub-directory here is an integration test
- Each test should have a `package.json` file
- The test runner will run `yarn` and `yarn test` on the package
This means that the test should be started by test script, like
```
"scripts": {"test": "runProgramA && assertResultIsGood"}
```
build: update lockfiles for integration projects (#33968) In the `integration_test` CircleCI job, we run `yarn install` on all projects in the `integration/` directory. If a project has no lockfile or if the lockfile is out-of-sync with the corresponding `package.json` file, then the installed dependency versions are no longer pinned, which can result in different versions being installed between different runs of the same job (if, for example, a new version is released for a package) and breaks hermeticity. This could be prevented by using the `--frozen-lockfile` option with `yarn install`, but this is not possible with the current setup, because yarn needs to be able to install the locally built Angular packages, whose checksums will be different from the ones in the lockfile. Therefore, we have to manually ensure that the lockfiles remain in-sync with the corresponding `package.json` files for the rest of the dependencies. For example, previously, [cli-hello-world-lazy/yarn.lock][1] had an entry for `@angular-devkit/build-angular@0.900.0-next.9` (pinned to `0.900.0-next.9`), but [cli-hello-world-lazy/package.json][2] specified the `@angular-devkit/build-angular` version as `^0.900.0-rc.0` (note the leading caret). As a result, since the version in the lock file does not much the one in `package.json`, the lockfile is ignored and the latest available version that matches `^0.900.0-rc.0` is installed. This, for example, started causing unrelated CI failures ([example][3]), when `@angular-devkit/build-angular@9.0.0-rc.3` was released with a size improvement. This commit ensures that all integration projects have a lockfile and that lockfiles are up-to-date (with the current `package.json` files). [1]: https://github.com/angular/angular/blob/fc2f6b845/integration/cli-hello-world-lazy/yarn.lock#L13 [2]: https://github.com/angular/angular/blob/fc2f6b845/integration/cli-hello-world-lazy/package.json#L26 [3]: https://circleci.com/gh/angular/angular/535959#tests/containers/2 PR Close #33968
2019-11-21 14:09:33 -05:00
Note that the `package.json` file uses a special `file:../../dist` scheme to reference the Angular
packages, so that the locally-built Angular is installed into the test app.
Also, beware of floating (non-locked) dependencies. If in doubt, you can install the package
directly from `file:../../node_modules`.
> WARNING
>
> Always ensure that `yarn.lock` files are up-to-date with the corresponding `package.json` files
> (wrt the non-local dependencies - i.e. dependencies whose versions do not start with `file:`).
>
> You can update a `yarn.lock` file by running `yarn install` in the project subdirectory.
## Running integration tests
```
$ ./integration/run_tests.sh
```
build: update lockfiles for integration projects (#33968) In the `integration_test` CircleCI job, we run `yarn install` on all projects in the `integration/` directory. If a project has no lockfile or if the lockfile is out-of-sync with the corresponding `package.json` file, then the installed dependency versions are no longer pinned, which can result in different versions being installed between different runs of the same job (if, for example, a new version is released for a package) and breaks hermeticity. This could be prevented by using the `--frozen-lockfile` option with `yarn install`, but this is not possible with the current setup, because yarn needs to be able to install the locally built Angular packages, whose checksums will be different from the ones in the lockfile. Therefore, we have to manually ensure that the lockfiles remain in-sync with the corresponding `package.json` files for the rest of the dependencies. For example, previously, [cli-hello-world-lazy/yarn.lock][1] had an entry for `@angular-devkit/build-angular@0.900.0-next.9` (pinned to `0.900.0-next.9`), but [cli-hello-world-lazy/package.json][2] specified the `@angular-devkit/build-angular` version as `^0.900.0-rc.0` (note the leading caret). As a result, since the version in the lock file does not much the one in `package.json`, the lockfile is ignored and the latest available version that matches `^0.900.0-rc.0` is installed. This, for example, started causing unrelated CI failures ([example][3]), when `@angular-devkit/build-angular@9.0.0-rc.3` was released with a size improvement. This commit ensures that all integration projects have a lockfile and that lockfiles are up-to-date (with the current `package.json` files). [1]: https://github.com/angular/angular/blob/fc2f6b845/integration/cli-hello-world-lazy/yarn.lock#L13 [2]: https://github.com/angular/angular/blob/fc2f6b845/integration/cli-hello-world-lazy/package.json#L26 [3]: https://circleci.com/gh/angular/angular/535959#tests/containers/2 PR Close #33968
2019-11-21 14:09:33 -05:00
The test runner will first re-build any stale npm packages, then `cd` into each subdirectory to
execute the test.
test: use puppeteer in integration tests and to download correct chromedriver (#35049) This means integration tests no longer need to depend on a $CI_CHROMEDRIVER_VERSION_ARG environment variable to specify which chromedriver version to download to match the locally installed chrome. This was bad DX and not having it specified was not reliable as webdriver-manager would not always download the chromedriver version to work with the locally installed chrome. webdriver-manager update --gecko=false --standalone=false $CI_CHROMEDRIVER_VERSION_ARG is now replaced with node webdriver-manager-update.js in the root package.json, which checks which version of chrome puppeteer has come bundled with & downloads informs webdriver-manager to download the corresponding chrome driver version. Integration tests now use "webdriver-manager": "file:../../node_modules/webdriver-manager" so they don't have to waste time calling webdriver-manager update in postinstall "// resolutions": "Ensure a single version of webdriver-manager which comes from root node_modules that has already run webdriver-manager update", "resolutions": { "**/webdriver-manager": "file:../../node_modules/webdriver-manager" } This should speed up each integration postinstall by a few seconds. Further, integration test package.json files link puppeteer via file:../../node_modules/puppeteer which is the ideal situation as the puppeteer post-install won't download chrome if it is already downloaded. In CI, since node_modules is cached it should not need to download Chrome either unless the node_modules cache is busted. NB: each version of puppeteer comes bundles with a specific version of chrome. Root package.json & yarn.lock currently pull down puppeteer 2.1.0 which comes with chrome 80. See https://github.com/puppeteer/puppeteer#q-which-chromium-version-does-puppeteer-use for more info. Only two references to CI_CHROMEDRIVER_VERSION_ARG left in integration tests at integration/bazel-schematics/test.sh which I'm not entirely sure how to get rid of it Use a lightweight puppeteer=>chrome version mapping instead of launching chrome and calling browser.version() Launching puppeteer headless chrome and calling browser.version() was a heavy-handed approach to determine the Chrome version. A small and easy to update mappings file is a better solution and it means that the `yarn install` step does not require chrome shared libs available on the system for its postinstall step PR Close #35049
2020-01-31 18:50:44 -05:00
build: add npm_integration_test && angular_integration_test (#33927) * it's tricky to get out of the runfiles tree with `bazel test` as `BUILD_WORKSPACE_DIRECTORY` is not set but I employed a trick to read the `DO_NOT_BUILD_HERE` file that is one level up from `execroot` and that contains the workspace directory. This is experimental and if `bazel test //:test.debug` fails than `bazel run` is still guaranteed to work as `BUILD_WORKSPACE_DIRECTORY` will be set in that context * test //integration:bazel_test and //integration:bazel-schematics_test exclusively * run "exclusive" and "manual" bazel-in-bazel integration tests in their own CI job as they take 8m+ to execute ``` //integration:bazel-schematics_test PASSED in 317.2s //integration:bazel_test PASSED in 167.8s ``` * Skip all integration tests that are now handled by angular_integration_test except the tests that are tracked for payload size; these are: - cli-hello-world* - hello_world__closure * add & pin @babel deps as newer versions of babel break //packages/localize/src/tools/test:test @babel/core dep had to be pinned to 7.6.4 or else //packages/localize/src/tools/test:test failed. Also //packages/localize uses @babel/generator, @babel/template, @babel/traverse & @babel/types so these deps were added to package.json as they were not being hoisted anymore from @babel/core transitive. NB: integration/hello_world__systemjs_umd test must run with systemjs 0.20.0 NB: systemjs must be at 0.18.10 for legacy saucelabs job to pass NB: With Bazel 2.0, the glob for the files to test `"integration/bazel/**"` is empty if integation/bazel is in .bazelignore. This glob worked under these conditions with 1.1.0. I did not bother testing with 1.2.x as not having integration/bazel in .bazelignore is correct. PR Close #33927
2020-02-04 14:45:40 -05:00
## Running integration tests under Bazel
The PR https://github.com/angular/angular/pull/33927 added the ability to run integration tests with Bazel. These tests can be resource intensive so it is recommended to limit the number of concurrent test jobs with the `--local_test_jobs` bazel flag.
Locally, if Bazel uses all of your cores to run the maximum number of integration tests in parallel then this can lead to test timeouts and flakes and freeze up your machine while these tests are running. You can limit the number of concurrent local integration tests that run with:
```
yarn bazel test --local_test_jobs=<N> //integration/...
```
Set a reasonable `local_test_jobs` limit for your local machine to prevent full cpu utilization during local development test runs.
To avoid having to specify this command line flag, you may want to include it in your `.bazelrc.user` file:
```
test --local_test_jobs=<N>
```
The downside of this is that this will apply to all tests and not just the resource intensive integration tests.
### Bazel-in-bazel integration tests
Two of the integration tests that run Bazel-in-Bazel are particularly resource intensive and are tagged "manual" and "exclusive". To run these tests use,
```
yarn bazel test //integration:bazel_test
yarn bazel test //integration:bazel-schematics_test
```
## Adding a new integration test
When adding a new integration test, follow the steps below to add a bazel test target for the new test.
1. Add new test to `INTEGRATION_TESTS` object in `/integration/BUILD.bazel` (and tag as `"no-ivy-aot"` if not meant to be run against ivy bundles). *NB: if the test requires any special attribute then make a new angular_integration_test target instead.*
2. If test requires ports and does not support ethereal ports then make sure the port is unique and add it to the "manually configured ports" comment to document which port it is using
3. Add at least the following two entries `.bazelignore` (as they may contain BUILD files)
1. `integration/new_test/node_modules`
2. `integration/new_test/.yarn_local_cache`
4. Add any other untracked folders to `.bazelignore` that may contain `BUILD` files
5. If there are tracked BUILD files in the integration test folder (`integration/bazel` has these for example) add those folders to the `build --deleted_packages` and `query --deleted_packages` lines in `.bazelrc`
test: use puppeteer in integration tests and to download correct chromedriver (#35049) This means integration tests no longer need to depend on a $CI_CHROMEDRIVER_VERSION_ARG environment variable to specify which chromedriver version to download to match the locally installed chrome. This was bad DX and not having it specified was not reliable as webdriver-manager would not always download the chromedriver version to work with the locally installed chrome. webdriver-manager update --gecko=false --standalone=false $CI_CHROMEDRIVER_VERSION_ARG is now replaced with node webdriver-manager-update.js in the root package.json, which checks which version of chrome puppeteer has come bundled with & downloads informs webdriver-manager to download the corresponding chrome driver version. Integration tests now use "webdriver-manager": "file:../../node_modules/webdriver-manager" so they don't have to waste time calling webdriver-manager update in postinstall "// resolutions": "Ensure a single version of webdriver-manager which comes from root node_modules that has already run webdriver-manager update", "resolutions": { "**/webdriver-manager": "file:../../node_modules/webdriver-manager" } This should speed up each integration postinstall by a few seconds. Further, integration test package.json files link puppeteer via file:../../node_modules/puppeteer which is the ideal situation as the puppeteer post-install won't download chrome if it is already downloaded. In CI, since node_modules is cached it should not need to download Chrome either unless the node_modules cache is busted. NB: each version of puppeteer comes bundles with a specific version of chrome. Root package.json & yarn.lock currently pull down puppeteer 2.1.0 which comes with chrome 80. See https://github.com/puppeteer/puppeteer#q-which-chromium-version-does-puppeteer-use for more info. Only two references to CI_CHROMEDRIVER_VERSION_ARG left in integration tests at integration/bazel-schematics/test.sh which I'm not entirely sure how to get rid of it Use a lightweight puppeteer=>chrome version mapping instead of launching chrome and calling browser.version() Launching puppeteer headless chrome and calling browser.version() was a heavy-handed approach to determine the Chrome version. A small and easy to update mappings file is a better solution and it means that the `yarn install` step does not require chrome shared libs available on the system for its postinstall step PR Close #35049
2020-01-31 18:50:44 -05:00
## Browser tests
For integration tests we use the puppeteer provisioned version of Chrome. For both Karma and Protractor tests we set a number of browser testing flags. To avoid duplication, they will be listed and explained here and the code will reference this file for more information.
### No Sandbox: --no-sandbox
The sandbox needs to be disabled with the `--no-sandbox` flag for both Karma and Protractor tests, because it causes Chrome to crash on some environments.
See: http://chromedriver.chromium.org/help/chrome-doesn-t-start
See: https://github.com/puppeteer/puppeteer/blob/v1.0.0/docs/troubleshooting.md#chrome-headless-fails-due-to-sandbox-issues
### Headless: --headless
So that browsers are not popping up and tearing down when running these tests we run Chrome in headless mode. The `--headless` flag puts Chrome in headless mode and a number of other flags are recommended in this mode as well:
* `--headless`
* `--disable-gpu`
* `--disable-dev-shm-usage`
* `--hide-scrollbars`
* `--mute-audio`
These come from the flags that puppeteer passes to chrome when it launches it in headless mode: https://github.com/puppeteer/puppeteer/blob/18f2ecdffdfc70e891750b570bfe8bea5b5ca8c2/lib/Launcher.js#L91
And from the flags that the Karma `ChromeHeadless` browser passes to Chrome: https://github.com/karma-runner/karma-chrome-launcher/blob/5f70a76de87ecbb57f3f3cb556aa6a2a1a4f643f/index.js#L171
#### Disable shared memory space: --disable-dev-shm-usage
The `--disable-dev-shm-usage` flag disables the usage of `/dev/shm` because it causes Chrome to crash on some environments.
On CircleCI, the puppeteer provisioned Chrome crashes with `CI we get Root cause: org.openqa.selenium.WebDriverException: unknown error: DevToolsActivePort file doesn't exist which resolves` without this flag.
See: https://github.com/puppeteer/puppeteer/blob/v1.0.0/docs/troubleshooting.md#tips
See: https://stackoverflow.com/questions/50642308/webdriverexception-unknown-error-devtoolsactiveport-file-doesnt-exist-while-t