* it's tricky to get out of the runfiles tree with `bazel test` as `BUILD_WORKSPACE_DIRECTORY` is not set but I employed a trick to read the `DO_NOT_BUILD_HERE` file that is one level up from `execroot` and that contains the workspace directory. This is experimental and if `bazel test //:test.debug` fails than `bazel run` is still guaranteed to work as `BUILD_WORKSPACE_DIRECTORY` will be set in that context * test //integration:bazel_test and //integration:bazel-schematics_test exclusively * run "exclusive" and "manual" bazel-in-bazel integration tests in their own CI job as they take 8m+ to execute ``` //integration:bazel-schematics_test PASSED in 317.2s //integration:bazel_test PASSED in 167.8s ``` * Skip all integration tests that are now handled by angular_integration_test except the tests that are tracked for payload size; these are: - cli-hello-world* - hello_world__closure * add & pin @babel deps as newer versions of babel break //packages/localize/src/tools/test:test @babel/core dep had to be pinned to 7.6.4 or else //packages/localize/src/tools/test:test failed. Also //packages/localize uses @babel/generator, @babel/template, @babel/traverse & @babel/types so these deps were added to package.json as they were not being hoisted anymore from @babel/core transitive. NB: integration/hello_world__systemjs_umd test must run with systemjs 0.20.0 NB: systemjs must be at 0.18.10 for legacy saucelabs job to pass NB: With Bazel 2.0, the glob for the files to test `"integration/bazel/**"` is empty if integation/bazel is in .bazelignore. This glob worked under these conditions with 1.1.0. I did not bother testing with 1.2.x as not having integration/bazel in .bazelignore is correct. PR Close #33927
5.6 KiB
Integration tests for Angular
This directory contains end-to-end tests for Angular. Each directory is a self-contained application that exactly mimics how a user might expect Angular to work, so they allow high-fidelity reproductions of real-world issues.
For this to work, we first build the Angular distribution via ./scripts/build-packages-dist.js
, then
install the distribution into each app.
To test Angular CLI applications, we use the cli-hello-world-*
integration tests.
When a significant change is released in the CLI, the applications should be updated with
ng update
:
$ cd integration/cli-hello-world[-*]
$ yarn install
$ yarn ng update @angular/cli @angular-devkit/build-angular
$ yarn build
$ yarn test
Afterwards the @angular/cli
and @angular-devkit/build-angular
should be reverted to the file:../
urls
and the main package.json
should be updated with the new versions.
Render3 tests
The directory cli-hello-world-ivy-compat
contains a test for render3 used with the angular cli.
The cli-hello-world-ivy-minimal
contains a minimal ivy app that is meant to mimic the bazel
equivalent in packages/core/test/bundling/hello_world
, and should be kept similar.
Writing an integration test
The API for each test is:
- Each sub-directory here is an integration test
- Each test should have a
package.json
file - The test runner will run
yarn
andyarn test
on the package
This means that the test should be started by test script, like
"scripts": {"test": "runProgramA && assertResultIsGood"}
Note that the package.json
file uses a special file:../../dist
scheme to reference the Angular
packages, so that the locally-built Angular is installed into the test app.
Also, beware of floating (non-locked) dependencies. If in doubt, you can install the package
directly from file:../../node_modules
.
WARNING
Always ensure that
yarn.lock
files are up-to-date with the correspondingpackage.json
files (wrt the non-local dependencies - i.e. dependencies whose versions do not start withfile:
).You can update a
yarn.lock
file by runningyarn install
in the project subdirectory.
Running integration tests
$ ./integration/run_tests.sh
The test runner will first re-build any stale npm packages, then cd
into each subdirectory to
execute the test.
Running integration tests under Bazel
The PR https://github.com/angular/angular/pull/33927 added the ability to run integration tests with Bazel. These tests can be resource intensive so it is recommended to limit the number of concurrent test jobs with the --local_test_jobs
bazel flag.
Locally, if Bazel uses all of your cores to run the maximum number of integration tests in parallel then this can lead to test timeouts and flakes and freeze up your machine while these tests are running. You can limit the number of concurrent local integration tests that run with:
yarn bazel test --local_test_jobs=<N> //integration/...
Set a reasonable local_test_jobs
limit for your local machine to prevent full cpu utilization during local development test runs.
To avoid having to specify this command line flag, you may want to include it in your .bazelrc.user
file:
test --local_test_jobs=<N>
The downside of this is that this will apply to all tests and not just the resource intensive integration tests.
Bazel-in-bazel integration tests
Two of the integration tests that run Bazel-in-Bazel are particularly resource intensive and are tagged "manual" and "exclusive". To run these tests use,
yarn bazel test //integration:bazel_test
yarn bazel test //integration:bazel-schematics_test
Browser tests
For integration tests we use the puppeteer provisioned version of Chrome. For both Karma and Protractor tests we set a number of browser testing flags. To avoid duplication, they will be listed and explained here and the code will reference this file for more information.
No Sandbox: --no-sandbox
The sandbox needs to be disabled with the --no-sandbox
flag for both Karma and Protractor tests, because it causes Chrome to crash on some environments.
See: http://chromedriver.chromium.org/help/chrome-doesn-t-start See: https://github.com/puppeteer/puppeteer/blob/v1.0.0/docs/troubleshooting.md#chrome-headless-fails-due-to-sandbox-issues
Headless: --headless
So that browsers are not popping up and tearing down when running these tests we run Chrome in headless mode. The --headless
flag puts Chrome in headless mode and a number of other flags are recommended in this mode as well:
--headless
--disable-gpu
--disable-dev-shm-usage
--hide-scrollbars
--mute-audio
These come from the flags that puppeteer passes to chrome when it launches it in headless mode: 18f2ecdffd/lib/Launcher.js (L91)
And from the flags that the Karma ChromeHeadless
browser passes to Chrome: 5f70a76de8/index.js (L171)
Disable shared memory space: --disable-dev-shm-usage
The --disable-dev-shm-usage
flag disables the usage of /dev/shm
because it causes Chrome to crash on some environments.
On CircleCI, the puppeteer provisioned Chrome crashes with CI we get Root cause: org.openqa.selenium.WebDriverException: unknown error: DevToolsActivePort file doesn't exist which resolves
without this flag.
See: https://github.com/puppeteer/puppeteer/blob/v1.0.0/docs/troubleshooting.md#tips See: https://stackoverflow.com/questions/50642308/webdriverexception-unknown-error-devtoolsactiveport-file-doesnt-exist-while-t