236 lines
9.6 KiB
Markdown
236 lines
9.6 KiB
Markdown
# Compliance test-cases
|
|
|
|
This directory contains rules, helpers and test-cases for the Angular compiler compliance tests.
|
|
|
|
There are three different types of tests that are run based on file-based "test-cases".
|
|
|
|
* **Full compile** - in this test the source files defined by the test-case are fully compiled by Angular.
|
|
The generated files are compared to "expected files" via a matching algorithm that is tolerant to
|
|
whitespace and variable name changes.
|
|
* **Partial compile** - in this test the source files defined by the test-case are "partially" compiled by
|
|
Angular to produce files that can be published. These partially compiled files are compared directly
|
|
against "golden files" to ensure that we do not inadvertently break the public API of partial
|
|
declarations.
|
|
* **Linked** - in this test the golden files mentioned in the previous bullet point, are passed to the
|
|
Angular linker, which generates files that are comparable to the fully compiled files. These linked
|
|
files are compared against the "expected files" in the same way as in the "full compile" tests.
|
|
|
|
This way the compliance tests are able to check each mode and stage of compilation is accurate and does
|
|
not change unexpectedly.
|
|
|
|
|
|
## Defining a test-case
|
|
|
|
To define a test-case, create a new directory below `test_cases`. In this directory
|
|
|
|
* add a new file called `TEST_CASES.json`. The format of this file is described below.
|
|
* add an empty `GOLDEN_PARTIAL.js` file. This file will be updated by the tooling later.
|
|
* add any `inputFiles` that will be compiled as part of the test-case.
|
|
* add any `expected` files that will be compared to the files generated by compiling the source files.
|
|
|
|
|
|
### TEST_CASES.json format
|
|
|
|
The `TEST_CASES.json` defines an object with one or more test-case definitions in the `cases` property.
|
|
|
|
Each test-case can specify:
|
|
|
|
* A `description` of the test.
|
|
* The `inputFiles` that will be compiled.
|
|
* Additional `compilerOptions` and `angularCompilerOptions` that are passed to the compiler.
|
|
* Whether to exclude this test-case from certain tests running under certain compilation modes (`compilationModeFilter`).
|
|
* A collection of `expectations` definitions that will be checked against the generated files.
|
|
|
|
Note that there is a JSON schema for the `TEST_CASES.json` file stored at `test_cases/test_case_schema.json`.
|
|
You should add a link to this schema at the top of your `TEST_CASES.json` file to provide
|
|
validation and intellisense for this file in your IDE.
|
|
|
|
For example:
|
|
|
|
```json
|
|
{
|
|
"$schema": "../test_case_schema.json",
|
|
"cases": [
|
|
{
|
|
"description": "description of the test - equivalent to an `it` clause message.",
|
|
"inputFiles": ["abc.ts"],
|
|
"expectations": [
|
|
{
|
|
"failureMessage": "message to display if this expectation fails",
|
|
"files": [
|
|
{ "expected": "xyz.js", "generated": "abc.js" }, ...
|
|
]
|
|
}, ...
|
|
],
|
|
"compilerOptions": { ... },
|
|
"angularCompilerOptions": { ... }
|
|
}
|
|
]
|
|
}
|
|
```
|
|
|
|
### Input files
|
|
|
|
The input files are the source file that will be compiled as part of this test-case.
|
|
Input files should be stored in the directory next to the `TEST_CASES.json`.
|
|
The paths to the input files should be listed in the `inputFiles` property of `TEST_CASES.json`.
|
|
The paths are relative to the `TEST_CASES.json` file.
|
|
|
|
If no `inputFiles` property is provided, the default is `["test.ts"]`.
|
|
|
|
Note that test-cases can share input files, but you should only do this if these input files are
|
|
going to be compiled using the same options. This is because only one version of the compiled input
|
|
file is retrieved from the golden partial file to be used in the linker tests. This can cause the
|
|
linker tests to fail if they are provided with a compiled file (from the golden partial) that was
|
|
compiled with different options to what are expected for that test-case.
|
|
|
|
|
|
### Expectations
|
|
|
|
An expectation consists of a `failureMessage`, which is displayed if the expectation check fails,
|
|
a collection of expected `files` pairs and/or a collection of `expectedErrors`.
|
|
|
|
Each expected file-pair consists of a path to a `generated` file (relative to the build output folder),
|
|
and a path to an `expected` file (relative to the test case).
|
|
|
|
The `generated` file is checked to see if it "matches" the `expected` file. The matching is
|
|
resilient to whitespace and variable name changes.
|
|
|
|
If no `files` property is provided, the default is a a collection of objects `{expected, generated}`,
|
|
where `expected` and `generated` are computed by taking each path in the `inputFiles` collection
|
|
and replacing the `.ts` extension with `.js`.
|
|
|
|
Each expected error must have a `message` property and, optionally, a `location` property. These are
|
|
parsed as regular expressions (so `.` and `(` etc must be escaped) and tested against the errors that
|
|
are returned as diagnostics from the compilation.
|
|
|
|
If no `failureMessage` property is provided, the default is `"Incorrect generated output."`.
|
|
|
|
|
|
### Expected file format
|
|
|
|
The expected files look like JavaScript but are actually specially formatted to allow matching
|
|
with the generated output. The generated and expected files are tokenized and then the tokens
|
|
are intelligently matched to check whether they are equivalent.
|
|
|
|
* Whitespace tolerant - the tokens can be separated by any amount of whitespace
|
|
* Code skipping - you can skip sections of code in the generated output by adding an ellipsis
|
|
(…) to the expectation file.
|
|
* Identifier tolerant - identifiers in the expectation file that start and end with a dollar
|
|
(e.g. `$r3$`) will be matched against any identifier. But the matching will ensure that the
|
|
same identifier name appears consistently elsewhere in the file.
|
|
* Macro expansion - we can add macros to the expected files that will be expanded to blocks
|
|
of code dynamically. The following macros are defined in the
|
|
`test_helpers/expected_file_macros.ts` file:
|
|
* I18n messages - for example:
|
|
`__i18nMsg__('message string', [ ['placeholder', 'pair] ], { meta: 'properties'})`.
|
|
* Attribute markers - for example: `__AttributeMarker.Bindings__`.
|
|
|
|
### Source-map checks
|
|
|
|
To check a mapping, add a `// SOURCE:` comment to the end of a line in an expectation file:
|
|
|
|
```
|
|
<generated code> // SOURCE: "<source-url>" <source code>
|
|
```
|
|
|
|
The generated code, stripped of the `// SOURCE: ` comment, will still be checked as normal by the
|
|
`expectEmit()` helper. But, prior to that, the source-map segments are checked to ensure that there
|
|
is a mapping from `<generated code>` to `<source code>` found in the file at `<source-url>`.
|
|
|
|
Note:
|
|
|
|
* The source-url should be absolute, with the directory containing the TEST_CASES.json file assumed
|
|
to be `/`.
|
|
* Whitespace is important and will be included when comparing the segments.
|
|
* There is a single space character between each part of the line.
|
|
* Newlines within a mapping must be escaped since the mapping and comment must all appear on a
|
|
single line of this file.
|
|
|
|
|
|
## Running tests
|
|
|
|
The simplest way to run all the compliance tests is:
|
|
|
|
```sh
|
|
yarn test-ivy-aot //packages/compiler-cli/test/compliance/...
|
|
```
|
|
|
|
If you only want to run one of the three types of test you can be more specific:
|
|
|
|
```sh
|
|
yarn test-ivy-aot //packages/compiler-cli/test/compliance/full
|
|
yarn test-ivy-aot //packages/compiler-cli/test/compliance/linked
|
|
yarn test-ivy-aot //packages/compiler-cli/test/compliance/test_cases/...
|
|
```
|
|
|
|
(The last command runs the partial compilation tests.)
|
|
|
|
## Updating a golden partial file
|
|
|
|
There is one golden partial file per `TEST_CASES.json` file. So even if this file defines multiple
|
|
test-cases, which each contain multiple input files, there will only be one golden file.
|
|
|
|
The golden file is generated by the tooling and should not be modified manually.
|
|
|
|
When you first create a test-case, with an empty `GOLDEN_PARTIAL.js` file, or a change is made to
|
|
the generated partial output, we must update the `GOLDEN_PARTIAL.js` file.
|
|
|
|
This is done by running a specific bazel rule of the form:
|
|
|
|
```sh
|
|
bazel run //packages/compiler-cli/test/compliance/test_cases:<path/to/test_case>.golden.update
|
|
```
|
|
|
|
where to replace `<path/to/test_case>` with the path (relative to `test_cases`) of the directory
|
|
that contains the `GOLDEN_PARTIAL.js` to update.
|
|
|
|
To update all golden partial files, the following command can be run:
|
|
|
|
```sh
|
|
node packages/compiler-cli/test/compliance/update_all_goldens.js
|
|
```
|
|
|
|
## Debugging test-cases
|
|
|
|
The full and linked compliance tests are basically `jasmine_node_test` rules. As such, they can be
|
|
debugged just like any other `jasmine_node_test`. The standard approach is to add `--config=debug`
|
|
to the Bazel test command.
|
|
|
|
For example:
|
|
|
|
```sg
|
|
yarn test-ivy-aot //packages/compiler-cli/test/compliance/full --config=debug
|
|
yarn test-ivy-aot //packages/compiler-cli/test/compliance/linked --config=debug
|
|
```
|
|
|
|
To debug generating the partial golden output use the following form of Bazel command:
|
|
|
|
```sh
|
|
yarn bazel run //packages/compiler-cli/test/compliance/test_cases:generate_partial_for_<path/to/test_case>.debug
|
|
```
|
|
|
|
The `path/to/test_case` is relative to the `test_cases` directory. So for this `TEST_CASES.json` file at:
|
|
|
|
```
|
|
packages/compiler-cli/test/compliance/test_cases/r3_view_compiler_directives/directives/matching/TEST_CASES.json
|
|
```
|
|
|
|
The command to debug the test-cases would be:
|
|
|
|
```
|
|
yarn bazel run //packages/compiler-cli/test/compliance/test_cases:generate_partial_for_r3_view_compiler_directives/directives/matching.debug
|
|
```
|
|
|
|
|
|
### Focusing test-cases
|
|
|
|
You can focus a test case by setting `"focusTest": true` in the `TEST_CASES.json` file.
|
|
This is equivalent to using jasmine `fit()`.
|
|
|
|
|
|
### Excluding test-cases
|
|
|
|
You can exclude a test case by setting `"excludeTest": true` in the `TEST_CASES.json` file.
|
|
This is equivalent to using jasmine `xit()`.
|