ngtsc has a robust suite of testing utilities, designed for in-memory
testing of a TypeScript compiler. Previously these utilities lived in the
`test` directory for the compiler-cli package.
This commit moves those utilities to an `ngtsc/testing` package, enabling
them to be depended on separately and opening the door for using them from
the upcoming language server testing infrastructure.
As part of this refactoring, the `fake_core` package (a lightweight API
replacement for @angular/core) is expanded to include functionality needed
for Language Service test use cases.
PR Close#39594
This reverts commit 561c0f81a0.
The original commit provided a quick escape from an already terminal
situation by killing the process if the PID in the lockfile was not
found in the list of processes running on the current machine.
But this broke use-cases where the node_modules was being shared between
multiple machines (or more commonly Docker containers on the same actual
machine).
Fixes#38875
PR Close#39435
Previously, UMD/CommonJS class inline declarations of the form:
```ts
exports.Foo = (function() { function Foo(); return Foo; })();
```
were capturing the whole IIFE as the implementation, rather than
the inner class (i.e. `function Foo() {}` in this case). This caused
the interpreter to break when it was trying to access such an export,
since it would try to evaluate the IIFE rather than treating it as a class
declaration.
PR Close#39346
Previously, inline exports of the form `exports.foo = <implementation>;` were
being interpreted (by the ngtsc `PartialInterpeter`) as `Reference` objects.
This is not what is desired since it prevents the value of the export
from being unpacked, such as when analyzing `NgModule` declarations:
```
exports.directives = [Directive1, Directive2];
@NgImport({declarations: [exports.directives]})
class AppModule {}
```
In this example the interpreter would think that `exports.directives`
was a reference rather than an array that needs to be unpacked.
This bug was picked up by the ngcc-validation repository. See
https://github.com/angular/ngcc-validation/pull/1990 and
https://circleci.com/gh/angular/ngcc-validation/17130
PR Close#39267
When ngcc is configured to run with the `--use-program-dependencies`
flag, as is the case in the CLI's asynchronous processing, it will scan
all source files in the program, starting from the program's root files
as configured in the tsconfig. Each individual root file could
potentially rescan files that had already been scanned for an earlier
root file, causing a severe performance penalty if the number of root
files is large. This would be the case if glob patterns are used in the
"include" specification of a tsconfig file.
This commit avoids the performance penalty by keeping track of the files
that have been scanned across all root files, such that no source file
is scanned multiple times.
Fixes#39240
PR Close#39254
Previously, any declarations that were defined "inline" were not
recognised by the `UmdReflectionHost`.
For example, the following syntax was completely unrecognized:
```ts
var Foo_1;
exports.Foo = Foo_1 = (function() {
function Foo() {}
return Foo;
})();
exports.Foo = Foo_1 = __decorate(SomeDecorator, Foo);
```
Such inline classes were ignored and not processed by ngcc.
This lack of processing led to failures in Ivy applications that relied
on UMD formats of libraries such as `syncfusion/ej2-angular-ui-components`.
Now all known inline UMD exports are recognized and processed accordingly.
Fixes#38947
PR Close#38959
Previously these tests were checking multiple specific expression
types. The new helper function is more general and will also support
`PropertyAccessExpression` nodes for `InlineDeclaration` types.
PR Close#38959
Previously the `ConcreteDeclaration` and `InlineDeclaration` had
different properties for the underlying node type. And the `InlineDeclaration`
did not store a value that represented its declaration.
It turns out that a natural declaration node for an inline type is the
expression. For example in UMD/CommonJS this would be the `exports.<name>`
property access node.
So this expression is now used for the `node` of `InlineDeclaration` types
and the `expression` property is dropped.
To support this the codebase has been refactored to use a new `DeclarationNode`
type which is a union of `ts.Declaration|ts.Expression` instead of `ts.Declaration`
throughout.
PR Close#38959
This makes these tests more resilient to changes in the test code
structure. For example switching from
```
var SomeClass = <implementation>;
exports.SomeClass = SomeClass;
```
to
```
exports.SomeClass = <implementation>;
```
PR Close#38959
Previously `getDeclaration()` would only return the first node that matched
the name passed in and then assert the predicate on this single node.
It also only considered a subset of possible declaration types that we might
care about.
Now the function will parse the whole tree collecting an array of all the
nodes that match the name. It then filters this array based on the predicate
and only errors if the filtered array is empty.
This makes this function much more resilient to more esoteric code formats
such as UMD.
PR Close#38959
Sometimes UMD exports appear in the following form:
```
exports.MyClass = alias1 = alias2 = <<declaration>>
```
Previously the declaration of the export would have been captured
as `alias1 = alias2 = <<declaration>>`, which the `PartialInterpreter`
would have failed on, since it cannot handle assignments.
Now we skip over these aliases capturing only the `<<declaration>>`
expression.
Fixes#38947
PR Close#38959
UMD files export values by assigning them to an `exports` variable.
When evaluating expressions ngcc was failing to cope with expressions
like `exports.MyComponent`.
This commit fixes the `UmdReflectionHost.getDeclarationOfIdentifier()`
method to map the `exports` variable to the current source file.
PR Close#38959
The `SIMPLE_CLASS_FILE` contained a `ChildClass` that had an
internal aliases implementation and extended a `SuperClass` base
class. The call to `__extends` was using the wrong argument for
the child class.
PR Close#38959
This is a precursor to introducing the Angular linker. As an initial
step, a compiler option to configure the compilation mode is introduced.
This option is initially internal until the linker is considered ready.
PR Close#38938
This commit refactors the `ExpressionTranslatorVisitor` so that it
is not tied directly to the TypeScript AST. Instead it uses generic
`TExpression` and `TStatement` types that are then converted
to concrete types by the `TypeScriptAstFactory`.
This paves the way for a `BabelAstFactory` that can be used to
generate Babel AST nodes instead of TypeScript, which will be
part of the new linker tool.
PR Close#38775
When the target of the compiler is ES2015 or newer then we should
be generating `let` and `const` variable declarations rather than `var`.
PR Close#38775
In #38666 we changed how ngcc deals with type expressions, where it
would now always emit the original type expression into the generated
code as a "local" type value reference instead of synthesizing new
imports using an "imported" type value reference. This was done as a fix
to properly deal with renamed symbols, however it turns out that the
compiler has special handling for certain imported symbols, e.g.
`ChangeDetectorRef` from `@angular/core`. The "local" type value
reference prevented this special logic from being hit, resulting in
incorrect compilation of pipe factories.
This commit fixes the issue by manually inspecting the import of the
type expression, in order to return an "imported" type value reference.
By manually inspecting the import we continue to handle renamed symbols.
Fixes#38883
PR Close#38892
Common AST formats such as TS and Babel do not use a separate
node for comments, but instead attach comments to other AST nodes.
Previously this was worked around in TS by creating a `NotEmittedStatement`
AST node to attach the comment to. But Babel does not have this facility,
so it will not be a viable approach for the linker.
This commit refactors the output AST, to remove the `CommentStmt` and
`JSDocCommentStmt` nodes. Instead statements have a collection of
`leadingComments` that are rendered/attached to the final AST nodes
when being translated or printed.
PR Close#38811
In the integration test suite of ngcc, we load a set of files from
`node_modules` into memory. This includes the `typescript` package and
`@angular` scoped packages, which account for a large number of large
files that needs to be loaded from disk. This commit moves this work
to the top-level, such that it doesn't have to be repeated in all tests.
PR Close#38840
Recent optimizations to ngcc have significantly reduced the total time
it takes to process `node_modules`, to such extend that sharding across
multiple processes has become less effective. Previously, running
ngcc asynchronously would allow for up to 8 workers to be allocated,
however these workers have to repeat work that could otherwise be shared.
Because ngcc is now able to reuse more shared computations, the overhead
of multiple workers is increased and therefore becomes less effective.
As an additional benefit, having fewer workers requires less memory and
less startup time.
To give an idea, using the following test setup:
```bash
npx @angular/cli new perf-test
cd perf-test
yarn ng add @angular/material
./node_modules/.bin/ngcc --properties es2015 module main \
--first-only --create-ivy-entry-points
```
We observe the following figures on CI:
| | 10.1.1 | PR #38840 |
| ----------------- | --------- | --------- |
| Sync | 85s | 25s |
| Async (8 workers) | 22s | 16s |
| Async (4 workers) | - | 11s |
In addition to changing the default number of workers, ngcc will now
use the environment variable `NGCC_MAX_WORKERS` that may be configured
to either reduce or increase the number of workers.
PR Close#38840
ngcc creates typically two `ts.Program` instances for each entry-point,
one for processing sources and another one for processing the typings.
The creation of these programs is somewhat expensive, as it concerns
module resolution and parsing of source files.
This commit implements several layers of caching to optimize the
creation of programs:
1. A shared module resolution cache across all entry-points within a
single invocation of ngcc. Both the sources and typings program
benefit from this cache.
2. Sharing the parsed `ts.SourceFile` for a single entry-point between
the sources and typings program.
3. Sharing parsed `ts.SourceFile`s of TypeScript's default libraries
across all entry-points within a single invocation. Some of these
default library typings are large and therefore expensive to parse,
so sharing the parsed source files across all entry-points offers
a significant performance improvement.
Using a bare CLI app created using `ng new` + `ng add @angular/material`,
the above changes offer a 3-4x improvement in ngcc's processing time
when running synchronously and ~2x improvement for asynchronous runs.
PR Close#38840
If a type has been renamed when it was exported, we need to
reference the external public alias name rather than the internal
original name for the type. Otherwise we will try to import the
type by its internal name, which is not publicly accessible.
Fixes#38238
PR Close#38666
Similarly to the change we landed in the `@angular/core` reflection
capabilities, we need to make sure that ngcc can detect pass-through
delegate constructors for classes using downleveled ES2015 output.
More details can be found in the preceding commit, and in the issue
outlining the problem: #38453.
Fixes#38453.
PR Close#38463
In TypeScript 3.8 support was added for type-only imports, which only brings in
the symbol as a type, not their value. The Angular compiler did not yet take
the type-only keyword into account when representing symbols in type positions
as value expressions. The class metadata that the compiler emits would include
the value expression for its parameter types, generating actual imports as
necessary. For type-only imports this should not be done, as it introduces an
actual import of the module that was originally just a type-only import.
This commit lets the compiler deal with type-only imports specially, preventing
a value expression from being created.
Fixes#37900
PR Close#37912
In CLI v10 there was a move to use the new solution-style tsconfig
which became available in TS 3.9.
The result of this is that the standard tsconfig.json no longer contains
important information such as "paths" mappings, which ngcc might need to
correctly compute dependencies.
ngcc (and ngc and tsc) infer the path to tsconfig.json if not given an
explicit tsconfig file-path. But now that means it infers the solution
tsconfig rather than one that contains the useful information it used to
get.
This commit logs a warning in this case to inform the developer
that they might not have meant to load this tsconfig and offer
alternative options.
Fixes#36386
PR Close#38003
When ngcc creates an entry-point program, the `allowJs` option is enabled
in order to operate on the JavaScript source files of the entry-point.
A side-effect of this approach is that external modules that don't ship
declaration files will also have their JavaScript source files loaded
into the program, as the `allowJs` flag allows for them to be imported.
This may pose an issue in certain edge cases, where ngcc would inadvertently
operate on these external modules. This can introduce all sorts of undesirable
behavior and incompatibilities, e.g. the reflection host that is selected for
the entry-point's format could be incompatible with that of the external
module's JavaScript bundles.
To avoid these kinds of issues, module resolution that would resolve to
a JavaScript file located outside of the package will instead be rejected,
as if the file would not exist. This would have been the behavior when
`allowJs` is set to false, which is the case in typical Angular compilations.
Fixes#37508
PR Close#37596
Changes `isWithinPackage` to take an `AbsoluteFsPath` instead of `ts.SourceFile`,
to allow for an upcoming change to use it when no `ts.SourceFile` is available,
but just a path.
PR Close#37596
Previously an error thrown in the `analyzeFn` would cause
the ngcc process to exit immediately without removing the
lockfile, and potentially before the unlocker process had been
successfully spawned resulting in the lockfile being orphaned
and left behind.
Now we catch these errors and remove the lockfile as needed.
PR Close#37739
Previously the `ProgramBasedEntryPointFinder` was parsing all the
entry-points referenced by the program for dependencies even if all the
entry-points had been processed already.
Now this entry-point finder will re-use the `EntryPointManifest` to load
the entry-point dependencies when possible which avoids having to parse
them all again, on every invocation of ngcc.
Previously the `EntryPointManifest` was only used in the
`DirectoryWalkerEntryPointFinder`, which also contained the logic for
computing the contents of the manifest. This logic has been factored out
into an `EntryPointCollector` class. Both the `ProgramBasedEntryPointFinder`
and `DirectoryWalkerEntryPointFinder` now use the `EntryPointManifest` and
the `EntryPointCollector`.
The result of this change is that there is a small cost on the first run of
ngcc to compute and store the manifest - the processing takes 102% of the
processing time before this PR. But on subsequent runs there is a
significant benefit on subsequent runs - the processing takes around 50%
of the processing time before this PR.
PR Close#37665
The file-writing error in the this commit can also be the result
of the ngcc process dying in the middle of writing files.
This commit improves the error message to offer a resolution
in case this is the reason for the error.
Fixes#36393
PR Close#37672
The `SourceFile` and associated code is general and reusable in
other projects (such as `@angular/localize`). Moving it to `ngtsc`
makes it more easily shared.
PR Close#37114
The `Logger` interface and its related classes are general purpose
and could be used by other tooling. Moving it into ngtsc is a more
suitable place from which to share it - similar to the FileSystem stuff.
PR Close#37114
Previously, ngcc would only be able to match an ngcc configuration to
packages that were located inside the project's top-level
`node_modules/`. However, if there are multiple versions of a package in
a project (e.g. as a transitive dependency of other packages), multiple
copies of a package (at different versions) may exist in nested
`node_modules/` directories. For example, one at
`<project-root>/node_modules/some-package/` and one at
`<project-root>/node_modules/other-package/node_modules/some-package/`.
In such cases, ngcc was only able to detect the config for the first
copy but not for the second.
This commit fixes this by returning a new instance of
`ProcessedNgccPackageConfig` for each different package path (even if
they refer to the same package name). In these
`ProcessedNgccPackageConfig`, the `entryPoints` paths have been
processed to take the package path into account.
PR Close#37040
This commit adds a `packageName` property to the `EntryPoint` interface.
In a subsequent commit this will be used to retrieve the correct ngcc
configuration for each package, regardless of its path.
PR Close#37040
In order to retrieve the ngcc configuration (if any) for an entry-point,
ngcc has to detect the containing package's version.
Previously, ngcc would try to read the version from the entry-point's
`package.json` file, which was different than the package's top-level
`package.json` for secondary entry-points. For example, it would try to
read it from `node_modules/@angular/common/http/package.json` for
entry-point `@angular/common/http`. However, the `package.json` files
for secondary entry-points are not guaranteed to include a `version`
property.
This commit fixes this by first trying to read the version from the
_package's_ `package.json` (falling back to the entry-point's
`package.json`). For example, it will first try to read it from
`@angular/common/package.json` for entry-point `@angular/common/http`.
PR Close#37040
Rename the `package` property to `packagePath` on the `EntryPoint`
interface. This makes it more clear that the `packagePath` property
holds the absolute path to the containing package (similar to how `path`
holds the path to the entry-point). This will also align with the
`packageName` property that will be added in a subsequent commit.
This commit also re-orders the `EntryPoint` properties to group related
properties together and to match the order of properties on instances
with that on the interface.
PR Close#37040
Previously, when an entry-point was ignored via an ngcc config, ngcc
would scan sub-directories for sub-entry-points, but would not use the
correct `packagePath`. For example, if `@angular/common` was ignored, it
would look at `@angular/common/http` but incorrectly use
`.../@angular/common/http` as the `packagePath` (instead of
`.../@angular/common`). As a result, it would not retrieve the correct
ngcc config for the actual package.
This commit fixes it by ensuring the correct `packagePath` is used, even
if the primary entry-point corresponding to that path is ignored. In
order to do this, a new return value for `getEntryPointInfo()` is added:
`IGNORED_ENTRY_POINT`. This is used to differentiate between directories
that correspond to no or an incompatible entry-point and those that
correspond to an entry-point that could otherwise be valid but is
explicitly ignored. Consumers of `getEntryPointInfo()` can then use this
info to discard ignored entry-points, but still use the correct
`packagePath` when scanning their sub-directories for secondary
entry-points.
PR Close#37040
Adds @nocollapse to static properties added by ngcc
iff annotateForClosureCompiler is true.
The Closure Compiler will collapse static properties
into the global namespace. Adding this annotation keeps
the properties attached to their respective object, which
allows them to be referenced via a class's constructor.
The annotation is already added by ngtsc and ngc under the
same option, this commit extends the functionality to ngcc.
Closes#36618.
PR Close#36652
It is quite common for the TS compiler to have to add synthetic
types to function signatures, where the developer has not
explicitly provided them. This results in `import(...)` expressions
appearing in typings files. For example in `@ngrx/data` there is a
class with a getter that has an implicit type:
```ts
export declare class EntityCollectionServiceBase<...> {
...
get store() {
return this.dispatcher.store;
}
...
}
```
In the d.ts file for this we get:
```ts
get store(): Store<import("@ngrx/data").EntityCache>;
```
Given that this file is within the `@ngrx/data` package already,
this caused ngcc to believe that there was a circular dependency,
causing it to fail to process the package - and in fact crash!
This commit resolves this problem by ignoring `import()` expressions
when scanning typings programs for dependencies. This ability was
only introduced very recently in a 10.0.0 RC release, and so it has
limited benefit given that up till now ngcc has been able to process
libraries effectively without it. Moreover, in the rare case that a
package does have such a dependency, it should get picked up
by the sync ngcc+CLI integration point.
PR Close#37503
Now in TS 3.9, classes in ES2015 can be wrapped in an IIFE.
This commit ensures that we still find the static properties that contain
decorator information, even if they are attached to the adjacent node
of the class, rather than the implementation or declaration.
Fixes#37330
PR Close#37436
This finder is designed to only process entry-points that are reachable
by the program defined by a tsconfig.json file.
It is triggered by calling `mainNgcc()` with the `findEntryPointsFromTsConfigProgram`
option set to true. It is ignored if a `targetEntryPointPath` has been
provided as well.
It is triggered from the command line by adding the `--use-program-dependencies`
option, which is also ignored if the `--target` option has been provided.
Using this option can speed up processing in cases where there is a large
number of dependencies installed but only a small proportion of the
entry-points are actually imported into the application.
PR Close#37075
Previously we only checked for static import declaration statements.
This commit also finds import paths from dynamic import expressions.
Also this commit should speed up processing: Previously we were parsing
the source code contents into a `ts.SourceFile` and then walking the parsed
AST to find import paths.
Generating an AST is unnecessary work and it is faster and creates less
memory pressure to just scan the source code contents with the TypeScript
scanner, identifying import paths from the tokens.
PR Close#37075
Previously this host was skipping files if they had imports that spanned
multiple lines, or if the import was a dynamic import expression.
PR Close#37075
This commit will store a cached copy of the parsed tsconfig
that can be reused if the tsconfig path is the same.
This will improve the ngcc "noop" case, where there is no processing
to do, when the entry-points have already been processed.
Previously we were parsing this config every time we checked for
entry-points to process, which can take up to seconds in some
cases.
Resolves#36882
PR Close#37417
Currently, if an ngcc process is killed in a manner that it doesn't clean
up its lock file (or is killed too quickly) the compiler reports that it
is waiting on the PID of a process that doesn't exist, and that it will
wait up to a maximum of N seconds. This PR updates the locking code to
additionally check if the process exists, and if it does not it will
immediately bail out, and print the location of the lock file so a user
may clean it up.
PR Close#37250
Inline source-maps in typings files can impact IDE performance
so ngcc should only add such maps if the original typings file
contains inline source-maps.
Fixes#37324
PR Close#37363
In #37221 we disabled tsickle passes from transforming the tsc output that is used to publish all
Angular framework and components packages (@angular/*).
This change however revealed a bug in the ngc that caused __decorate and __metadata calls to still
be emitted in the JS code even though we don't depend on them.
Additionally it was these calls that caused code in @angular/material packages to fail at runtime
due to circular dependency in the emitted decorator code documeted as
https://github.com/microsoft/TypeScript/issues/27519.
This change partially rolls back #37221 by reenabling the decorator to static fields (static
properties) downleveling.
This is just a temporary workaround while we are also fixing root cause in `ngc` - tracked as
FW-2199.
Resolves FW-2198.
Related to FW-2196
PR Close#37317
Alex Rickabaugh
Pete Bacon Darwin
JoostK
Paul Gschwendtner
George Kalpakas
David Neil
Terence D. Honles
Igor Minar
Joey Perrott