Previously, when running in parallel mode and a worker process crashed
while processing a task, it was not possible for ngcc to continue
without risking ending up with a corrupted entry-point and therefore it
exited with an error. This, for example, could happen when a worker
process received a `SIGKILL` signal, which was frequently observed in CI
environments. This was probably the result of Docker killing processes
due to increased memory pressure.
One factor that amplifies the problem under Docker (which is often used
in CI) is that it is not possible to distinguish between the available
CPU cores on the host machine and the ones made available to Docker
containers, thus resulting in ngcc spawning too many worker processes.
This commit addresses these issues in the following ways:
1. We take advantage of the fact that files are written to disk only
after an entry-point has been fully analyzed/compiled. The master
process can now determine whether a worker process has not yet
started writing files to disk (even if it was in the middle of
processing a task) and just put the task back into the tasks queue if
the worker process crashes.
2. The master process keeps track of the transformed files that a worker
process will attempt to write to disk. If the worker process crashes
while writing files, the master process can revert any changes and
put the task back into the tasks queue (without risking corruption).
3. When a worker process crashes while processing a task (which can be a
result of increased memory pressure or too many worker processes),
the master process will not try to re-spawn it. This way the number
or worker processes is gradually adjusted to a level that can be
accomodated by the system's resources.
Examples of ngcc being able to recover after a worker process crashed:
- While idling: https://circleci.com/gh/angular/angular/682197
- While compiling: https://circleci.com/gh/angular/angular/682209
- While writing files: https://circleci.com/gh/angular/angular/682267
Jira issue: [FW-2008](https://angular-team.atlassian.net/browse/FW-2008)
Fixes#36278
PR Close#36626
This commit adds a `revertFile()` method to `FileWriter`, which can
revert a transformed file (and its backup - if any) written by the
`FileWriter`.
In a subsequent commit, this will be used to allow ngcc to recover
when a worker process crashes in the middle of processing a task.
PR Close#36626
With this commit, the master process will keep track of the transformed
files that each worker process is intending to write to disk.
In a subsequent commit, this info will be used to allow ngcc to recover
when a worker process crashes in the middle of processing a task.
PR Close#36626
With this commit, worker processes will notify the master process about
the transformed files they are about to write to disk before starting
writing them.
In a subsequent commit, this will be used to allow ngcc to recover when
a worker process crashes in the middle of processing a task.
PR Close#36626
This commit enhances the `CompileFn`, which is used to process each
entry-point, to support running a passed-in callback (and wait for it to
complete) before proceeding with writing the transformed files to disk.
This functionality is currently not used. In a subsequent commit, it
will be used for passing info from worker processes to the master
process that will allow ngcc to recover when a worker process crashes in
the middle of processing a task.
PR Close#36626
Rename the `markTaskCompleted()` method to be consistent with the other
similar methods of `TaskQueue` (`markAsFailed()` and
`markAsUnprocessed()`).
PR Close#36626
This commit adds support for stopping processing an in-progress task
and moving it back to the list of pending tasks.
In a subsequent commit, this will be used to allow ngcc to recover when
a worker process crashes in the middle of processing a task.
PR Close#36626
Previously, ngcc would run in parallel mode (using the
`ClusterExecutor`) when there were at least 2 CPU cores (and all other
requirements where met). On systems with just 2 CPU cores, this meant
there would only be one worker process (since one CPU core is always
reserved for the master process). In these cases, the tasks would still
be processed serially (on the one worker process), but we would also pay
the overhead of communicating between the master and worker processes.
This commit fixes this by only running in parallel mode if there are
more than 2 CPU cores (i.e. at least 2 worker processes).
PR Close#36626
Previously, the "Compiling <entryPoint>" log message was printed before
starting to analyze and transform files, but after creating the
`EntryPointBundle` (which includes creating the TS program).
Since creating the `EntryPointBundle` involves some work, it is more
accurate to move the log message before creating the bundle.
PR Close#36626
An enum declaration in TypeScript code will be emitted into JavaScript
as a regular variable declaration, with the enum members being declared
inside an IIFE. For ngcc to support interpreting such variable
declarations as enum declarations with its members, ngcc needs to
recognize the enum declaration emit structure and extract all member
from the statements in the IIFE.
This commit extends the `ConcreteDeclaration` structure in the
`ReflectionHost` abstraction to be able to capture the enum members
on a variable declaration, as a substitute for the original
`ts.EnumDeclaration` as it existed in TypeScript code. The static
interpreter has been extended to handle the extracted enum members
as it would have done for `ts.EnumDeclaration`.
Fixes#35584
Resolves FW-2069
PR Close#36550
The html parser already normalizes line endings (converting `\r\n` to `\n`)
for most text in templates but it was missing the expressions of ICU expansions.
In ViewEngine backticked literal strings, used to define inline templates,
were already normalized by the TypeScript parser.
In Ivy we are parsing the raw text of the source file directly so the line
endings need to be manually normalized.
This change ensures that inline templates have the line endings of ICU
expression normalized correctly, which matches the ViewEngine.
In ViewEngine external templates, defined in HTML files, the behavior was
different, since TypeScript was not normalizing the line endings.
Specifically, ICU expansion "expressions" are not being normalized.
This is a problem because it means that i18n message ids can be different on
different machines that are setup with different line ending handling,
or if the developer moves a template from inline to external or vice versa.
The goal is always to normalize line endings, whether inline or external.
But this would be a breaking change since it would change i18n message ids
that have been previously computed. Therefore this commit aligns the ivy
template parsing to have the same "buggy" behavior for external templates.
There is now a compiler option `i18nNormalizeLineEndingsInICUs`, which
if set to `true` will ensure the correct non-buggy behavior. For the time
being this option defaults to `false` to ensure backward compatibility while
allowing opt-in to the desired behavior. This option's default will be
flipped in a future breaking change release.
Further, when this option is set to `false`, any ICU expression tokens,
which have not been normalized, are added to the `ParseResult` from the
`HtmlParser.parse()` method. In the future, this collection of tokens could
be used to diagnose and encourage developers to migrate their i18n message
ids. See FW-2106.
Closes#36725
PR Close#36741
The cached file-system was implemented to speed up ngcc
processing, but in reality most files are not accessed many times
and there is no noticeable degradation in speed by removing it.
Benchmarking `ngcc -l debug` for AIO on a local machine
gave a range of 196-236 seconds with the cache and 197-224
seconds without the cache.
Moreover, when running in parallel mode, ngcc has a separate
file cache for each process. This results in excess memory usage.
Notably the master process, which only does analysis of entry-points
holds on to up to 500Mb for AIO when using the cache compared to
only around 30Mb when not using the cache.
Finally, the file-system cache being incorrectly primed with file
contents before being processed has been the cause of a number
of bugs. For example https://github.com/angular/angular-cli/issues/16860#issuecomment-614694269.
PR Close#36687
The change in e041ac6f0d
to support sending unlocker process output to the main ngcc
console output prevents messages require that the main process
relinquishes the event-loop to allow the `stdout.on()` handler to
run. This results in none of the messages being written when ngcc
is run in `--no-async` mode, and some messages failing to be
written if the main process is killed (e.g. ctrl-C).
It appears that the problem with Windows and detached processes
is known - see https://github.com/nodejs/node/issues/3596#issuecomment-250890218.
But in the meantime, this commit is a workaround, where non-Windows
`inherit` the main process `stdout` while on Windows it reverts
to the async handler approach, which is better than nothing.
PR Close#36637
On Windows, the output of a detached process (such as the unlocker
process used by `LockFileWithChildProcess`) is not shown in the parent
process' stdout.
This commit addresses this by piping the spawned process' stdin/stdout
and manually writing to the parent process' stdout.
PR Close#36569
The current ngcc lock-file strategy spawns a new process in order to
capture a potential `SIGINT` and remove the lock-file. For more
information see #35861.
Previously, this unlocker process was spawned as soon as the `LockFile`
was instantiated in order to have it available as soon as possible
(given that spawning a process is an asynchronous operation). Since the
`LockFile` was instantiated and passed to the `Executor`, this meant
that an unlocker process was spawned for each cluster worker, when
running ngcc in parallel mode. These processes were not needed, since
the `LockFile` was not used in cluster workers, but we still had to pay
the overhead of each process' own memory and V8 instance.
(NOTE: This overhead was small compared to the memory consumed by ngcc's
normal operations, but still unnecessary.)
This commit avoids the extra processes by only spawning an unlocker
process when running on the cluster master process and not on worker
processes.
PR Close#36569
For some reason (possibly related to async/await promises)
the ngcc process is not exiting when spawned from the CLI
when there has been an error (such as when it timesout waiting
for a lockfile to become free).
Calling `process.exit()` directly fixes this.
Fixes#36616
PR Close#36622
Previously, when we needed to detect whether a file is external to a
package, we only checked whether the relative path to the file from the
package's root started with `..`. This would detect external imports
when the packages were siblings (e.g. peer dependencies or hoisted to
the top of `node_modules/` by the package manager), but would fail to
detect imports from packages located in nested `node_modules/` as
external. For example, importing `node_modules/foo/node_modules/bar`
from a file in `node_modules/foo/` would be considered internal to the
`foo` package.
This could result in processing/analyzing more files than necessary.
More importantly it could lead to errors due to trying to analyze
non-Angular packages that were direct dependencies of Angular packages.
This commit fixes it by also verifying that the relative path to a file
does not start with `node_modules/`.
Jira issue: [FW-2068](https://angular-team.atlassian.net/browse/FW-2068)
Fixes#36526
PR Close#36559
The base path for package and entry-points is known so there is
no need to store these in the file. Also this commit avoids storing
empty arrays unnecessarily.
PR Close#36486
Previously, even if an entry-point did not need to be processed,
ngcc would always parse the files of the entry-point to compute
its dependencies. This can take a lot of time for large node_modules.
Now these dependencies are cached in the entry-point manifest,
and read from there rather than computing them every time.
See https://github.com/angular/angular/issues/36414\#issuecomment-608401834
FW-2047
PR Close#36486
When the compiler needs to convert a type reference to a value
expression, it may encounter a type that refers to a namespaced symbol.
Such namespaces need to be handled specially as there's various forms
available. Consider a namespace named "ns":
1. One can refer to a namespace by itself: `ns`. A namespace is only
allowed to be used in a type position if it has been merged with a
class, but even if this is the case it may not be possible to convert
that type into a value expression depending on the import form. More
on this later (case a below)
2. One can refer to a type within the namespace: `ns.Foo`. An import
needs to be generated to `ns`, from which the `Foo` property can then
be read.
3. One can refer to a type in a nested namespace within `ns`:
`ns.Foo.Bar` and possibly even deeper nested. The value
representation is similar to case 2, but includes additional property
accesses.
The exact strategy of how to deal with these cases depends on the type
of import used. There's two flavors available:
a. A namespaced import like `import * as ns from 'ns';` that creates
a local namespace that is irrelevant to the import that needs to be
generated (as said import would be used instead of the original
import).
If the local namespace "ns" itself is referred to in a type position,
it is invalid to convert it into a value expression. Some JavaScript
libraries publish a value as default export using `export = MyClass;`
syntax, however it is illegal to refer to that value using "ns".
Consequently, such usage in a type position *must* be accompanied by
an `@Inject` decorator to provide an explicit token.
b. An explicit namespace declaration within a module, that can be
imported using a named import like `import {ns} from 'ns';` where the
"ns" module declares a namespace using `declare namespace ns {}`.
In this case, it's the namespace itself that needs to be imported,
after which any qualified references into the namespace are converted
into property accesses.
Before this change, support for namespaces in the type-to-value
conversion was limited and only worked correctly for a single qualified
name using a namespace import (case 2a). All other cases were either
producing incorrect code or would crash the compiler (case 1a).
Crashing the compiler is not desirable as it does not indicate where
the issue is. Moreover, the result of a type-to-value conversion is
irrelevant when an explicit injection token is provided using `@Inject`,
so referring to a namespace in a type position (case 1) could still be
valid.
This commit introduces logic to the type-to-value conversion to be able
to properly deal with all type references to namespaced symbols.
Fixes#36006
Resolves FW-1995
PR Close#36106
In cc4b813e75 the `getBasePaths()`
function was changed to log a warning if a `basePath()` computed from
the `paths` mappings did not exist. It turns out this is a common and
accepted scenario, so we should not log warnings in this case.
Fixes#36518
PR Close#36525
1. update jasmine to 3.5
2. update @types/jasmine to 3.5
3. update @types/jasminewd2 to 2.0.8
Also fix several cases, the new jasmine 3 will help to create test cases correctly,
such as in the `jasmine 2.x` version, the following case will pass
```
expect(1 == 2);
```
But in jsamine 3, the case will need to be
```
expect(1 == 2).toBeTrue();
```
PR Close#34625
Previously, `isRelativePath()` assumed paths are *nix-style. This caused
Windows-style paths (such as `C:\foo\some-package\some-file.js`) to not
be recognized as "relative" imports.
This commit fixes this by using the OS-agnostic `isRooted()` helper and
also accounting for both styles of path delimiters: `/` and `\`
PR Close#36372
When TypeScript downlevels ES2015+ code to ES5, it uses some helper
functions to emulate some ES2015+ features, such as spread syntax. The
TypeScript compiler can be configured to emit these helpers into the
transpiled code (which is controlled by the `noEmitHelpers` option -
false by default). It can also be configured to import these helpers
from the `tslib` module (which is controlled by the `importHelpers`
option - false by default).
While most of the time the helpers will be either emitted or imported,
it is possible that one configures their app to neither emit nor import
them. In that case, the helpers could, for example, be made available on
the global object. This is what `@nativescript/angular`
v9.0.0-next-2019-11-12-155500-01 does. See, for example, [common.js][1].
Ngcc must be able to detect and statically evaluate these helpers.
Previously, it was only able to detect emitted or imported helpers.
This commit adds support for detecting these helpers if they are neither
emitted nor imported. It does this by checking identifiers for which no
declaration (either concrete or inline) can be found against a list of
known TypeScript helper function names.
[1]: https://unpkg.com/browse/@nativescript/angular@9.0.0-next-2019-11-12-155500-01/common.js
PR Close#36418
The source-map flattening was throwing an error when there
is a cyclic dependency between source files and source-maps.
The error was either a custom one describing the cycle, or a
"Maximum call stack size exceeded" one.
Now this is handled more leniently, resulting in a partially loaded
source file (or source-map) and a warning logged.
Fixes#35727Fixes#35757
Fixes https://github.com/angular/angular-cli/issues/17106
Fixes https://github.com/angular/angular-cli/issues/17115
PR Close#36452
Recently we added support for ignoring specified deep-import
warnings by providing sets of regular expressions within the
`ngcc.config.js` file. But this was only working for the project
level configuration.
This commit fixes ngcc so that it will also read these regular
expressions from package level configuration too.
Fixes#35750
PR Close#36423
The `browser` package.json property is now supported to the same
level as `main` - i.e. it is sniffed for UMD, ESM5 and CommonJS.
The `browser` property can also contain an object with file overrides
but this is not supported by ngcc.
Fixes#36062
PR Close#36396
Previously, `main` was only checked for `umd` or `commonjs`
formats. Now if there are `import` or `export` statements in the
source file it will be deemed to be in `esm5` format.
Fixes#35788
PR Close#36396
The `NgccReflectionHost`s have logic for detecting certain known
declarations (such as `Object.assign()` and TypeScript helpers), which
allows the `PartialEvaluator` to evaluate expressions it would not be
able to statically evaluate otherwise.
In #36089, `DelegatingReflectionHost` was introduced, which delegates to
a TypeScript `ReflectionHost` when reflecting on TypeScript files, which
for ngcc's case means `.d.ts` files of dependencies. As a result, ngcc
lost the ability to detect TypeScript helpers imported from `tslib`,
because `DelegatingReflectionHost` was not able to apply the known
declaration detection logic while reflecting on `tslib`'s `.d.ts` files.
This commit fixes this by ensuring `DelegatingReflectionHost` calls the
`NgccReflectionHost`'s `detectKnownDeclaration()` method as necessary,
even when using the TypeScript `ReflectionHost`.
NOTE:
The previous commit exposed a bug in ngcc that was hidden due to the
tests' being inconsistent with how the `ReflectionHost`s are used in the
actual program. The changes in this commit are verified by ensuring the
failing tests are now passing (hence no new tests are added).
PR Close#36284
In #36089, `DelegatingReflectionHost` was introduced. Under the hood, it
delegates another `NgccReflectionHost` in order to reflect over the
program's source files, while using a different TypeScript
`ReflectionHost` to reflect over `.d.ts` files (which is how external
dependencies are represented in the program).
Previously, the `NgccReflectionHost`s were used directly in tests. This
does not exercise them in the way they are exercised in the actual
program, because (when used directly) they will also reflect on `.d.ts`
files too (instead of delegating to the TypeScript `ReflectionHost`).
This could hide bugs that would happen on the actual program.
This commit fixes this by using the `DelegatingReflectionHost` in the
various `NgccReflectionHost` tests.
NOTE:
This change will cause some of the existing tests to start failing.
These failures demonstrate pre-existing bugs in ngcc, that were hidden
due to the tests' being inconsistent with how the `ReflectionHost`s are
used in the actual program. They will be fixed in the next commit.
PR Close#36284
The `NgccReflectionHost`s have logic for detecting certain known
declarations (such as `Object.assign()` and TypeScript helpers), which
allows the `PartialEvaluator` to evaluate expressions it would not be
able to statically evaluate otherwise.
This commit moves the logic for identifying these known declarations to
dedicated methods. This is in preparation of allowing ngcc's
`DelegatingReflectionHost` (introduced in #36089) to also apply the
known declaration detection logic when reflecting on TypeScript sources.
PR Close#36284
Ngcc supports providing a project-level configuration to affect how
certain dependencies are processed and also has a built-in fallback
configuration for some unmaintained packages. Each entry in these
configurations could be scoped to specific versions of a package by
providing a version range. If no version range is provided for a
package, it defaults to `*` (with the intention of matching any
version).
Previously, the installed version of a package was tested against the
version range using the [semver][1] package's `satisfies()` function
with the default options. By default, `satisfies()` does not match
pre-releases (see [here][2] for more details on reasoning). While this
makes sense when determining what version of a dependency to install
(trying to avoid unexpected breaking changes), it is not desired in the
case of ngcc.
This commit fixes it by explicitly specifying that pre-release versions
should be matched normally.
[1]: https://www.npmjs.com/package/semver
[2]: https://github.com/npm/node-semver#prerelease-tags
PR Close#36370
Previously, a bad baseUrl or path mapping passed to an `EntryPointFinder`
could cause the original `sourceDirectory` to be superceded by a higher
directory. This could result in none of the sourceDirectory entry-points being
processed.
Now missing basePaths computed from path-mappings are discarded with
a warning. Further, if the `baseUrl` is the root directory then a warning is
given as this is most likely an error in the tsconfig.json.
Resolves#36313Resolves#36283
PR Close#36331
The previous optimizations in #35756 to the
`DirectoryWalkerEntryPointFinder` were over zealous
with regard to packages that have entry-points stored
in "container" directories in the package, where the
container directory was not an entry-point itself.
Now we will also walk such "container" folders as long
as they do not contain `.js` files, which we regard as an
indicator that the directory will not contain entry-points.
Fixes#36216
PR Close#36305
This commit simplifies the `DirectoryWalkerEntryPointFinder` inter-method
calling to make it easier to follow, and also to support controlling
walking of a directory based on its children.
PR Close#36305
Previously we only searched for package paths below the set of `basePaths`
that were computed from the `basePath` provided to ngcc and the set of
`pathMappings`.
In some scenarios, such as hoisted packages, the entry-point is not within
any of the `basePaths` identified above. For example:
```
project
packages
app
node_modules
app-lib (depends on lib1)
node_modules
lib1 (depends on lib2)
node_modules
lib2 (depends on lib3/entry-point)
lib3
entry-point
```
When CLI is compiling `app-lib` ngcc will be given
`project/packages/app/node_modules` as the `basePath.
If ngcc is asked to target `lib2`, the `targetPath` will be
`project/node_modules/lib1/node_modules/lib2`.
Since `lib2` depends upon `lib3/entry-point`, ngcc will need to compute
the package path for `project/node_modules/lib3/entry-point`.
Since `project/node_modules/lib3/entry-point` is not contained in the `basePath`
`project/packages/app/node_modules`, ngcc failed to compute the `packagePath`
correctly, instead assuming that it was the same as the entry-point path.
Now we also consider the nearest `node_modules` folder to the entry-point
path as an additional `basePath`. If one is found then we use the first
directory directly below that `node_modules` directory as the package path.
In the case of our example this extra `basePath` would be `project/node_modules`
which allows us to compute the `packagePath` of `project/node_modules/lib3`.
Fixes#35747
PR Close#36249
Previously ngcc never preserved whitespaces but this is at odds
with how the ViewEngine compiler works. In ViewEngine, library
templates are recompiled with the current application's tsconfig
settings, which meant that whitespace preservation could be set
in the application tsconfig file.
This commit allows ngcc to use the `preserveWhitespaces` setting
from tsconfig when compiling library templates. One should be aware
that this disallows different projects with different tsconfig settings
to share the same node_modules folder, with regard to whitespace
preservation. But this is already the case in the current ngcc since
this configuration is hard coded right now.
Fixes#35871
PR Close#36189
This commit augments the `FactoryDef` declaration of Angular decorated
classes to contain information about the parameter decorators used in
the constructor. If no constructor is present, or none of the parameters
have any Angular decorators, then this will be represented using the
`null` type. Otherwise, a tuple type is used where the entry at index `i`
corresponds with parameter `i`. Each tuple entry can be one of two types:
1. If the associated parameter does not have any Angular decorators,
the tuple entry will be the `null` type.
2. Otherwise, a type literal is used that may declare at least one of
the following properties:
- "attribute": if `@Attribute` is present. The injected attribute's
name is used as string literal type, or the `unknown` type if the
attribute name is not a string literal.
- "self": if `@Self` is present, always of type `true`.
- "skipSelf": if `@SkipSelf` is present, always of type `true`.
- "host": if `@Host` is present, always of type `true`.
- "optional": if `@Optional` is present, always of type `true`.
A property is only present if the corresponding decorator is used.
Note that the `@Inject` decorator is currently not included, as it's
non-trivial to properly convert the token's value expression to a
type that is valid in a declaration file.
Additionally, the `ComponentDefWithMeta` declaration that is created for
Angular components has been extended to include all selectors on
`ng-content` elements within the component's template.
This additional metadata is useful for tooling such as the Angular
Language Service, as it provides the ability to offer suggestions for
directives/components defined in libraries. At the moment, such
tooling extracts the necessary information from the _metadata.json_
manifest file as generated by ngc, however this metadata representation
is being replaced by the information emitted into the declaration files.
Resolves FW-1870
PR Close#35695
When computing the dependencies between packages which are not in
node_modules, we may need to rely upon path-mappings to find the path
to the imported entry-point.
This commit allows ngcc to use the path-mappings from a tsconfig
file to find dependencies. By default any tsconfig.json file in the directory
above the `basePath` is loaded but it is possible to use a path to a
specific file by providing the `tsConfigPath` property to mainNgcc,
or to turn off loading any tsconfig file by setting `tsConfigPath` to `null`.
At the command line this is controlled via the `--tsconfig` option.
Fixes#36119
PR Close#36180
When two entry-points overlap, ngcc may attempt to process some
files twice. Previously, when this occured ngcc would just exit with an
error preventing any other entry-points from being processed.
This commit changes ngcc so that if `errorOnFailedEntryPoint` is false, it will
simply log an error and continue to process entry-points. This is useful when
ngcc is processing the entire node_modules folder and there are some invalid
entry-points that the project doesn't actually use.
PR Close#36083
Previously, when an entry-point contained code that caused its compilation
to fail, ngcc would exit in the middle of processing, possibly leaving other
entry-points in a corrupt state.
This change adds a new `errorOnFailedEntryPoint` option to `mainNgcc` that
specifies whether ngcc should exit immediately or log an error and continue
processing other entry-points.
The default is `false` so that ngcc will not error but continue processing
as much as possible. This is useful in post-install hooks, and async CLI
integration, where we do not have as much control over which entry-points
should be processed.
The option is forced to true if the `targetEntryPointPath` is provided,
such as the sync integration with the CLI, since in that case it is targeting
an entry-point that will actually be used in the current project so we do want
ngcc to exit with an error at that point.
PR Close#36083
Later when we implement the ability to continue processing when tasks have
failed to compile, we will also need to avoid processing tasks that depend
upon the failed task.
This refactoring exposes this list of dependent tasks in a way that can be
used to skip processing of tasks that depend upon a failed task.
It also changes the blocking model of the parallel mode of operation so
that non-typings tasks are now blocked on their corresponding typings task.
Previously the non-typings tasks could be triggered to run in parallel to
the typings task, since they do not have a hard dependency on each other,
but this made it difficult to skip task correctly if the typings task failed,
since it was possible that a non-typings task was already in flight when
the typings task failed. The result of this is a small potential degradation
of performance in async parallel processing mode, in the rare cases that
there were not enough unblocked tasks to make use of all the available
workers.
PR Close#36083
Moving the definition of the `onTaskCompleted` callback into `mainNgcc()`
allows it to be configured based on options passed in there more easily.
This will be the case when we want to configure whether to log or throw
an error for tasks that failed to be processed successfully.
This commit also creates two new folders and moves the code around a bit
to make it easier to navigate the code§:
* `execution/tasks`: specific helpers such as task completion handlers
* `execution/tasks/queues`: the `TaskQueue` implementations and helpers
PR Close#36083
When ngcc is compiling an entry-point, it uses a `ReflectionHost` that
is specific to its format, e.g. ES2015, ES5, UMD or CommonJS. During the
compilation of that entry-point however, the reflector may be used to
reflect into external libraries using their declaration files.
Up until now this was achieved by letting all `ReflectionHost` classes
consider their parent class for reflector queries, thereby ending up in
the `TypeScriptReflectionHost` that is a common base class for all
reflector hosts. This approach has proven to be prone to bugs, as
failing to call into the base class would cause incompatibilities with
reading from declaration files.
The observation can be made that there's only two distinct kinds of
reflection host queries:
1. the reflector query is about code that is part of the entry-point
that is being compiled, or
2. the reflector query is for an external library that the entry-point
depends on, in which case the information is reflected
from the declaration files.
The `ReflectionHost` that was chosen for the entry-point should serve
only reflector queries for the first case, whereas a regular
`TypeScriptReflectionHost` should be used for the second case. This
avoids the problem where a format-specific `ReflectionHost` fails to
handle the second case correctly, as it isn't even considered for such
reflector queries.
This commit introduces a `ReflectionHost` that delegates to the
`TypeScriptReflectionHost` for AST nodes within declaration files,
otherwise delegating to the format-specific `ReflectionHost`.
Fixes#35078
Resolves FW-1859
PR Close#36089
The format property for ES5 bundles should be "module" or "es5"/"esm5",
but was "main" instead. The "main" property is appropriate for CommonJS
and UMD bundles, not for ES5 bundles.
PR Close#36089
Previously, calculations related to the position of and difference between
SegmentMarkers required extensive computation based around the line,
line start positions and columns of each segment.
PR Close#36027
The merging algorithm needs to find, for a given segment, what the next
segment in the source file is. This change modifies the `generatedSegment`
properties in the mappings so that they have a link directly to the following
segment.
PR Close#36027
By computing and caching the start of each line, rather than the length
of each line, we can save a lot of duplicated computation in the `segmentDiff()`
and `offsetSegment()` functions.
PR Close#36027
Previously the list of original segments that was searched for incoming
mappings did not differentiate between different original source files.
Now there is a separate array of segments to search for each of the
original source files.
PR Close#36027
The `@angular/core` package has a large number of source files
and mappings which exposed performance issues in the new source-map
flattening algorithm.
This change uses a binary search (rather than linear) when finding
matching mappings to merge. Initial measurements indicate that this
reduces processing time for `@angular/core` by about 50%.
PR Close#36027
In some scenarios it is useful for the developer to indicate
to ngcc that it should not use the entry-point manifest
file, and instead write a new one.
In the ngcc command line tool, this option is set by specfying
```
--invalidate-entry-point-manifest
```
PR Close#35931
The `DirectoryWalkerEntryPointFinder` has to traverse the
entire node_modules library everytime it executes in order to
identify the entry-points that need to be processed. This is
very time consuming (several seconds for big projects on
Windows).
This commit changes the `DirectoryWalkerEntryPointFinder` to
use the `EntryPointManifest` to store the paths to entry-points
that were found when doing this initial node_modules traversal
in a file to be reused for subsequent calls.
This dramatically speeds up ngcc processing when it has been run once
already.
PR Close#35931
The new `EntryPointManifest` class can read and write a
manifest file that contains all the paths to the entry-points
that have been found in a node_modules folder.
This can be used to speed up finding entry-points in
subsequent runs.
The manifest file stores the ngcc version and hashes of
the package lock-file and project config, since if these
change the manifest will need to be recomputed.
PR Close#35931
Currently, when running the ngcc binary directly and provide an invalid option ngcc will not error out and the user might have a hard time telling why ngcc is behaving not as expected.
With this change we now output an actionable error:
```
yarn ngcc --unknown-option
Options:
--version Show version number [boolean]
-s, --source A path (relative to the working directory)
of the `node_modules` folder to process.
[default: "./node_modules"]
-p, --properties An array of names of properties in
package.json to compile (e.g. `module` or
`es2015`)
Each of these properties should hold the
path to a bundle-format.
If provided, only the specified properties
are considered for processing.
If not provided, all the supported format
properties (e.g. fesm2015, fesm5, es2015,
esm2015, esm5, main, module) in the
package.json are considered. [array]
-t, --target A relative path (from the `source` path) to
a single entry-point to process (plus its
dependencies).
--first-only If specified then only the first matching
package.json property will be compiled.
[boolean]
--create-ivy-entry-points If specified then new `*_ivy_ngcc`
entry-points will be added to package.json
rather than modifying the ones in-place.
For this to work you need to have custom
resolution set up (e.g. in webpack) to look
for these new entry-points.
The Angular CLI does this already, so it is
safe to use this option if the project is
being built via the CLI. [boolean]
--legacy-message-ids Render `$localize` messages with legacy
format ids.
The default value is `true`. Only set this
to `false` if you do not want legacy
message ids to
be rendered. For example, if you are not
using legacy message ids in your
translation files
AND are not doing compile-time inlining of
translations, in which case the extra
message ids
would add unwanted size to the final source
bundle.
It is safe to leave this set to true if you
are doing compile-time inlining because the
extra
legacy message ids will all be stripped
during translation.
[boolean] [default: true]
--async Whether to compile asynchronously. This is
enabled by default as it allows
compilations to be parallelized.
Disabling asynchronous compilation may be
useful for debugging.
[boolean] [default: true]
-l, --loglevel The lowest severity logging message that
should be output.
[choices: "debug", "info", "warn", "error"]
--invalidate-entry-point-manifest If this is set then ngcc will not read an
entry-point manifest file from disk.
Instead it will walking the directory tree
as normal looking for entry-points, and
then write a new manifest file.
[boolean] [default: false]
--help Show help [boolean]
Unknown arguments: unknown-option, unknownOption
```
PR Close#36010
This commit adds support in the Angular monorepo and in the Angular
compiler(s) for TypeScript 3.8. All packages can now compile with
TS 3.8.
For most of the repo, only a handful few typings adjustments were needed:
* TS 3.8 has a new `CustomElementConstructor` DOM type, which enforces a
zero-argument constructor. The `NgElementConstructor` type previously
declared a required `injector` argument despite the fact that its
implementation allowed `injector` to be optional. The interface type was
updated to reflect the optionality of the argument.
* Certain error messages were changed, and expectations in tests were
updated as a result.
* tsserver (part of language server) now returns performance information in
responses, so test expectations were changed to only assert on the actual
body content of responses.
For compiler-cli and schematics (which use the TypeScript AST) a major
breaking change was the introduction of the export form:
```typescript
export * as foo from 'bar';
```
This is a `ts.NamespaceExport`, and the `exportClause` of a
`ts.ExportDeclaration` can now take this type as well as `ts.NamedExports`.
This broke a lot of places where `exportClause` was assumed to be
`ts.NamedExports`.
For the most part these breakages were in cases where it is not necessary
to handle the new `ts.NamedExports` anyway. ngtsc's design uses the
`ts.TypeChecker` APIs to understand syntax and so automatically supports the
new form of exports.
The View Engine compiler on the other hand extracts TS structures into
metadata.json files, and that format was not designed for namespaced
exports. As a result it will take a nontrivial amount of work if we want to
support such exports in View Engine. For now, these new exports are not
accounted for in metadata.json, and so using them in "folded" Angular
expressions will result in errors (probably claiming that the referenced
exported namespace doesn't exist).
Care was taken to only use TS APIs which are present in 3.7/3.6, as Angular
needs to remain compatible with these for the time being.
This commit does not update angular.io.
PR Close#35864
This version of `LockFile` creates an "unlocker" child-process that monitors
the main ngcc process and deletes the lock file if it exits unexpectedly.
This resolves the issue where the main process could not be killed by pressing
Ctrl-C at the terminal.
Fixes#35761
PR Close#35861
The previous implementation mixed up the management
of locking a piece of code (both sync and async) with the
management of writing and removing the lockFile that is
used as the flag for which process has locked the code.
This change splits these two concepts up. Apart from
avoiding the awkward base class it allows the `LockFile`
implementation to be replaced cleanly.
PR Close#35861
This reduces the time that `findEntryPoints` takes from 9701.143ms to 4177.278ms, by reducing the file operations done.
Reference: #35717
PR Close#35756
With this change we spawn workers lazily based on the amount of work that needs to be done.
Before this change we spawned the maximum of workers possible. However, in some cases there are less tasks than the max number of workers which resulted in created unnecessary workers
Reference: #35717
PR Close#35719
Source-maps in the wild could be badly formatted,
causing the source-map flattening processing to fail
unexpectedly. Rather than causing the whole of ngcc
to crash, we gracefully fallback to just returning the
generated source-map instead.
PR Close#35718
Previously when rendering flattened source-maps, it was assumed that no
mapping would come from a line that is outside the lines of the actual
source content. It turns out this is not a valid assumption.
Now the code that renders flattened source-maps will handle such
mappings, with the additional benefit that the rendered source-map
will only contain mapping lines up to the last mapping, rather than a
mapping line for every content line.
Fixes#35709
PR Close#35718
If a package has a source-map but it does not provide
the actual content of the sources, then the source-map
flattening was crashing.
Now we ignore such mappings that have no source
since we are not able to compute the merged
mapping if there is no source file.
Fixes#35709
PR Close#35718
This commit adds a new ngcc configuration, `ignorableDeepImportMatchers`
for packages. This is a list of regular expressions matching deep imports
that can be safely ignored from that package. Deep imports that are not
ignored cause a warning to be logged.
// FW-1892
Fixes#35615
PR Close#35683
The library used by ngcc to update the source files (MagicString) is able
to generate a source-map but it is not able to account for any previous
source-map that the input text is already associated with.
There have been various attempts to fix this but none have been very
successful, since it is not a trivial problem to solve.
This commit contains a novel approach that is able to load up a tree of
source-files connected by source-maps and flatten them down into a single
source-map that maps directly from the final generated file to the original
sources referenced by the intermediate source-maps.
PR Close#35132
Previously if there were two path-mapped libraries that are in
different directories but the path of one started with same string
as the path of the other, we would incorrectly return the shorter
path - e.g. `dist/my-lib` and `dist/my-lib-second`. This was because
the list of `basePaths` was searched in ascending alphabetic order and
we were using `startsWith()` to match the path.
Now the `basePaths` are searched in reverse alphabetic order so the
longer path will be matched correctly.
// FW-1873
Fixes#35536
PR Close#35592
In ES5 code, TypeScript requires certain helpers (such as
`__spreadArrays()`) to be able to support ES2015+ features. These
helpers can be either imported from `tslib` (by setting the
`importHelpers` TS compiler option to `true`) or emitted inline (by
setting the `importHelpers` and `noEmitHelpers` TS compiler options to
`false`, which is the default value for both).
Ngtsc's `StaticInterpreter` (which is also used during ngcc processing)
is able to statically evaluate some of these helpers (currently
`__assign()`, `__spread()` and `__spreadArrays()`), as long as
`ReflectionHost#getDefinitionOfFunction()` correctly detects the
declaration of the helper. For this to happen, the left-hand side of the
corresponding call expression (i.e. `__spread(...)` or
`tslib.__spread(...)`) must be evaluated as a function declaration for
`getDefinitionOfFunction()` to be called with.
In the case of imported helpers, the `tslib.__someHelper` expression was
resolved to a function declaration of the form
`export declare function __someHelper(...args: any[][]): any[];`, which
allows `getDefinitionOfFunction()` to correctly map it to a TS helper.
In contrast, in the case of emitted helpers (and regardless of the
module format: `CommonJS`, `ESNext`, `UMD`, etc.)), the `__someHelper`
identifier was resolved to a variable declaration of the form
`var __someHelper = (this && this.__someHelper) || function () { ... }`,
which upon further evaluation was categorized as a `DynamicValue`
(prohibiting further evaluation by the `getDefinitionOfFunction()`).
As a result of the above, emitted TypeScript helpers were not evaluated
in ES5 code.
---
This commit changes the detection of TS helpers to leverage the existing
`KnownFn` feature (previously only used for built-in functions).
`Esm5ReflectionHost` is changed to always return `KnownDeclaration`s for
TS helpers, both imported (`getExportsOfModule()`) as well as emitted
(`getDeclarationOfIdentifier()`).
Similar changes are made to `CommonJsReflectionHost` and
`UmdReflectionHost`.
The `KnownDeclaration`s are then mapped to `KnownFn`s in
`StaticInterpreter`, allowing it to statically evaluate call expressions
involving any kind of TS helpers.
Jira issue: https://angular-team.atlassian.net/browse/FW-1689
PR Close#35191
When statically evaluating CommonJS code it is possible to find that we
are looking for the declaration of an identifier that actually came from
a typings file (rather than a CommonJS file).
Previously, the CommonJS reflection host would always try to use a
CommonJS specific algorithm for finding identifier declarations, but
when the id is actually in a typings file this resulted in the returned
declaration being the containing file of the declaration rather than the
declaration itself.
Now the CommonJS reflection host will check to see if the file
containing the identifier is a typings file and use the appropriate
stategy.
(Note: This is the equivalent of #34356 but for CommonJS.)
PR Close#35191
In ES5 and ES2015, class identifiers may have aliases. Previously, the
`NgccReflectionHost`s recognized the following formats:
- ES5:
```js
var MyClass = (function () {
function InnerClass() {}
InnerClass_1 = InnerClass;
...
}());
```
- ES2015:
```js
let MyClass = MyClass_1 = class MyClass { ... };
```
In addition to the above, this commit adds support for recognizing an
alias outside the IIFE in ES5 classes (which was previously not
supported):
```js
var MyClass = MyClass_1 = (function () { ... }());
```
Jira issue: [FW-1869](https://angular-team.atlassian.net/browse/FW-1869)
Partially addresses #35399.
PR Close#35527
`Esm5ReflectionHost#getInnerFunctionDeclarationFromClassDeclaration()`
was already called with `ts.Declaration`, not `ts.Node`, so we can
tighten its parameter type and get rid of a redundant check.
`getIifeBody()` (called inside
`getInnerFunctionDeclarationFromClassDeclaration()`) will check whether
the given `ts.Declaration` is a `ts.VariableDeclaration`.
PR Close#35527
ngcc uses a lockfile to prevent two ngcc instances from executing at the
same time. Previously, if a lockfile was found the current process would
error and exit.
Now, when in async mode, the current process is able to wait for the previous
process to release the lockfile before continuing itself.
PR Close#35131
Support for re-exports in UMD were added in e9fb5fdb8. This commit adds
some tests for re-exports (similar to the ones used for
`CommonJsReflectionHost`).
PR Close#35312
The `TargetedEntryPointFinder` must work out what the
containing package is for each entry-point that it finds.
The logic for doing this was flawed in the case that the
package was in a path-mapped directory and not in a
node_modules folder. This meant that secondary entry-points
were incorrectly setting their own path as the package
path, rather than the primary entry-point path.
Fixes#35188
PR Close#35227
To support parallel CLI builds we instruct developers to pre-process
their node_modules via ngcc at the command line.
Despite doing this ngcc was still trying to set a lock when it was being
triggered by the CLI for packages that are not going to be processed,
since they are not compiled by Angular for instance.
This commit checks whether a target package needs to be compiled
at all before attempting to set the lock.
Fixes#35000
PR Close#35057
If ngcc gets updated to a new version then the artifacts
left in packages that were processed by the previous
version are possibly invalid.
Previously we just errored if we found packages that
had already been processed by an outdated version.
Now we automatically clean the packages that have
outdated artifacts so that they can be reprocessed
correctly with the current ngcc version.
Fixes#35082
PR Close#35079
George Kalpakas
JoostK
Pete Bacon Darwin
JiaLiPassion
Alan Agius
Alex Rickabaugh