When ngcc is having to pause and wait for another process
it provides a message to the user. This commit adds the extra
information about how to remove the lockfile if desired, since
this message is not shown if you Ctrl-C out of the process before
the timeout period ends.
PR Close#36838
Now that `ngcc/src/ngcc_options` imports `FileWriter` type, there is a
circular dependency detected by the `ts-circular-deps:check` lint check:
```
ngcc/src/ngcc_options.ts
→ ngcc/src/writing/file_writer.ts
→ ngcc/src/packages/entry_point_bundle.ts
→ ngcc/src/ngcc_options.ts
```
This commit moves the `PathMappings` type (and related helpers) to a
separate file to avoid the circular dependency.
NOTE:
The circular dependency was only with taking types into account. There
was no circular dependency for the actual (JS) code.
PR Close#36626
When running in parallel mode, worker processes forward errors thrown
during task processing to the master process, which in turn exits with
an error.
However, there are cases where the error is not directly related to
processing the entry-point. One such case is when there is not enough
memory (for example, due to all the other tasks being processed
simultaneously).
Previously, an `ENOMEM` error thrown on a worker process would propagate
to the master process, eventually causing ngcc to exit with an error.
Example failure: https://circleci.com/gh/angular/angular/682198
This commit improves handling of these low-memory situations by
detecting `ENOMEM` errors and killing the worker process, thus allowing
the master process to decide how to handle that. The master process will
put the task back into the tasks queue and continue processing tasks
with the rest of the worker processes (and thus with lower memory
pressure).
PR Close#36626
Previously, when the last worker process crashed, the master process
would try to re-spawn it indefinitely. This could lead to an infinite
loop (if for some reason the worker process kept crashing).
This commit avoids this by limiting the number of re-spawn attempts to
3, after which ngcc will exit with an error.
PR Close#36626
Previously, when running in parallel mode and a worker process crashed
while processing a task, it was not possible for ngcc to continue
without risking ending up with a corrupted entry-point and therefore it
exited with an error. This, for example, could happen when a worker
process received a `SIGKILL` signal, which was frequently observed in CI
environments. This was probably the result of Docker killing processes
due to increased memory pressure.
One factor that amplifies the problem under Docker (which is often used
in CI) is that it is not possible to distinguish between the available
CPU cores on the host machine and the ones made available to Docker
containers, thus resulting in ngcc spawning too many worker processes.
This commit addresses these issues in the following ways:
1. We take advantage of the fact that files are written to disk only
after an entry-point has been fully analyzed/compiled. The master
process can now determine whether a worker process has not yet
started writing files to disk (even if it was in the middle of
processing a task) and just put the task back into the tasks queue if
the worker process crashes.
2. The master process keeps track of the transformed files that a worker
process will attempt to write to disk. If the worker process crashes
while writing files, the master process can revert any changes and
put the task back into the tasks queue (without risking corruption).
3. When a worker process crashes while processing a task (which can be a
result of increased memory pressure or too many worker processes),
the master process will not try to re-spawn it. This way the number
or worker processes is gradually adjusted to a level that can be
accomodated by the system's resources.
Examples of ngcc being able to recover after a worker process crashed:
- While idling: https://circleci.com/gh/angular/angular/682197
- While compiling: https://circleci.com/gh/angular/angular/682209
- While writing files: https://circleci.com/gh/angular/angular/682267
Jira issue: [FW-2008](https://angular-team.atlassian.net/browse/FW-2008)
Fixes#36278
PR Close#36626
This commit adds a `revertFile()` method to `FileWriter`, which can
revert a transformed file (and its backup - if any) written by the
`FileWriter`.
In a subsequent commit, this will be used to allow ngcc to recover
when a worker process crashes in the middle of processing a task.
PR Close#36626
With this commit, the master process will keep track of the transformed
files that each worker process is intending to write to disk.
In a subsequent commit, this info will be used to allow ngcc to recover
when a worker process crashes in the middle of processing a task.
PR Close#36626
With this commit, worker processes will notify the master process about
the transformed files they are about to write to disk before starting
writing them.
In a subsequent commit, this will be used to allow ngcc to recover when
a worker process crashes in the middle of processing a task.
PR Close#36626
This commit enhances the `CompileFn`, which is used to process each
entry-point, to support running a passed-in callback (and wait for it to
complete) before proceeding with writing the transformed files to disk.
This functionality is currently not used. In a subsequent commit, it
will be used for passing info from worker processes to the master
process that will allow ngcc to recover when a worker process crashes in
the middle of processing a task.
PR Close#36626
Rename the `markTaskCompleted()` method to be consistent with the other
similar methods of `TaskQueue` (`markAsFailed()` and
`markAsUnprocessed()`).
PR Close#36626
This commit adds support for stopping processing an in-progress task
and moving it back to the list of pending tasks.
In a subsequent commit, this will be used to allow ngcc to recover when
a worker process crashes in the middle of processing a task.
PR Close#36626
Previously, ngcc would run in parallel mode (using the
`ClusterExecutor`) when there were at least 2 CPU cores (and all other
requirements where met). On systems with just 2 CPU cores, this meant
there would only be one worker process (since one CPU core is always
reserved for the master process). In these cases, the tasks would still
be processed serially (on the one worker process), but we would also pay
the overhead of communicating between the master and worker processes.
This commit fixes this by only running in parallel mode if there are
more than 2 CPU cores (i.e. at least 2 worker processes).
PR Close#36626
Previously, the "Compiling <entryPoint>" log message was printed before
starting to analyze and transform files, but after creating the
`EntryPointBundle` (which includes creating the TS program).
Since creating the `EntryPointBundle` involves some work, it is more
accurate to move the log message before creating the bundle.
PR Close#36626
An enum declaration in TypeScript code will be emitted into JavaScript
as a regular variable declaration, with the enum members being declared
inside an IIFE. For ngcc to support interpreting such variable
declarations as enum declarations with its members, ngcc needs to
recognize the enum declaration emit structure and extract all member
from the statements in the IIFE.
This commit extends the `ConcreteDeclaration` structure in the
`ReflectionHost` abstraction to be able to capture the enum members
on a variable declaration, as a substitute for the original
`ts.EnumDeclaration` as it existed in TypeScript code. The static
interpreter has been extended to handle the extracted enum members
as it would have done for `ts.EnumDeclaration`.
Fixes#35584
Resolves FW-2069
PR Close#36550
The html parser already normalizes line endings (converting `\r\n` to `\n`)
for most text in templates but it was missing the expressions of ICU expansions.
In ViewEngine backticked literal strings, used to define inline templates,
were already normalized by the TypeScript parser.
In Ivy we are parsing the raw text of the source file directly so the line
endings need to be manually normalized.
This change ensures that inline templates have the line endings of ICU
expression normalized correctly, which matches the ViewEngine.
In ViewEngine external templates, defined in HTML files, the behavior was
different, since TypeScript was not normalizing the line endings.
Specifically, ICU expansion "expressions" are not being normalized.
This is a problem because it means that i18n message ids can be different on
different machines that are setup with different line ending handling,
or if the developer moves a template from inline to external or vice versa.
The goal is always to normalize line endings, whether inline or external.
But this would be a breaking change since it would change i18n message ids
that have been previously computed. Therefore this commit aligns the ivy
template parsing to have the same "buggy" behavior for external templates.
There is now a compiler option `i18nNormalizeLineEndingsInICUs`, which
if set to `true` will ensure the correct non-buggy behavior. For the time
being this option defaults to `false` to ensure backward compatibility while
allowing opt-in to the desired behavior. This option's default will be
flipped in a future breaking change release.
Further, when this option is set to `false`, any ICU expression tokens,
which have not been normalized, are added to the `ParseResult` from the
`HtmlParser.parse()` method. In the future, this collection of tokens could
be used to diagnose and encourage developers to migrate their i18n message
ids. See FW-2106.
Closes#36725
PR Close#36741
The cached file-system was implemented to speed up ngcc
processing, but in reality most files are not accessed many times
and there is no noticeable degradation in speed by removing it.
Benchmarking `ngcc -l debug` for AIO on a local machine
gave a range of 196-236 seconds with the cache and 197-224
seconds without the cache.
Moreover, when running in parallel mode, ngcc has a separate
file cache for each process. This results in excess memory usage.
Notably the master process, which only does analysis of entry-points
holds on to up to 500Mb for AIO when using the cache compared to
only around 30Mb when not using the cache.
Finally, the file-system cache being incorrectly primed with file
contents before being processed has been the cause of a number
of bugs. For example https://github.com/angular/angular-cli/issues/16860#issuecomment-614694269.
PR Close#36687
The change in e041ac6f0d
to support sending unlocker process output to the main ngcc
console output prevents messages require that the main process
relinquishes the event-loop to allow the `stdout.on()` handler to
run. This results in none of the messages being written when ngcc
is run in `--no-async` mode, and some messages failing to be
written if the main process is killed (e.g. ctrl-C).
It appears that the problem with Windows and detached processes
is known - see https://github.com/nodejs/node/issues/3596#issuecomment-250890218.
But in the meantime, this commit is a workaround, where non-Windows
`inherit` the main process `stdout` while on Windows it reverts
to the async handler approach, which is better than nothing.
PR Close#36637
On Windows, the output of a detached process (such as the unlocker
process used by `LockFileWithChildProcess`) is not shown in the parent
process' stdout.
This commit addresses this by piping the spawned process' stdin/stdout
and manually writing to the parent process' stdout.
PR Close#36569
The current ngcc lock-file strategy spawns a new process in order to
capture a potential `SIGINT` and remove the lock-file. For more
information see #35861.
Previously, this unlocker process was spawned as soon as the `LockFile`
was instantiated in order to have it available as soon as possible
(given that spawning a process is an asynchronous operation). Since the
`LockFile` was instantiated and passed to the `Executor`, this meant
that an unlocker process was spawned for each cluster worker, when
running ngcc in parallel mode. These processes were not needed, since
the `LockFile` was not used in cluster workers, but we still had to pay
the overhead of each process' own memory and V8 instance.
(NOTE: This overhead was small compared to the memory consumed by ngcc's
normal operations, but still unnecessary.)
This commit avoids the extra processes by only spawning an unlocker
process when running on the cluster master process and not on worker
processes.
PR Close#36569
For some reason (possibly related to async/await promises)
the ngcc process is not exiting when spawned from the CLI
when there has been an error (such as when it timesout waiting
for a lockfile to become free).
Calling `process.exit()` directly fixes this.
Fixes#36616
PR Close#36622
Previously, when we needed to detect whether a file is external to a
package, we only checked whether the relative path to the file from the
package's root started with `..`. This would detect external imports
when the packages were siblings (e.g. peer dependencies or hoisted to
the top of `node_modules/` by the package manager), but would fail to
detect imports from packages located in nested `node_modules/` as
external. For example, importing `node_modules/foo/node_modules/bar`
from a file in `node_modules/foo/` would be considered internal to the
`foo` package.
This could result in processing/analyzing more files than necessary.
More importantly it could lead to errors due to trying to analyze
non-Angular packages that were direct dependencies of Angular packages.
This commit fixes it by also verifying that the relative path to a file
does not start with `node_modules/`.
Jira issue: [FW-2068](https://angular-team.atlassian.net/browse/FW-2068)
Fixes#36526
PR Close#36559
The base path for package and entry-points is known so there is
no need to store these in the file. Also this commit avoids storing
empty arrays unnecessarily.
PR Close#36486
Previously, even if an entry-point did not need to be processed,
ngcc would always parse the files of the entry-point to compute
its dependencies. This can take a lot of time for large node_modules.
Now these dependencies are cached in the entry-point manifest,
and read from there rather than computing them every time.
See https://github.com/angular/angular/issues/36414\#issuecomment-608401834
FW-2047
PR Close#36486
When the compiler needs to convert a type reference to a value
expression, it may encounter a type that refers to a namespaced symbol.
Such namespaces need to be handled specially as there's various forms
available. Consider a namespace named "ns":
1. One can refer to a namespace by itself: `ns`. A namespace is only
allowed to be used in a type position if it has been merged with a
class, but even if this is the case it may not be possible to convert
that type into a value expression depending on the import form. More
on this later (case a below)
2. One can refer to a type within the namespace: `ns.Foo`. An import
needs to be generated to `ns`, from which the `Foo` property can then
be read.
3. One can refer to a type in a nested namespace within `ns`:
`ns.Foo.Bar` and possibly even deeper nested. The value
representation is similar to case 2, but includes additional property
accesses.
The exact strategy of how to deal with these cases depends on the type
of import used. There's two flavors available:
a. A namespaced import like `import * as ns from 'ns';` that creates
a local namespace that is irrelevant to the import that needs to be
generated (as said import would be used instead of the original
import).
If the local namespace "ns" itself is referred to in a type position,
it is invalid to convert it into a value expression. Some JavaScript
libraries publish a value as default export using `export = MyClass;`
syntax, however it is illegal to refer to that value using "ns".
Consequently, such usage in a type position *must* be accompanied by
an `@Inject` decorator to provide an explicit token.
b. An explicit namespace declaration within a module, that can be
imported using a named import like `import {ns} from 'ns';` where the
"ns" module declares a namespace using `declare namespace ns {}`.
In this case, it's the namespace itself that needs to be imported,
after which any qualified references into the namespace are converted
into property accesses.
Before this change, support for namespaces in the type-to-value
conversion was limited and only worked correctly for a single qualified
name using a namespace import (case 2a). All other cases were either
producing incorrect code or would crash the compiler (case 1a).
Crashing the compiler is not desirable as it does not indicate where
the issue is. Moreover, the result of a type-to-value conversion is
irrelevant when an explicit injection token is provided using `@Inject`,
so referring to a namespace in a type position (case 1) could still be
valid.
This commit introduces logic to the type-to-value conversion to be able
to properly deal with all type references to namespaced symbols.
Fixes#36006
Resolves FW-1995
PR Close#36106
In cc4b813e75 the `getBasePaths()`
function was changed to log a warning if a `basePath()` computed from
the `paths` mappings did not exist. It turns out this is a common and
accepted scenario, so we should not log warnings in this case.
Fixes#36518
PR Close#36525
1. update jasmine to 3.5
2. update @types/jasmine to 3.5
3. update @types/jasminewd2 to 2.0.8
Also fix several cases, the new jasmine 3 will help to create test cases correctly,
such as in the `jasmine 2.x` version, the following case will pass
```
expect(1 == 2);
```
But in jsamine 3, the case will need to be
```
expect(1 == 2).toBeTrue();
```
PR Close#34625
Previously, `isRelativePath()` assumed paths are *nix-style. This caused
Windows-style paths (such as `C:\foo\some-package\some-file.js`) to not
be recognized as "relative" imports.
This commit fixes this by using the OS-agnostic `isRooted()` helper and
also accounting for both styles of path delimiters: `/` and `\`
PR Close#36372
When TypeScript downlevels ES2015+ code to ES5, it uses some helper
functions to emulate some ES2015+ features, such as spread syntax. The
TypeScript compiler can be configured to emit these helpers into the
transpiled code (which is controlled by the `noEmitHelpers` option -
false by default). It can also be configured to import these helpers
from the `tslib` module (which is controlled by the `importHelpers`
option - false by default).
While most of the time the helpers will be either emitted or imported,
it is possible that one configures their app to neither emit nor import
them. In that case, the helpers could, for example, be made available on
the global object. This is what `@nativescript/angular`
v9.0.0-next-2019-11-12-155500-01 does. See, for example, [common.js][1].
Ngcc must be able to detect and statically evaluate these helpers.
Previously, it was only able to detect emitted or imported helpers.
This commit adds support for detecting these helpers if they are neither
emitted nor imported. It does this by checking identifiers for which no
declaration (either concrete or inline) can be found against a list of
known TypeScript helper function names.
[1]: https://unpkg.com/browse/@nativescript/angular@9.0.0-next-2019-11-12-155500-01/common.js
PR Close#36418
The source-map flattening was throwing an error when there
is a cyclic dependency between source files and source-maps.
The error was either a custom one describing the cycle, or a
"Maximum call stack size exceeded" one.
Now this is handled more leniently, resulting in a partially loaded
source file (or source-map) and a warning logged.
Fixes#35727Fixes#35757
Fixes https://github.com/angular/angular-cli/issues/17106
Fixes https://github.com/angular/angular-cli/issues/17115
PR Close#36452
Recently we added support for ignoring specified deep-import
warnings by providing sets of regular expressions within the
`ngcc.config.js` file. But this was only working for the project
level configuration.
This commit fixes ngcc so that it will also read these regular
expressions from package level configuration too.
Fixes#35750
PR Close#36423
The `browser` package.json property is now supported to the same
level as `main` - i.e. it is sniffed for UMD, ESM5 and CommonJS.
The `browser` property can also contain an object with file overrides
but this is not supported by ngcc.
Fixes#36062
PR Close#36396
Previously, `main` was only checked for `umd` or `commonjs`
formats. Now if there are `import` or `export` statements in the
source file it will be deemed to be in `esm5` format.
Fixes#35788
PR Close#36396
The `NgccReflectionHost`s have logic for detecting certain known
declarations (such as `Object.assign()` and TypeScript helpers), which
allows the `PartialEvaluator` to evaluate expressions it would not be
able to statically evaluate otherwise.
In #36089, `DelegatingReflectionHost` was introduced, which delegates to
a TypeScript `ReflectionHost` when reflecting on TypeScript files, which
for ngcc's case means `.d.ts` files of dependencies. As a result, ngcc
lost the ability to detect TypeScript helpers imported from `tslib`,
because `DelegatingReflectionHost` was not able to apply the known
declaration detection logic while reflecting on `tslib`'s `.d.ts` files.
This commit fixes this by ensuring `DelegatingReflectionHost` calls the
`NgccReflectionHost`'s `detectKnownDeclaration()` method as necessary,
even when using the TypeScript `ReflectionHost`.
NOTE:
The previous commit exposed a bug in ngcc that was hidden due to the
tests' being inconsistent with how the `ReflectionHost`s are used in the
actual program. The changes in this commit are verified by ensuring the
failing tests are now passing (hence no new tests are added).
PR Close#36284
In #36089, `DelegatingReflectionHost` was introduced. Under the hood, it
delegates another `NgccReflectionHost` in order to reflect over the
program's source files, while using a different TypeScript
`ReflectionHost` to reflect over `.d.ts` files (which is how external
dependencies are represented in the program).
Previously, the `NgccReflectionHost`s were used directly in tests. This
does not exercise them in the way they are exercised in the actual
program, because (when used directly) they will also reflect on `.d.ts`
files too (instead of delegating to the TypeScript `ReflectionHost`).
This could hide bugs that would happen on the actual program.
This commit fixes this by using the `DelegatingReflectionHost` in the
various `NgccReflectionHost` tests.
NOTE:
This change will cause some of the existing tests to start failing.
These failures demonstrate pre-existing bugs in ngcc, that were hidden
due to the tests' being inconsistent with how the `ReflectionHost`s are
used in the actual program. They will be fixed in the next commit.
PR Close#36284
The `NgccReflectionHost`s have logic for detecting certain known
declarations (such as `Object.assign()` and TypeScript helpers), which
allows the `PartialEvaluator` to evaluate expressions it would not be
able to statically evaluate otherwise.
This commit moves the logic for identifying these known declarations to
dedicated methods. This is in preparation of allowing ngcc's
`DelegatingReflectionHost` (introduced in #36089) to also apply the
known declaration detection logic when reflecting on TypeScript sources.
PR Close#36284
Ngcc supports providing a project-level configuration to affect how
certain dependencies are processed and also has a built-in fallback
configuration for some unmaintained packages. Each entry in these
configurations could be scoped to specific versions of a package by
providing a version range. If no version range is provided for a
package, it defaults to `*` (with the intention of matching any
version).
Previously, the installed version of a package was tested against the
version range using the [semver][1] package's `satisfies()` function
with the default options. By default, `satisfies()` does not match
pre-releases (see [here][2] for more details on reasoning). While this
makes sense when determining what version of a dependency to install
(trying to avoid unexpected breaking changes), it is not desired in the
case of ngcc.
This commit fixes it by explicitly specifying that pre-release versions
should be matched normally.
[1]: https://www.npmjs.com/package/semver
[2]: https://github.com/npm/node-semver#prerelease-tags
PR Close#36370
Previously, a bad baseUrl or path mapping passed to an `EntryPointFinder`
could cause the original `sourceDirectory` to be superceded by a higher
directory. This could result in none of the sourceDirectory entry-points being
processed.
Now missing basePaths computed from path-mappings are discarded with
a warning. Further, if the `baseUrl` is the root directory then a warning is
given as this is most likely an error in the tsconfig.json.
Resolves#36313Resolves#36283
PR Close#36331
The previous optimizations in #35756 to the
`DirectoryWalkerEntryPointFinder` were over zealous
with regard to packages that have entry-points stored
in "container" directories in the package, where the
container directory was not an entry-point itself.
Now we will also walk such "container" folders as long
as they do not contain `.js` files, which we regard as an
indicator that the directory will not contain entry-points.
Fixes#36216
PR Close#36305
This commit simplifies the `DirectoryWalkerEntryPointFinder` inter-method
calling to make it easier to follow, and also to support controlling
walking of a directory based on its children.
PR Close#36305
Pete Bacon Darwin
George Kalpakas
JoostK
JiaLiPassion
Alan Agius