Currently, if an ngcc process is killed in a manner that it doesn't clean
up its lock file (or is killed too quickly) the compiler reports that it
is waiting on the PID of a process that doesn't exist, and that it will
wait up to a maximum of N seconds. This PR updates the locking code to
additionally check if the process exists, and if it does not it will
immediately bail out, and print the location of the lock file so a user
may clean it up.
PR Close#37250
Inline source-maps in typings files can impact IDE performance
so ngcc should only add such maps if the original typings file
contains inline source-maps.
Fixes#37324
PR Close#37363
In #37221 we disabled tsickle passes from transforming the tsc output that is used to publish all
Angular framework and components packages (@angular/*).
This change however revealed a bug in the ngc that caused __decorate and __metadata calls to still
be emitted in the JS code even though we don't depend on them.
Additionally it was these calls that caused code in @angular/material packages to fail at runtime
due to circular dependency in the emitted decorator code documeted as
https://github.com/microsoft/TypeScript/issues/27519.
This change partially rolls back #37221 by reenabling the decorator to static fields (static
properties) downleveling.
This is just a temporary workaround while we are also fixing root cause in `ngc` - tracked as
FW-2199.
Resolves FW-2198.
Related to FW-2196
PR Close#37317
As of TypeScript 3.9, the tsc emit is not compatible with Closure
Compiler due to
https://github.com/microsoft/TypeScript/pull/32011.
There is some hope that this will be fixed by a solution like the one
proposed in
https://github.com/microsoft/TypeScript/issues/38374 but currently it's
unclear if / when that will
happen.
Since the Closure support has been somewhat already broken, and the
tsickle pass has been a source
of headaches for some time for Angular packages, we are removing it for
now while we rethink our
strategy to make Angular Closure compatible outside of Google.
This change has no effect on our Closure compatibility within Google
which work well because all the
code is compiled from sources and passed through tsickle.
This change only disables the tsickle pass but doesn't remove it.
A follow up PR should either remove all the traces of tscikle or
re-enable the fixed version.
BREAKING CHANGE: Angular npm packages no longer contain jsdoc comments
to support Closure Compiler's advanced optimizations
The support for Closure compiler in Angular packages has been
experimental and broken for quite some
time.
As of TS3.9 Closure is unusable with the JavaScript emit. Please follow
https://github.com/microsoft/TypeScript/issues/38374 for more
information and updates.
If you used Closure compiler with Angular in the past, you will likely
be better off consuming
Angular packages built from sources directly rather than consuming the
version we publish on npm
which is primarily optimized for Webpack/Rollup + Terser build pipeline.
As a temporary workaround you might consider using your current build
pipeline with Closure flag
`--compilation_level=SIMPLE`. This flag will ensure that your build
pipeline produces buildable and
runnable artifacts, at the cost of increased payload size due to
advanced optimizations being disabled.
If you were affected by this change, please help us understand your
needs by leaving a comment on https://github.com/angular/angular/issues/37234.
PR Close#37221
In ES2015 IIFE wrapped classes, the identifier that would reference the class
of the NgModule may be an alias variable. Previously the `Esm2015ReflectionHost`
was not able to match this alias to the original class declaration. This resulted
in failing to identify some `ModuleWithProviders` functions in such case.
These IIFE wrapped classes were introduced in TypeScript 3.9, which is why
this issue is only recently appearing. Since 9.1.x does not support TS 3.9
there is no reason to backport this commit to that branch.
Fixes#37189
PR Close#37206
To better check that the code is working, this commit gives a
distinct name (`DecoratedWrappedClass_1`) to the "adjacent"
class declaration in the tests.
PR Close#37206
In TypeScript 3.9 some re-export syntaxes have changed to be getter
functions (created by calls to `Object.defineProperty()`) rather than
simple property accessors.
This commit adds support into the CommonJS and UMD reflection hosts
for this style of re-export syntax.
PR Close#36989
In the CommonJS and UMD reflection hosts, the logic for computing the
`viaModule` property of `Declaration` objects was not correct for some
cases when getting the exports of modules.
In these cases it was setting `viaModule` to the path of the local module
rather than `null`.
PR Close#36989
The term `ReexportStatement` is too general for this particular concept.
Here the re-export actually refers to a wildcard where all the module
exports are being re-exported.
When we introduce other re-export statement types later this will be
confusing.
PR Close#36989
Using backtick multiline strings leads to confusing layout
that does not fit with the surrounding indentation. Also it
can lead to test fragility due to automated code formatting.
This commit changes just one set of subject code to use
a more resilient string concatenation approach.
PR Close#36989
After the refactoring of the reflection hosts to accommodate
ES2015 classes wrapped in IIFEs. The same treatment needs to
be applied to the rendering formatters.
PR Close#36989
In TS 3.9, ES2015 output can contain ES classes that are wrapped in an
IIFE. So now ES2015 class declarations can look like one of:
```
class OuterClass1 {}
```
```
let OuterClass = class InnerClass {};
```
```
var AliasClass;
let OuterClass = AliasClass = class InnerClass {};
```
```
let OuterClass = (() => class InnerClass {}};
```
```
var AliasClass;
let OuterClass = AliasClass = (() => class InnerClass {})();
```
```
let OuterClass = (() => {
let AdjacentClass = class InnerClass {};
// ... static properties or decorators attached to `AdjacentClass`
return AdjacentClass;
})();
```
```
var AliasClass;
let OuterClass = AliasClass = (() => {
let AdjacentClass = class InnerClass {};
// ... static properties or decorators attached to `AdjacentClass`
return AdjacentClass;
})();
```
The `Esm5ReflectionHost` already handles slightly different IIFE wrappers
around function-based classes. This can be substantially reused when
fixing `Esm2015ReflectionHost`, since there is a lot of commonality
between the two.
This commit moves code from the `Esm5ReflectionHost` into the `Esm2015ReflectionHost`
and looks to share as much as possible between the two hosts.
PR Close#36989
Previously the path to the unlocker process was being resolved by the
current file-system. In the case that this was a `MockFileSystemWindows`
on a non-Windows operating system, this resulted in an incorrect path
to the entry-point.
Now the path to the entry-point is hand-crafted to avoid being broken by
whatever FileSystem is in use.
PR Close#36989
The previous implementations of `hasBaseClass()` are almost
identical to the implementation of `getBaseClassExpression()`.
There is little benefit in duplicating this code so this refactoring
changes `hasBaseClass()` to just call `getBaseClassExpression()`.
This allows the various hosts that implement this to be simplified.
PR Close#36989
A number of overloads were added to `detectKnownDeclaration()` to
allow it to support `null` being passed through. In practice this could
easily be avoided, which allows the overloads to be removed and the
method signature and implementations to be simplified.
PR Close#36989
Previously in v9, we deprecated the pattern of undecorated base classes
that rely on Angular features. We ran a migration for this in version 9
and will run the same on in version 10 again.
To ensure that projects do not regress and start using the unsupported
pattern again, we report an error in ngtsc if such undecorated classes
are discovered.
We keep the compatibility code enabled in ngcc so that libraries
can be still be consumed, even if they have not been migrated yet.
Resolves FW-2130.
PR Close#36921
As of version 10, libraries following the APF will no longer contain
ESM5 output. Hence, tests in ngcc need to be updated as they currently
rely on the release output of `@angular/core`.
Additionally, we'd need to support in ngcc that the `module`
property of entry-points no longer necessarily refers to
`esm5` output, but instead can also target `esm2015`.
We currently achieve this by checking the path the `module`
property points to. We can do this because as per APF, the
folder name is known for the esm2015 output. Long-term for
more coverage, we want to sniff the format by looking for
known ES2015 constructs in the file `module` refers to.
PR Close#36944
In #36892 the `ModuleWithProviders` type parameter becomes required.
This exposes a bug in ngcc, where it can only handle functions that have a
specific form:
```
function forRoot() {
return { ... };
}
```
In other words, it only accepts functions that return an object literal.
In some libraries, the function instead returns a call to another function.
For example in `angular-in-memory-web-api`:
```
InMemoryWebApiModule.forFeature = function (dbCreator, options) {
return InMemoryWebApiModule_1.forRoot(dbCreator, options);
};
```
This commit changes the parsing of such functions to use the
`PartialEvaluator`, which can evaluate these more complex function
bodies.
PR Close#36948
Previously this method was implemented on the `NgccReflectionHost`,
but really it is asking too much of the host, since it actually needs to do
some static evaluation of the code to be able to support a wider range
of function shapes. Also there was only one implementation of the method
in the `Esm2015ReflectionHost` since it has no format specific code in
in.
This commit moves the whole function (and supporting helpers) into the
`ModuleWithProvidersAnalyzer`, which is the only place it was being used.
This class will be able to do further static evaluation of the function bodies
in order to support more function shapes than the host can do on its own.
The commit removes a whole set of reflection host tests but these are
already covered by the tests of the analyzer.
PR Close#36948
This optimization builds on a lot of prior work to finally make type-
checking of templates incremental.
Incrementality requires two main components:
- the ability to reuse work from a prior compilation.
- the ability to know when changes in the current program invalidate that
prior work.
Prior to this commit, on every type-checking pass the compiler would
generate new .ngtypecheck files for each original input file in the program.
1. (Build #1 main program): empty .ngtypecheck files generated for each
original input file.
2. (Build #1 type-check program): .ngtypecheck contents overridden for those
which have corresponding components that need type-checked.
3. (Build #2 main program): throw away old .ngtypecheck files and generate
new empty ones.
4. (Build #2 type-check program): same as step 2.
With this commit, the `IncrementalDriver` now tracks template type-checking
_metadata_ for each input file. The metadata contains information about
source mappings for generated type-checking code, as well as some
diagnostics which were discovered at type-check analysis time. The actual
type-checking code is stored in the TypeScript AST for type-checking files,
which is now re-used between programs as follows:
1. (Build #1 main program): empty .ngtypecheck files generated for each
original input file.
2. (Build #1 type-check program): .ngtypecheck contents overridden for those
which have corresponding components that need type-checked, and the
metadata registered in the `IncrementalDriver`.
3. (Build #2 main program): The `TypeCheckShimGenerator` now reuses _all_
.ngtypecheck `ts.SourceFile` shims from build #1's type-check program in
the construction of build #2's main program. Some of the contents of
these files might be stale (if a component's template changed, for
example), but wholesale reuse here prevents unnecessary changes in the
contents of the program at this point and makes TypeScript's job a lot
easier.
4. (Build #2 type-check program): For those input files which have not
"logically changed" (meaning components within are semantically the same
as they were before), the compiler will re-use the type-check file
metadata from build #1, and _not_ generate a new .ngtypecheck shim.
For components which have logically changed or where the previous
.ngtypecheck contents cannot otherwise be reused, code generation happens
as before.
PR Close#36211
This function needs to deduplicate the paths that are found from the
paths mappings. Previously this deduplication was not linear and also
called the expensive `relative()` function many times.
This commit, suggested by @JoostK, reduces the complexity of the deduplication
by using a tree structure built from the segments of each path.
PR Close#36881
Previously the `basePaths` were computed when the finder was instantiated.
This was a waste of effort in the case that the targeted entry-point is already
processed.
This change makes the computation of `basePaths` lazy, so that the work is
only done if they are actually needed.
Fixes#36874
PR Close#36881
In TS 3.9 the compiler will start to wrap ES2015 classes in an IIFE to help with
tree-shaking when the class has "associated" statements.
E.g.
```ts
let PlatformLocation = /** @class */ (() => {
...
class PlatformLocation {
}
...
return PlatformLocation;
})();
```
This commit updates `Esm2015ReflectionHost` to support this format.
PR Close#36884
Previously the `AsyncLocker` was configured to only wait
50x500ms before timing out. This is 25secs, which is often
less than a normal run of ngcc, so the chance of a timeout
flake was quite high.
The default is now 500x500ms, which is 250secs. If this is
too high for some projects then it can be changed via the
`ngcc.config.js` project file.
PR Close#36838
The commit adds support to the ngcc.config.js file for setting the
`retryAttempts` and `retryDelay` options for the `AsyncLocker`.
An integration test adds a new check for a timeout and actually uses the
ngcc.config.js to reduce the timeout time to prevent the test from taking
too long to complete.
PR Close#36838
Strictly this method only returns config for packages. So this commit
renames it to `getPackageConfig()`, which frees us up to add other
"getXxxxConfig()` methods later.
PR Close#36838
This test is basically duplicated (and slightly enhanced) in the
following test. So it is superfluous. (I suspect it was the result
of a broken rebase.)
PR Close#36838
When ngcc fails due to a timeout waiting for another process
to complete, it was not failing with a unique exit code, so that it
was not possible to know if the process can be restarted; compared to
ngcc failing for some more fatal reason.
Now if ngcc exits because of a timeout, the exit code will be 177.
PR Close#36838
When ngcc is having to pause and wait for another process
it provides a message to the user. This commit adds the extra
information about how to remove the lockfile if desired, since
this message is not shown if you Ctrl-C out of the process before
the timeout period ends.
PR Close#36838
Now that `ngcc/src/ngcc_options` imports `FileWriter` type, there is a
circular dependency detected by the `ts-circular-deps:check` lint check:
```
ngcc/src/ngcc_options.ts
→ ngcc/src/writing/file_writer.ts
→ ngcc/src/packages/entry_point_bundle.ts
→ ngcc/src/ngcc_options.ts
```
This commit moves the `PathMappings` type (and related helpers) to a
separate file to avoid the circular dependency.
NOTE:
The circular dependency was only with taking types into account. There
was no circular dependency for the actual (JS) code.
PR Close#36626
When running in parallel mode, worker processes forward errors thrown
during task processing to the master process, which in turn exits with
an error.
However, there are cases where the error is not directly related to
processing the entry-point. One such case is when there is not enough
memory (for example, due to all the other tasks being processed
simultaneously).
Previously, an `ENOMEM` error thrown on a worker process would propagate
to the master process, eventually causing ngcc to exit with an error.
Example failure: https://circleci.com/gh/angular/angular/682198
This commit improves handling of these low-memory situations by
detecting `ENOMEM` errors and killing the worker process, thus allowing
the master process to decide how to handle that. The master process will
put the task back into the tasks queue and continue processing tasks
with the rest of the worker processes (and thus with lower memory
pressure).
PR Close#36626
Previously, when the last worker process crashed, the master process
would try to re-spawn it indefinitely. This could lead to an infinite
loop (if for some reason the worker process kept crashing).
This commit avoids this by limiting the number of re-spawn attempts to
3, after which ngcc will exit with an error.
PR Close#36626
Previously, when running in parallel mode and a worker process crashed
while processing a task, it was not possible for ngcc to continue
without risking ending up with a corrupted entry-point and therefore it
exited with an error. This, for example, could happen when a worker
process received a `SIGKILL` signal, which was frequently observed in CI
environments. This was probably the result of Docker killing processes
due to increased memory pressure.
One factor that amplifies the problem under Docker (which is often used
in CI) is that it is not possible to distinguish between the available
CPU cores on the host machine and the ones made available to Docker
containers, thus resulting in ngcc spawning too many worker processes.
This commit addresses these issues in the following ways:
1. We take advantage of the fact that files are written to disk only
after an entry-point has been fully analyzed/compiled. The master
process can now determine whether a worker process has not yet
started writing files to disk (even if it was in the middle of
processing a task) and just put the task back into the tasks queue if
the worker process crashes.
2. The master process keeps track of the transformed files that a worker
process will attempt to write to disk. If the worker process crashes
while writing files, the master process can revert any changes and
put the task back into the tasks queue (without risking corruption).
3. When a worker process crashes while processing a task (which can be a
result of increased memory pressure or too many worker processes),
the master process will not try to re-spawn it. This way the number
or worker processes is gradually adjusted to a level that can be
accomodated by the system's resources.
Examples of ngcc being able to recover after a worker process crashed:
- While idling: https://circleci.com/gh/angular/angular/682197
- While compiling: https://circleci.com/gh/angular/angular/682209
- While writing files: https://circleci.com/gh/angular/angular/682267
Jira issue: [FW-2008](https://angular-team.atlassian.net/browse/FW-2008)
Fixes#36278
PR Close#36626
This commit adds a `revertFile()` method to `FileWriter`, which can
revert a transformed file (and its backup - if any) written by the
`FileWriter`.
In a subsequent commit, this will be used to allow ngcc to recover
when a worker process crashes in the middle of processing a task.
PR Close#36626
With this commit, the master process will keep track of the transformed
files that each worker process is intending to write to disk.
In a subsequent commit, this info will be used to allow ngcc to recover
when a worker process crashes in the middle of processing a task.
PR Close#36626
With this commit, worker processes will notify the master process about
the transformed files they are about to write to disk before starting
writing them.
In a subsequent commit, this will be used to allow ngcc to recover when
a worker process crashes in the middle of processing a task.
PR Close#36626
This commit enhances the `CompileFn`, which is used to process each
entry-point, to support running a passed-in callback (and wait for it to
complete) before proceeding with writing the transformed files to disk.
This functionality is currently not used. In a subsequent commit, it
will be used for passing info from worker processes to the master
process that will allow ngcc to recover when a worker process crashes in
the middle of processing a task.
PR Close#36626
Rename the `markTaskCompleted()` method to be consistent with the other
similar methods of `TaskQueue` (`markAsFailed()` and
`markAsUnprocessed()`).
PR Close#36626
This commit adds support for stopping processing an in-progress task
and moving it back to the list of pending tasks.
In a subsequent commit, this will be used to allow ngcc to recover when
a worker process crashes in the middle of processing a task.
PR Close#36626
Previously, ngcc would run in parallel mode (using the
`ClusterExecutor`) when there were at least 2 CPU cores (and all other
requirements where met). On systems with just 2 CPU cores, this meant
there would only be one worker process (since one CPU core is always
reserved for the master process). In these cases, the tasks would still
be processed serially (on the one worker process), but we would also pay
the overhead of communicating between the master and worker processes.
This commit fixes this by only running in parallel mode if there are
more than 2 CPU cores (i.e. at least 2 worker processes).
PR Close#36626
Previously, the "Compiling <entryPoint>" log message was printed before
starting to analyze and transform files, but after creating the
`EntryPointBundle` (which includes creating the TS program).
Since creating the `EntryPointBundle` involves some work, it is more
accurate to move the log message before creating the bundle.
PR Close#36626
An enum declaration in TypeScript code will be emitted into JavaScript
as a regular variable declaration, with the enum members being declared
inside an IIFE. For ngcc to support interpreting such variable
declarations as enum declarations with its members, ngcc needs to
recognize the enum declaration emit structure and extract all member
from the statements in the IIFE.
This commit extends the `ConcreteDeclaration` structure in the
`ReflectionHost` abstraction to be able to capture the enum members
on a variable declaration, as a substitute for the original
`ts.EnumDeclaration` as it existed in TypeScript code. The static
interpreter has been extended to handle the extracted enum members
as it would have done for `ts.EnumDeclaration`.
Fixes#35584
Resolves FW-2069
PR Close#36550
The html parser already normalizes line endings (converting `\r\n` to `\n`)
for most text in templates but it was missing the expressions of ICU expansions.
In ViewEngine backticked literal strings, used to define inline templates,
were already normalized by the TypeScript parser.
In Ivy we are parsing the raw text of the source file directly so the line
endings need to be manually normalized.
This change ensures that inline templates have the line endings of ICU
expression normalized correctly, which matches the ViewEngine.
In ViewEngine external templates, defined in HTML files, the behavior was
different, since TypeScript was not normalizing the line endings.
Specifically, ICU expansion "expressions" are not being normalized.
This is a problem because it means that i18n message ids can be different on
different machines that are setup with different line ending handling,
or if the developer moves a template from inline to external or vice versa.
The goal is always to normalize line endings, whether inline or external.
But this would be a breaking change since it would change i18n message ids
that have been previously computed. Therefore this commit aligns the ivy
template parsing to have the same "buggy" behavior for external templates.
There is now a compiler option `i18nNormalizeLineEndingsInICUs`, which
if set to `true` will ensure the correct non-buggy behavior. For the time
being this option defaults to `false` to ensure backward compatibility while
allowing opt-in to the desired behavior. This option's default will be
flipped in a future breaking change release.
Further, when this option is set to `false`, any ICU expression tokens,
which have not been normalized, are added to the `ParseResult` from the
`HtmlParser.parse()` method. In the future, this collection of tokens could
be used to diagnose and encourage developers to migrate their i18n message
ids. See FW-2106.
Closes#36725
PR Close#36741
The cached file-system was implemented to speed up ngcc
processing, but in reality most files are not accessed many times
and there is no noticeable degradation in speed by removing it.
Benchmarking `ngcc -l debug` for AIO on a local machine
gave a range of 196-236 seconds with the cache and 197-224
seconds without the cache.
Moreover, when running in parallel mode, ngcc has a separate
file cache for each process. This results in excess memory usage.
Notably the master process, which only does analysis of entry-points
holds on to up to 500Mb for AIO when using the cache compared to
only around 30Mb when not using the cache.
Finally, the file-system cache being incorrectly primed with file
contents before being processed has been the cause of a number
of bugs. For example https://github.com/angular/angular-cli/issues/16860#issuecomment-614694269.
PR Close#36687
Terence D. Honles
Pete Bacon Darwin
Igor Minar
Joey Perrott
Alan Agius
Paul Gschwendtner
Alex Rickabaugh
George Kalpakas
JoostK