Commit Graph

31 Commits

Author SHA1 Message Date
Pete Bacon Darwin 20b0c80b0b fix(ngcc): allow deep-import warnings to be ignored (#35683)
This commit adds a new ngcc configuration, `ignorableDeepImportMatchers`
for packages. This is a list of regular expressions matching deep imports
that can be safely ignored from that package. Deep imports that are not
ignored cause a warning to be logged.

// FW-1892

Fixes #35615

PR Close #35683
2020-02-27 10:48:48 -08:00
Pete Bacon Darwin 3d5bcd5883 test(ngcc): update dependency host test description (#34695)
The `describe` description did not match the name of the
method.

PR Close #34695
2020-01-15 10:24:50 -08:00
Pete Bacon Darwin 85b5c365fc fix(ngcc): do not add DTS deep imports to missing packages list (#34695)
When searching the typings program for a package for imports a
distinction is drawn between missing entry-points and deep imports.

Previously in the `DtsDependencyHost` these deep imports may be
marked as missing if there was no typings file at the deep import path.
Instead there may be a javascript file instead. In practice this means
the import is "deep" and not "missing".

Now the `DtsDependencyHost` will also consider `.js` files when checking
for deep-imports, and it will also look inside `@types/...` for a suitable
deep-imported typings file.

Fixes #34720

PR Close #34695
2020-01-15 10:24:50 -08:00
Pete Bacon Darwin da51d884a1 test(ngcc): remove `declare` from JS classes (#34695)
PR Close #34695
2020-01-15 10:24:49 -08:00
George Kalpakas cfbb1a1e77 fix(ngcc): correctly detect dependencies in CommonJS (#34528)
Previously, `CommonJsDependencyHost.collectDependencies()` would only
find dependencies via imports of the form `var foo = require('...');` or
`var foo = require('...'), bar = require('...');` However, CommonJS
files can have imports in many different forms. By failing to recognize
other forms of imports, the associated dependencies were missed, which
in turn resulted in entry-points being compiled out-of-order and failing
due to that.

While we cannot easily capture all different types of imports, this
commit enhances `CommonJsDependencyHost` to recognize the following
common forms of imports:

- Imports in property assignments. E.g.:
  `exports.foo = require('...');` or
  `module.exports = {foo: require('...')};`

- Imports for side-effects only. E.g.:
  `require('...');`

- Star re-exports (with both emitted and imported heleprs). E.g.:
  `__export(require('...'));` or
  `tslib_1.__exportStar(require('...'), exports);`

PR Close #34528
2020-01-13 09:48:20 -08:00
George Kalpakas eb6e1af46d test(ngcc): fix typos in `CommonJsDependencyHost` tests (to avoid confusion) (#34528)
PR Close #34528
2020-01-13 09:48:20 -08:00
George Kalpakas 07ea6cf582 fix(ngcc): avoid error due to circular dependency in `EsmDependencyHost` (#34512)
Previously, there was circular dependency between `ngcc/src/utils.ts`,
`ngcc/src/dependencies/dependency_host.ts` and
`ngcc/src/dependencies/esm_dependency_host.ts`. More specifically,
`utils.ts` would [import from `esm_dependency_host.ts`][1], which would
[import from `dependency_host.ts`][2], which would in turn
[import from `utils.ts`][3].

This might be fine in some environments/module formats, but it can cause
unclear errors in the transpiled CommonJS/UMD format (given how Node.js
handles [cycles in module resolution][4]).
(An example error can be found [here][5].)

This commit fixes the problem by moving the code that depends on
`EsmDependencyHost` out of `utils.ts` and into a dedicated file under
`dependencies/`. It also converts the `createDtsDependencyHost()`
function to a class for consistency with the rest of the
`DependencyHost`s.

[1]: https://github.com/angular/angular/blob/18d89c9c8/packages/compiler-cli/ngcc/src/utils.ts#L10
[2]: https://github.com/angular/angular/blob/18d89c9c8/packages/compiler-cli/ngcc/src/dependencies/esm_dependency_host.ts#L10
[3]: https://github.com/angular/angular/blob/18d89c9c8/packages/compiler-cli/ngcc/src/dependencies/dependency_host.ts#L9
[4]: https://nodejs.org/api/modules.html#modules_cycles
[5]: https://circleci.com/gh/angular/angular/577581

PR Close #34512
2020-01-08 15:00:50 -08:00
Pete Bacon Darwin 4f42de9704 fix(ngcc): capture entry-point dependencies from typings as well as source (#34494)
ngcc computes a dependency graph of entry-points to ensure that
entry-points are processed in the correct order. Previously only the imports
in source files were analysed to determine the dependencies for each
entry-point.

This is not sufficient when an entry-point has a "type-only" dependency
 - for example only importing an interface from another entry-point.
In this case the "type-only" import does not appear in the
source code. It only appears in the typings files. This can cause a
dependency to be missed on the entry-point.

This commit fixes this by additionally processing the imports in the
typings program, as well as the source program.

Note that these missing dependencies could cause unexpected flakes when
running ngcc in async mode on multiple processes due to the way that
ngcc caches files when they are first read from disk.

Fixes #34411

// FW-1781

PR Close #34494
2020-01-07 10:35:03 -08:00
Pete Bacon Darwin 69950e3888 refactor(ngcc): resolve modules based on the provided `moduleResolver` (#34494)
The `DependencyHost` implementations were duplicating the "postfix" strings
which are used to find matching paths when resolving module specifiers.
Now the hosts reuse the postfixes given to the `ModuleResolver` that is
passed to the host.

PR Close #34494
2020-01-07 10:35:03 -08:00
Pete Bacon Darwin e2b184515b refactor(ngcc): pass dependency info to `collectDependencies()` (#34494)
Rather than return a new object of dependency info from calls to
`collectDependencies()` we now pass in an object that will be updated
with the dependency info. This is in preparation of a change where
we will collect dependency information from more than one
`DependencyHost`.

Also to better fit with this approach the name is changed from
`findDependencies()` to `collectDependencies()`.

PR Close #34494
2020-01-07 10:35:03 -08:00
Alex Rickabaugh 4da2dda647 feat(ngcc): support ignoreMissingDependencies in ngcc config (#33192)
Normally, when ngcc encounters a package with missing dependencies while
attempting to determine a compilation ordering, it will ignore that package.
This commit adds a configuration for a flag to tell ngcc to compile the
package anyway, regardless of any missing dependencies.

FW-1931 #resolve

PR Close #33192
2019-10-17 19:43:39 -04:00
George Kalpakas e36e6c85ef perf(ngcc): process tasks in parallel in async mode (#32427)
`ngcc` supports both synchronous and asynchronous execution. The default
mode when using `ngcc` programmatically (which is how `@angular/cli` is
using it) is synchronous. When running `ngcc` from the command line
(i.e. via the `ivy-ngcc` script), it runs in async mode.

Previously, the work would be executed in the same way in both modes.

This commit improves the performance of `ngcc` in async mode by
processing tasks in parallel on multiple processes. It uses the Node.js
built-in [`cluster` module](https://nodejs.org/api/cluster.html) to
launch a cluster of Node.js processes and take advantage of multi-core
systems.

Preliminary comparisons indicate a 1.8x to 2.6x speed improvement when
processing the angular.io app (apparently depending on the OS, number of
available cores, system load, etc.). Further investigation is needed to
better understand these numbers and identify potential areas of
improvement.

Inspired by/Based on @alxhub's prototype: alxhub/angular@cb631bdb1
Original design doc: https://hackmd.io/uYG9CJrFQZ-6FtKqpnYJAA?view

Jira issue: [FW-1460](https://angular-team.atlassian.net/browse/FW-1460)

PR Close #32427
2019-09-09 15:55:13 -04:00
JoostK 4bbf16e654 fix(ngcc): handle deep imports that already have an extension (#32181)
During the dependency analysis phase of ngcc, imports are resolved to
files on disk according to certain module resolution rules. Since module
specifiers are typically missing extensions, or can refer to index.js
barrel files within a directory, the module resolver attempts several
postfixes when searching for a module import on disk. Module  specifiers
that already include an extension, however, would fail to be resolved as
ngcc's module resolver failed to check the location on disk without
adding any postfixes.

Closes #32097

PR Close #32181
2019-08-19 10:12:03 -07:00
JoostK 57e15fc08b fix(ivy): ngcc - do not consider builtin NodeJS modules as missing (#31872)
ngcc analyzes the dependency structure of the entrypoints it needs to
process, as the compilation of entrypoints is ordering sensitive: any
dependent upon entrypoint must be compiled before its dependees. As part
of the analysis of the dependency graph, it is detected when a
dependency of entrypoint is not installed, in which case that entrypoint
will be marked as ignored.

For libraries that work with Angular Universal to run in NodeJS, imports
into builtin NodeJS modules can be present. ngcc's dependency analyzer
can only resolve imports within the TypeScript compilation, which
builtin modules are not part of. Therefore, such imports would
erroneously cause the entrypoint to become ignored.

This commit fixes the problem by taking the NodeJS builtins into account
when dealing with missing imports.

Fixes #31522

PR Close #31872
2019-08-05 13:06:49 -07:00
JoostK b70746a113 fix(ivy): ngcc - prevent crash when analyzed target is ignored (#31872)
ngcc analyzes the dependency structure of the entrypoints it needs to
process, as the compilation of entrypoints is ordering sensitive: any
dependent upon entrypoint must be compiled before its dependees. As part
of the analysis of the dependency graph, it is detected when a
dependency of entrypoint is not installed, in which case that entrypoint
will be marked as ignored.

When a target entrypoint to compile is provided, it could occur that
given target is considered ignored because one of its dependencies might
be missing. This situation was not dealt with currently, instead
resulting in a crash of ngcc.

This commit prevents the crash by taking the above scenario into account.

PR Close #31872
2019-08-05 13:06:49 -07:00
Pete Bacon Darwin fac20bd8d1 fix(ivy): ngcc - resolve `main` property paths correctly (#31509)
There are two places in the ngcc processing where it needs to load the
content of a file given by a general path:

* when determining the format of an entry-point.
 To do this ngcc uses the value of the relevant property in package.json.
 But in the case of `main` it must parse the contents of the entry-point
 file to decide whether the format is UMD or CommonJS.

* when parsing the source files for dependencies to determine the order in
which compilation must occur. The relative imports in each file are parsed
and followed recursively, looking for external imports.

Previously, we naively assumed that the path would match the file name exactly.
But actually we must consider the standard module resolution conventions.
E.g. the extension (.js) may be missing, or the path may refer to a directory
containing an index.js file.

This commit fixes both places.

This commit now requires the `DependencyHost` instances to check
the existence of more files than before (at worst all the different possible
post-fixes). This should not create a significant performance reduction for
ngcc. Since the results of the checks will be cached, and similar work is
done inside the TS compiler, so what we lose in doing it here, is saved later
in the processing. The main performance loss would be where there are lots
of files that need to be parsed for dependencies that do not end up being
processed by TS. But compared to the main ngcc processing this dependency
parsing is a small proportion of the work done and so should not impact
much on the overall performance of ngcc.

// FW-1444

PR Close #31509
2019-07-12 11:37:35 -04:00
Matias Niemelä db557221bc revert: fix(ivy): ngcc - resolve `main` property paths correctly (#31509)
This reverts commit 103a5b42ec.
2019-07-11 11:51:13 -04:00
Pete Bacon Darwin 103a5b42ec fix(ivy): ngcc - resolve `main` property paths correctly (#31509)
When determining if a `main` path points to a UMD or CommonJS
format, the contents of the file need to be loaded and parsed.

Previously, it was assumed that the path referred to the exact filename,
but did not account for normal module resolution semantics, where the
path may be missing an extension or refer to a directory containing an
`index.js` file.

// FW-1444

PR Close #31509
2019-07-11 11:41:11 -04:00
Pete Bacon Darwin 50c4ec6687 fix(ivy): ngcc - resolve path-mapped modules correctly (#31450)
Non-wild-card path-mappings were not being matched correctly.

Further path-mapped secondary entry-points that
were imported from the associated primary entry-point were not
being martched correctly.

Fixes #31274

PR Close #31450
2019-07-08 10:28:13 -07:00
Pete Bacon Darwin 3788ebb714 fix(ivy): ngcc - don't crash if entry-points have multiple invalid dependencies (#31276)
If an entry-point has missing dependencies then it cannot be
processed and is marked as invalid. Similarly, if an entry-point
has dependencies that have been marked as invalid then that
entry-point too is invalid. In all these cases, ngcc should quietly
ignore these entry-points and continue processing what it can.

Previously, if an entry-point had more than one entry-point that
was transitively invalid then ngcc was crashing rather than
ignoring the entry-point.

PR Close #31276
2019-06-26 08:01:43 -07:00
Pete Bacon Darwin 7186f9c016 refactor(ivy): implement a virtual file-system layer in ngtsc + ngcc (#30921)
To improve cross platform support, all file access (and path manipulation)
is now done through a well known interface (`FileSystem`).

For testing a number of `MockFileSystem` implementations are provided.
These provide an in-memory file-system which emulates operating systems
like OS/X, Unix and Windows.

The current file system is always available via the static method,
`FileSystem.getFileSystem()`. This is also used by a number of static
methods on `AbsoluteFsPath` and `PathSegment`, to avoid having to pass
`FileSystem` objects around all the time. The result of this is that one
must be careful to ensure that the file-system has been initialized before
using any of these static methods. To prevent this happening accidentally
the current file system always starts out as an instance of `InvalidFileSystem`,
which will throw an error if any of its methods are called.

You can set the current file-system by calling `FileSystem.setFileSystem()`.
During testing you can call the helper function `initMockFileSystem(os)`
which takes a string name of the OS to emulate, and will also monkey-patch
aspects of the TypeScript library to ensure that TS is also using the
current file-system.

Finally there is the `NgtscCompilerHost` to be used for any TypeScript
compilation, which uses a given file-system.

All tests that interact with the file-system should be tested against each
of the mock file-systems. A series of helpers have been provided to support
such tests:

* `runInEachFileSystem()` - wrap your tests in this helper to run all the
wrapped tests in each of the mock file-systems.
* `addTestFilesToFileSystem()` - use this to add files and their contents
to the mock file system for testing.
* `loadTestFilesFromDisk()` - use this to load a mirror image of files on
disk into the in-memory mock file-system.
* `loadFakeCore()` - use this to load a fake version of `@angular/core`
into the mock file-system.

All ngcc and ngtsc source and tests now use this virtual file-system setup.

PR Close #30921
2019-06-25 16:25:24 -07:00
Pete Bacon Darwin e20b92ba37 feat(ivy): ngcc - turn on CommonJS support (#30200)
PR Close #30200
2019-05-22 16:24:15 -07:00
Pete Bacon Darwin 1bdec3ed6a feat(ivy): ngcc - implement CommonJsDependencyHost (#30200)
PR Close #30200
2019-05-22 16:24:15 -07:00
Pete Bacon Darwin 73e3f565e0 test(ivy): ngcc - fix tests to work on Windows (#30520)
PR Close #30520
2019-05-16 13:32:02 -07:00
Pete Bacon Darwin edd775eabc feat(ivy): ngcc - implement UmdDependencyHost (#25445)
The dependency resolution that works out the order in which
to process entry-points must also understand UMD formats.

PR Close #25445
2019-05-16 12:11:04 -07:00
Pete Bacon Darwin f4655ea98a feat(ivy): ngcc - turn on UMD processing (#25445)
PR Close #25445
2019-05-16 12:11:04 -07:00
Alan 1bd4891c9a test: fix ngcc unit tests in windows (#30297)
```
//packages/compiler-cli/ngcc/test:test
```

Partially addresses #29785

PR Close #30297
2019-05-13 11:26:56 -07:00
Pete Bacon Darwin c59717571e fix(ivy): ngcc - handle missing entry-point dependencies better (#30270)
If an entry-point has a missing dependency then all the entry-points
that would have pointed to that dependency are also removed from
the dependency graph.

Previously we were still processing the dependencies of an entry-point
even if it had already been removed from the graph because it depended
upon a missing dependency that had previously been removed due to another
entry-point depending upon it.

This caused the dependency processing to crash rather than gracefully
logging and handling the missing invalid entry-point.

Fixes #29624

PR Close #30270
2019-05-07 10:24:48 -07:00
Pete Bacon Darwin ef861958a9 refactor(ivy): ngcc - add MockFileSystem (#29643)
PR Close #29643
2019-04-29 12:37:21 -07:00
Pete Bacon Darwin 16d7dde2ad refactor(ivy): ngcc - implement abstract FileSystem (#29643)
This commit introduces a new interface, which abstracts access
to the underlying `FileSystem`. There is initially one concrete
implementation, `NodeJsFileSystem`, which is simply wrapping the
`fs` library of NodeJs.

Going forward, we can provide a `MockFileSystem` for test, which
should allow us to stop using `mock-fs` for most of the unit tests.
We could also implement a `CachedFileSystem` that may improve the
performance of ngcc.

PR Close #29643
2019-04-29 12:37:21 -07:00
Pete Bacon Darwin 1fd2cc6340 refactor(ivy): ngcc - move the dependency resolving stuff around (#29643)
For UMD/RequireJS support we will need to have multiple
`DependencyHost` implementations. This commit  prepares the
ground for that.

PR Close #29643
2019-04-29 12:37:21 -07:00