Commit Graph

45 Commits

Author SHA1 Message Date
Alex Rickabaugh 3613e7c4e5 test(compiler-cli): move testing utils to separate package (#39594)
ngtsc has a robust suite of testing utilities, designed for in-memory
testing of a TypeScript compiler. Previously these utilities lived in the
`test` directory for the compiler-cli package.

This commit moves those utilities to an `ngtsc/testing` package, enabling
them to be depended on separately and opening the door for using them from
the upcoming language server testing infrastructure.

As part of this refactoring, the `fake_core` package (a lightweight API
replacement for @angular/core) is expanded to include functionality needed
for Language Service test use cases.

PR Close #39594
2020-11-17 11:59:56 -08:00
Pete Bacon Darwin 6de5a12a9d refactor(ngcc): move `logging` code into `ngtsc` (#37114)
The `Logger` interface and its related classes are general purpose
and could be used by other tooling. Moving it into ngtsc is a more
suitable place from which to share it - similar to the FileSystem stuff.

PR Close #37114
2020-06-22 13:38:47 -07:00
George Kalpakas bf682d73d4 fix(ngcc): correctly get config for packages in nested `node_modules/` (#37040)
Previously, ngcc would only be able to match an ngcc configuration to
packages that were located inside the project's top-level
`node_modules/`. However, if there are multiple versions of a package in
a project (e.g. as a transitive dependency of other packages), multiple
copies of a package (at different versions) may exist in nested
`node_modules/` directories. For example, one at
`<project-root>/node_modules/some-package/` and one at
`<project-root>/node_modules/other-package/node_modules/some-package/`.
In such cases, ngcc was only able to detect the config for the first
copy but not for the second.

This commit fixes this by returning a new instance of
`ProcessedNgccPackageConfig` for each different package path (even if
they refer to the same package name). In these
`ProcessedNgccPackageConfig`, the `entryPoints` paths have been
processed to take the package path into account.

PR Close #37040
2020-06-11 18:58:36 -07:00
George Kalpakas 8f3695e20e refactor(ngcc): add `packageName` property to `EntryPoint` interface (#37040)
This commit adds a `packageName` property to the `EntryPoint` interface.
In a subsequent commit this will be used to retrieve the correct ngcc
configuration for each package, regardless of its path.

PR Close #37040
2020-06-11 18:58:36 -07:00
George Kalpakas 8197557fcf refactor(ngcc): rename `EntryPoint#package` to `EntryPoint#packagePath` (#37040)
Rename the `package` property to `packagePath` on the `EntryPoint`
interface. This makes it more clear that the `packagePath` property
holds the absolute path to the containing package (similar to how `path`
holds the path to the entry-point). This will also align with the
`packageName` property that will be added in a subsequent commit.

This commit also re-orders the `EntryPoint` properties to group related
properties together and to match the order of properties on instances
with that on the interface.

PR Close #37040
2020-06-11 18:58:36 -07:00
Pete Bacon Darwin ae364864f6 fix(ngcc): do not scan import expressions in d.ts files (#37503)
It is quite common for the TS compiler to have to add synthetic
types to function signatures, where the developer has not
explicitly provided them.  This results in `import(...)` expressions
appearing in typings files.  For example in `@ngrx/data` there is a
class with a getter that has an implicit type:

```ts
export declare class EntityCollectionServiceBase<...> {
  ...
  get store() {
    return this.dispatcher.store;
  }
  ...
}
```

In the d.ts file for this we get:

```ts
get store(): Store<import("@ngrx/data").EntityCache>;
```

Given that this file is within the `@ngrx/data` package already,
this caused ngcc to believe that there was a circular dependency,
causing it to fail to process the package - and in fact crash!

This commit resolves this problem by ignoring `import()` expressions
when scanning typings programs for dependencies. This ability was
only introduced very recently in a 10.0.0 RC release, and so it has
limited benefit given that up till now ngcc has been able to process
libraries effectively without it. Moreover, in the rare case that a
package does have such a dependency, it should get picked up
by the sync ngcc+CLI integration point.

PR Close #37503
2020-06-10 11:51:18 -07:00
Pete Bacon Darwin 57411c85b9 feat(ngcc): implement a program-based entry-point finder (#37075)
This finder is designed to only process entry-points that are reachable
by the program defined by a tsconfig.json file.

It is triggered by calling `mainNgcc()` with the `findEntryPointsFromTsConfigProgram`
option set to true. It is ignored if a `targetEntryPointPath` has been
provided as well.

It is triggered from the command line by adding the `--use-program-dependencies`
option, which is also ignored if the `--target` option has been provided.

Using this option can speed up processing in cases where there is a large
number of dependencies installed but only a small proportion of the
entry-points are actually imported into the application.

PR Close #37075
2020-06-04 09:22:39 -07:00
Pete Bacon Darwin 07a8016118 fix(ngcc): capture dynamic import expressions as well as declarations (#37075)
Previously we only checked for static import declaration statements.
This commit also finds import paths from dynamic import expressions.

Also this commit should speed up processing: Previously we were parsing
the source code contents into a `ts.SourceFile` and then walking the parsed
AST to find import paths.
Generating an AST is unnecessary work and it is faster and creates less
memory pressure to just scan the source code contents with the TypeScript
scanner, identifying import paths from the tokens.

PR Close #37075
2020-06-04 09:22:39 -07:00
Pete Bacon Darwin 4d69da57ca refactor(ngcc): move shared code into `DependencyHostBase` (#37075)
The various dependency hosts had a lot of duplicated code.
This commit refactors them to move this into the base class.

PR Close #37075
2020-06-04 09:22:39 -07:00
Pete Bacon Darwin 7f98b87ca0 fix(ngcc): ensure that more dependencies are found by `EsmDependencyHost` (#37075)
Previously this host was skipping files if they had imports that spanned
multiple lines, or if the import was a dynamic import expression.

PR Close #37075
2020-06-04 09:22:39 -07:00
Joey Perrott d1ea1f4c7f build: update license headers to reference Google LLC (#37205)
Update the license headers throughout the repository to reference Google LLC
rather than Google Inc, for the required license headers.

PR Close #37205
2020-05-26 14:26:58 -04:00
Pete Bacon Darwin a185efbd60 perf(ngcc): read dependencies from entry-point manifest (#36486)
Previously, even if an entry-point did not need to be processed,
ngcc would always parse the files of the entry-point to compute
its dependencies. This can take a lot of time for large node_modules.

Now these dependencies are cached in the entry-point manifest,
and read from there rather than computing them every time.

See https://github.com/angular/angular/issues/36414\#issuecomment-608401834
FW-2047

PR Close #36486
2020-04-09 11:33:28 -07:00
George Kalpakas aecf9de738 fix(ngcc): correctly identify relative Windows-style import paths (#36372)
Previously, `isRelativePath()` assumed paths are *nix-style. This caused
Windows-style paths (such as `C:\foo\some-package\some-file.js`) to not
be recognized as "relative" imports.

This commit fixes this by using the OS-agnostic `isRooted()` helper and
also accounting for both styles of path delimiters: `/` and `\`

PR Close #36372
2020-04-07 15:21:27 -07:00
Pete Bacon Darwin 8be8466a00 style(ngcc): reformat of ngcc after clang update (#36447)
PR Close #36447
2020-04-06 09:26:57 -07:00
Pete Bacon Darwin 20b0c80b0b fix(ngcc): allow deep-import warnings to be ignored (#35683)
This commit adds a new ngcc configuration, `ignorableDeepImportMatchers`
for packages. This is a list of regular expressions matching deep imports
that can be safely ignored from that package. Deep imports that are not
ignored cause a warning to be logged.

// FW-1892

Fixes #35615

PR Close #35683
2020-02-27 10:48:48 -08:00
Pete Bacon Darwin 3d5bcd5883 test(ngcc): update dependency host test description (#34695)
The `describe` description did not match the name of the
method.

PR Close #34695
2020-01-15 10:24:50 -08:00
Pete Bacon Darwin 85b5c365fc fix(ngcc): do not add DTS deep imports to missing packages list (#34695)
When searching the typings program for a package for imports a
distinction is drawn between missing entry-points and deep imports.

Previously in the `DtsDependencyHost` these deep imports may be
marked as missing if there was no typings file at the deep import path.
Instead there may be a javascript file instead. In practice this means
the import is "deep" and not "missing".

Now the `DtsDependencyHost` will also consider `.js` files when checking
for deep-imports, and it will also look inside `@types/...` for a suitable
deep-imported typings file.

Fixes #34720

PR Close #34695
2020-01-15 10:24:50 -08:00
Pete Bacon Darwin da51d884a1 test(ngcc): remove `declare` from JS classes (#34695)
PR Close #34695
2020-01-15 10:24:49 -08:00
George Kalpakas cfbb1a1e77 fix(ngcc): correctly detect dependencies in CommonJS (#34528)
Previously, `CommonJsDependencyHost.collectDependencies()` would only
find dependencies via imports of the form `var foo = require('...');` or
`var foo = require('...'), bar = require('...');` However, CommonJS
files can have imports in many different forms. By failing to recognize
other forms of imports, the associated dependencies were missed, which
in turn resulted in entry-points being compiled out-of-order and failing
due to that.

While we cannot easily capture all different types of imports, this
commit enhances `CommonJsDependencyHost` to recognize the following
common forms of imports:

- Imports in property assignments. E.g.:
  `exports.foo = require('...');` or
  `module.exports = {foo: require('...')};`

- Imports for side-effects only. E.g.:
  `require('...');`

- Star re-exports (with both emitted and imported heleprs). E.g.:
  `__export(require('...'));` or
  `tslib_1.__exportStar(require('...'), exports);`

PR Close #34528
2020-01-13 09:48:20 -08:00
George Kalpakas eb6e1af46d test(ngcc): fix typos in `CommonJsDependencyHost` tests (to avoid confusion) (#34528)
PR Close #34528
2020-01-13 09:48:20 -08:00
George Kalpakas 07ea6cf582 fix(ngcc): avoid error due to circular dependency in `EsmDependencyHost` (#34512)
Previously, there was circular dependency between `ngcc/src/utils.ts`,
`ngcc/src/dependencies/dependency_host.ts` and
`ngcc/src/dependencies/esm_dependency_host.ts`. More specifically,
`utils.ts` would [import from `esm_dependency_host.ts`][1], which would
[import from `dependency_host.ts`][2], which would in turn
[import from `utils.ts`][3].

This might be fine in some environments/module formats, but it can cause
unclear errors in the transpiled CommonJS/UMD format (given how Node.js
handles [cycles in module resolution][4]).
(An example error can be found [here][5].)

This commit fixes the problem by moving the code that depends on
`EsmDependencyHost` out of `utils.ts` and into a dedicated file under
`dependencies/`. It also converts the `createDtsDependencyHost()`
function to a class for consistency with the rest of the
`DependencyHost`s.

[1]: https://github.com/angular/angular/blob/18d89c9c8/packages/compiler-cli/ngcc/src/utils.ts#L10
[2]: https://github.com/angular/angular/blob/18d89c9c8/packages/compiler-cli/ngcc/src/dependencies/esm_dependency_host.ts#L10
[3]: https://github.com/angular/angular/blob/18d89c9c8/packages/compiler-cli/ngcc/src/dependencies/dependency_host.ts#L9
[4]: https://nodejs.org/api/modules.html#modules_cycles
[5]: https://circleci.com/gh/angular/angular/577581

PR Close #34512
2020-01-08 15:00:50 -08:00
Pete Bacon Darwin 4f42de9704 fix(ngcc): capture entry-point dependencies from typings as well as source (#34494)
ngcc computes a dependency graph of entry-points to ensure that
entry-points are processed in the correct order. Previously only the imports
in source files were analysed to determine the dependencies for each
entry-point.

This is not sufficient when an entry-point has a "type-only" dependency
 - for example only importing an interface from another entry-point.
In this case the "type-only" import does not appear in the
source code. It only appears in the typings files. This can cause a
dependency to be missed on the entry-point.

This commit fixes this by additionally processing the imports in the
typings program, as well as the source program.

Note that these missing dependencies could cause unexpected flakes when
running ngcc in async mode on multiple processes due to the way that
ngcc caches files when they are first read from disk.

Fixes #34411

// FW-1781

PR Close #34494
2020-01-07 10:35:03 -08:00
Pete Bacon Darwin 69950e3888 refactor(ngcc): resolve modules based on the provided `moduleResolver` (#34494)
The `DependencyHost` implementations were duplicating the "postfix" strings
which are used to find matching paths when resolving module specifiers.
Now the hosts reuse the postfixes given to the `ModuleResolver` that is
passed to the host.

PR Close #34494
2020-01-07 10:35:03 -08:00
Pete Bacon Darwin e2b184515b refactor(ngcc): pass dependency info to `collectDependencies()` (#34494)
Rather than return a new object of dependency info from calls to
`collectDependencies()` we now pass in an object that will be updated
with the dependency info. This is in preparation of a change where
we will collect dependency information from more than one
`DependencyHost`.

Also to better fit with this approach the name is changed from
`findDependencies()` to `collectDependencies()`.

PR Close #34494
2020-01-07 10:35:03 -08:00
Alex Rickabaugh 4da2dda647 feat(ngcc): support ignoreMissingDependencies in ngcc config (#33192)
Normally, when ngcc encounters a package with missing dependencies while
attempting to determine a compilation ordering, it will ignore that package.
This commit adds a configuration for a flag to tell ngcc to compile the
package anyway, regardless of any missing dependencies.

FW-1931 #resolve

PR Close #33192
2019-10-17 19:43:39 -04:00
George Kalpakas e36e6c85ef perf(ngcc): process tasks in parallel in async mode (#32427)
`ngcc` supports both synchronous and asynchronous execution. The default
mode when using `ngcc` programmatically (which is how `@angular/cli` is
using it) is synchronous. When running `ngcc` from the command line
(i.e. via the `ivy-ngcc` script), it runs in async mode.

Previously, the work would be executed in the same way in both modes.

This commit improves the performance of `ngcc` in async mode by
processing tasks in parallel on multiple processes. It uses the Node.js
built-in [`cluster` module](https://nodejs.org/api/cluster.html) to
launch a cluster of Node.js processes and take advantage of multi-core
systems.

Preliminary comparisons indicate a 1.8x to 2.6x speed improvement when
processing the angular.io app (apparently depending on the OS, number of
available cores, system load, etc.). Further investigation is needed to
better understand these numbers and identify potential areas of
improvement.

Inspired by/Based on @alxhub's prototype: alxhub/angular@cb631bdb1
Original design doc: https://hackmd.io/uYG9CJrFQZ-6FtKqpnYJAA?view

Jira issue: [FW-1460](https://angular-team.atlassian.net/browse/FW-1460)

PR Close #32427
2019-09-09 15:55:13 -04:00
JoostK 4bbf16e654 fix(ngcc): handle deep imports that already have an extension (#32181)
During the dependency analysis phase of ngcc, imports are resolved to
files on disk according to certain module resolution rules. Since module
specifiers are typically missing extensions, or can refer to index.js
barrel files within a directory, the module resolver attempts several
postfixes when searching for a module import on disk. Module  specifiers
that already include an extension, however, would fail to be resolved as
ngcc's module resolver failed to check the location on disk without
adding any postfixes.

Closes #32097

PR Close #32181
2019-08-19 10:12:03 -07:00
JoostK 57e15fc08b fix(ivy): ngcc - do not consider builtin NodeJS modules as missing (#31872)
ngcc analyzes the dependency structure of the entrypoints it needs to
process, as the compilation of entrypoints is ordering sensitive: any
dependent upon entrypoint must be compiled before its dependees. As part
of the analysis of the dependency graph, it is detected when a
dependency of entrypoint is not installed, in which case that entrypoint
will be marked as ignored.

For libraries that work with Angular Universal to run in NodeJS, imports
into builtin NodeJS modules can be present. ngcc's dependency analyzer
can only resolve imports within the TypeScript compilation, which
builtin modules are not part of. Therefore, such imports would
erroneously cause the entrypoint to become ignored.

This commit fixes the problem by taking the NodeJS builtins into account
when dealing with missing imports.

Fixes #31522

PR Close #31872
2019-08-05 13:06:49 -07:00
JoostK b70746a113 fix(ivy): ngcc - prevent crash when analyzed target is ignored (#31872)
ngcc analyzes the dependency structure of the entrypoints it needs to
process, as the compilation of entrypoints is ordering sensitive: any
dependent upon entrypoint must be compiled before its dependees. As part
of the analysis of the dependency graph, it is detected when a
dependency of entrypoint is not installed, in which case that entrypoint
will be marked as ignored.

When a target entrypoint to compile is provided, it could occur that
given target is considered ignored because one of its dependencies might
be missing. This situation was not dealt with currently, instead
resulting in a crash of ngcc.

This commit prevents the crash by taking the above scenario into account.

PR Close #31872
2019-08-05 13:06:49 -07:00
Pete Bacon Darwin fac20bd8d1 fix(ivy): ngcc - resolve `main` property paths correctly (#31509)
There are two places in the ngcc processing where it needs to load the
content of a file given by a general path:

* when determining the format of an entry-point.
 To do this ngcc uses the value of the relevant property in package.json.
 But in the case of `main` it must parse the contents of the entry-point
 file to decide whether the format is UMD or CommonJS.

* when parsing the source files for dependencies to determine the order in
which compilation must occur. The relative imports in each file are parsed
and followed recursively, looking for external imports.

Previously, we naively assumed that the path would match the file name exactly.
But actually we must consider the standard module resolution conventions.
E.g. the extension (.js) may be missing, or the path may refer to a directory
containing an index.js file.

This commit fixes both places.

This commit now requires the `DependencyHost` instances to check
the existence of more files than before (at worst all the different possible
post-fixes). This should not create a significant performance reduction for
ngcc. Since the results of the checks will be cached, and similar work is
done inside the TS compiler, so what we lose in doing it here, is saved later
in the processing. The main performance loss would be where there are lots
of files that need to be parsed for dependencies that do not end up being
processed by TS. But compared to the main ngcc processing this dependency
parsing is a small proportion of the work done and so should not impact
much on the overall performance of ngcc.

// FW-1444

PR Close #31509
2019-07-12 11:37:35 -04:00
Matias Niemelä db557221bc revert: fix(ivy): ngcc - resolve `main` property paths correctly (#31509)
This reverts commit 103a5b42ec.
2019-07-11 11:51:13 -04:00
Pete Bacon Darwin 103a5b42ec fix(ivy): ngcc - resolve `main` property paths correctly (#31509)
When determining if a `main` path points to a UMD or CommonJS
format, the contents of the file need to be loaded and parsed.

Previously, it was assumed that the path referred to the exact filename,
but did not account for normal module resolution semantics, where the
path may be missing an extension or refer to a directory containing an
`index.js` file.

// FW-1444

PR Close #31509
2019-07-11 11:41:11 -04:00
Pete Bacon Darwin 50c4ec6687 fix(ivy): ngcc - resolve path-mapped modules correctly (#31450)
Non-wild-card path-mappings were not being matched correctly.

Further path-mapped secondary entry-points that
were imported from the associated primary entry-point were not
being martched correctly.

Fixes #31274

PR Close #31450
2019-07-08 10:28:13 -07:00
Pete Bacon Darwin 3788ebb714 fix(ivy): ngcc - don't crash if entry-points have multiple invalid dependencies (#31276)
If an entry-point has missing dependencies then it cannot be
processed and is marked as invalid. Similarly, if an entry-point
has dependencies that have been marked as invalid then that
entry-point too is invalid. In all these cases, ngcc should quietly
ignore these entry-points and continue processing what it can.

Previously, if an entry-point had more than one entry-point that
was transitively invalid then ngcc was crashing rather than
ignoring the entry-point.

PR Close #31276
2019-06-26 08:01:43 -07:00
Pete Bacon Darwin 7186f9c016 refactor(ivy): implement a virtual file-system layer in ngtsc + ngcc (#30921)
To improve cross platform support, all file access (and path manipulation)
is now done through a well known interface (`FileSystem`).

For testing a number of `MockFileSystem` implementations are provided.
These provide an in-memory file-system which emulates operating systems
like OS/X, Unix and Windows.

The current file system is always available via the static method,
`FileSystem.getFileSystem()`. This is also used by a number of static
methods on `AbsoluteFsPath` and `PathSegment`, to avoid having to pass
`FileSystem` objects around all the time. The result of this is that one
must be careful to ensure that the file-system has been initialized before
using any of these static methods. To prevent this happening accidentally
the current file system always starts out as an instance of `InvalidFileSystem`,
which will throw an error if any of its methods are called.

You can set the current file-system by calling `FileSystem.setFileSystem()`.
During testing you can call the helper function `initMockFileSystem(os)`
which takes a string name of the OS to emulate, and will also monkey-patch
aspects of the TypeScript library to ensure that TS is also using the
current file-system.

Finally there is the `NgtscCompilerHost` to be used for any TypeScript
compilation, which uses a given file-system.

All tests that interact with the file-system should be tested against each
of the mock file-systems. A series of helpers have been provided to support
such tests:

* `runInEachFileSystem()` - wrap your tests in this helper to run all the
wrapped tests in each of the mock file-systems.
* `addTestFilesToFileSystem()` - use this to add files and their contents
to the mock file system for testing.
* `loadTestFilesFromDisk()` - use this to load a mirror image of files on
disk into the in-memory mock file-system.
* `loadFakeCore()` - use this to load a fake version of `@angular/core`
into the mock file-system.

All ngcc and ngtsc source and tests now use this virtual file-system setup.

PR Close #30921
2019-06-25 16:25:24 -07:00
Pete Bacon Darwin e20b92ba37 feat(ivy): ngcc - turn on CommonJS support (#30200)
PR Close #30200
2019-05-22 16:24:15 -07:00
Pete Bacon Darwin 1bdec3ed6a feat(ivy): ngcc - implement CommonJsDependencyHost (#30200)
PR Close #30200
2019-05-22 16:24:15 -07:00
Pete Bacon Darwin 73e3f565e0 test(ivy): ngcc - fix tests to work on Windows (#30520)
PR Close #30520
2019-05-16 13:32:02 -07:00
Pete Bacon Darwin edd775eabc feat(ivy): ngcc - implement UmdDependencyHost (#25445)
The dependency resolution that works out the order in which
to process entry-points must also understand UMD formats.

PR Close #25445
2019-05-16 12:11:04 -07:00
Pete Bacon Darwin f4655ea98a feat(ivy): ngcc - turn on UMD processing (#25445)
PR Close #25445
2019-05-16 12:11:04 -07:00
Alan 1bd4891c9a test: fix ngcc unit tests in windows (#30297)
```
//packages/compiler-cli/ngcc/test:test
```

Partially addresses #29785

PR Close #30297
2019-05-13 11:26:56 -07:00
Pete Bacon Darwin c59717571e fix(ivy): ngcc - handle missing entry-point dependencies better (#30270)
If an entry-point has a missing dependency then all the entry-points
that would have pointed to that dependency are also removed from
the dependency graph.

Previously we were still processing the dependencies of an entry-point
even if it had already been removed from the graph because it depended
upon a missing dependency that had previously been removed due to another
entry-point depending upon it.

This caused the dependency processing to crash rather than gracefully
logging and handling the missing invalid entry-point.

Fixes #29624

PR Close #30270
2019-05-07 10:24:48 -07:00
Pete Bacon Darwin ef861958a9 refactor(ivy): ngcc - add MockFileSystem (#29643)
PR Close #29643
2019-04-29 12:37:21 -07:00
Pete Bacon Darwin 16d7dde2ad refactor(ivy): ngcc - implement abstract FileSystem (#29643)
This commit introduces a new interface, which abstracts access
to the underlying `FileSystem`. There is initially one concrete
implementation, `NodeJsFileSystem`, which is simply wrapping the
`fs` library of NodeJs.

Going forward, we can provide a `MockFileSystem` for test, which
should allow us to stop using `mock-fs` for most of the unit tests.
We could also implement a `CachedFileSystem` that may improve the
performance of ngcc.

PR Close #29643
2019-04-29 12:37:21 -07:00
Pete Bacon Darwin 1fd2cc6340 refactor(ivy): ngcc - move the dependency resolving stuff around (#29643)
For UMD/RequireJS support we will need to have multiple
`DependencyHost` implementations. This commit  prepares the
ground for that.

PR Close #29643
2019-04-29 12:37:21 -07:00