128 lines
4.4 KiB
TypeScript
Raw Normal View History

/**
* @license
* Copyright Google LLC All Rights Reserved.
*
* Use of this source code is governed by an MIT-style license that can be
* found in the LICENSE file at https://angular.io/license
*/
import * as ts from 'typescript';
import {NgCompiler, NgCompilerHost} from './core';
import {NgCompilerOptions, UnifiedModulesHost} from './core/api';
import {NodeJSFileSystem, setFileSystem} from './file_system';
import {PatchedProgramIncrementalBuildStrategy} from './incremental';
import {NOOP_PERF_RECORDER} from './perf';
perf(compiler-cli): fix regressions in incremental program reuse (#37641) Commit 4213e8d5 introduced shim reference tagging into the compiler, and changed how the `TypeCheckProgramHost` worked under the hood during the creation of a template type-checking program. This work enabled a more incremental flow for template type-checking, but unintentionally introduced several regressions in performance, caused by poor incrementality during `ts.Program` creation. 1. The `TypeCheckProgramHost` was made to rely on the `ts.CompilerHost` to retrieve instances of `ts.SourceFile`s from the original program. If the host does not return the original instance of such files, but instead creates new instances, this has two negative effects: it incurs additional parsing time, and it interferes with TypeScript's ability to reuse information about such files. 2. During the incremental creation of a `ts.Program`, TypeScript compares the `referencedFiles` of `ts.SourceFile` instances from the old program with those in the new program. If these arrays differ, TypeScript cannot fully reuse the old program. The implementation of reference tagging introduced in 4213e8d5 restores the original `referencedFiles` array after a `ts.Program` is created, which means that future incremental operations involving that program will always fail this comparison, effectively limiting the incrementality TypeScript can achieve. Problem 1 exacerbates problem 2: if a new `ts.SourceFile` is created by the host after shim generation has been disabled, it will have an untagged `referencedFiles` array even if the original file's `referencedFiles` was not restored, triggering problem 2 when creating the template type-checking program. To fix these issues, `referencedFiles` arrays are now restored on the old `ts.Program` prior to the creation of a new incremental program. This allows TypeScript to get the most out of reusing the old program's data. Additionally, the `TypeCheckProgramHost` now uses the original `ts.Program` to retrieve original instances of `ts.SourceFile`s where possible, preventing issues when a host would otherwise return fresh instances. Together, these fixes ensure that program reuse is as incremental as possible, and tests have been added to verify this for certain scenarios. An optimization was further added to prevent the creation of a type-checking `ts.Program` in the first place if no type-checking is necessary. PR Close #37641
2020-06-19 12:55:13 -07:00
import {untagAllTsFiles} from './shims';
perf(compiler-cli): split Ivy template type-checking into multiple files (#36211) As a performance optimization, this commit splits the single __ngtypecheck__.ts file which was previously added to the user's program as a container for all template type-checking code into multiple .ngtypecheck shim files, one for each original file in the user's program. In larger applications, the generation, parsing, and checking of this single type-checking file was a huge performance bottleneck, with the file often exceeding 1 MB in text content. Particularly in incremental builds, regenerating this single file for the entire application proved especially expensive. This commit introduces a new strategy for template type-checking code which makes use of a new interface, the `TypeCheckingProgramStrategy`. This interface abstracts the process of creating a new `ts.Program` to type-check a particular compilation, and allows the mechanism there to be kept separate from the more complex logic around dealing with multiple .ngtypecheck files. A new `TemplateTypeChecker` hosts that logic and interacts with the `TypeCheckingProgramStrategy` to actually generate and return diagnostics. The `TypeCheckContext` class, previously the workhorse of template type- checking, is now solely focused on collecting and generating type-checking file contents. A side effect of implementing the new `TypeCheckingProgramStrategy` in this way is that the API is designed to be suitable for use by the Angular Language Service as well. The LS also needs to type-check components, but has its own method for constructing a `ts.Program` with type-checking code. Note that this commit does not make the actual checking of templates at all _incremental_ just yet. That will happen in a future commit. PR Close #36211
2020-03-04 15:50:12 -08:00
import {ReusedProgramStrategy} from './typecheck/src/augmented_program';
// The following is needed to fix a the chicken-and-egg issue where the sync (into g3) script will
// refuse to accept this file unless the following string appears:
// import * as plugin from '@bazel/typescript/internal/tsc_wrapped/plugin_api';
/**
* A `ts.CompilerHost` which also returns a list of input files, out of which the `ts.Program`
* should be created.
*
* Currently mirrored from @bazel/typescript/internal/tsc_wrapped/plugin_api (with the naming of
* `fileNameToModuleName` corrected).
*/
interface PluginCompilerHost extends ts.CompilerHost, Partial<UnifiedModulesHost> {
readonly inputFiles: ReadonlyArray<string>;
}
/**
* Mirrors the plugin interface from tsc_wrapped which is currently under active development. To
* enable progress to be made in parallel, the upstream interface isn't implemented directly.
* Instead, `TscPlugin` here is structurally assignable to what tsc_wrapped expects.
*/
interface TscPlugin {
readonly name: string;
wrapHost(
host: ts.CompilerHost&Partial<UnifiedModulesHost>, inputFiles: ReadonlyArray<string>,
options: ts.CompilerOptions): PluginCompilerHost;
setupCompilation(program: ts.Program, oldProgram?: ts.Program): {
ignoreForDiagnostics: Set<ts.SourceFile>,
ignoreForEmit: Set<ts.SourceFile>,
};
getDiagnostics(file?: ts.SourceFile): ts.Diagnostic[];
getOptionDiagnostics(): ts.Diagnostic[];
getNextProgram(): ts.Program;
createTransformers(): ts.CustomTransformers;
}
/**
* A plugin for `tsc_wrapped` which allows Angular compilation from a plain `ts_library`.
*/
export class NgTscPlugin implements TscPlugin {
name = 'ngtsc';
private options: NgCompilerOptions|null = null;
private host: NgCompilerHost|null = null;
private _compiler: NgCompiler|null = null;
get compiler(): NgCompiler {
if (this._compiler === null) {
throw new Error('Lifecycle error: setupCompilation() must be called first.');
}
return this._compiler;
}
constructor(private ngOptions: {}) {
setFileSystem(new NodeJSFileSystem());
}
wrapHost(
host: ts.CompilerHost&Partial<UnifiedModulesHost>, inputFiles: readonly string[],
options: ts.CompilerOptions): PluginCompilerHost {
perf(compiler-cli): fix regressions in incremental program reuse (#37641) Commit 4213e8d5 introduced shim reference tagging into the compiler, and changed how the `TypeCheckProgramHost` worked under the hood during the creation of a template type-checking program. This work enabled a more incremental flow for template type-checking, but unintentionally introduced several regressions in performance, caused by poor incrementality during `ts.Program` creation. 1. The `TypeCheckProgramHost` was made to rely on the `ts.CompilerHost` to retrieve instances of `ts.SourceFile`s from the original program. If the host does not return the original instance of such files, but instead creates new instances, this has two negative effects: it incurs additional parsing time, and it interferes with TypeScript's ability to reuse information about such files. 2. During the incremental creation of a `ts.Program`, TypeScript compares the `referencedFiles` of `ts.SourceFile` instances from the old program with those in the new program. If these arrays differ, TypeScript cannot fully reuse the old program. The implementation of reference tagging introduced in 4213e8d5 restores the original `referencedFiles` array after a `ts.Program` is created, which means that future incremental operations involving that program will always fail this comparison, effectively limiting the incrementality TypeScript can achieve. Problem 1 exacerbates problem 2: if a new `ts.SourceFile` is created by the host after shim generation has been disabled, it will have an untagged `referencedFiles` array even if the original file's `referencedFiles` was not restored, triggering problem 2 when creating the template type-checking program. To fix these issues, `referencedFiles` arrays are now restored on the old `ts.Program` prior to the creation of a new incremental program. This allows TypeScript to get the most out of reusing the old program's data. Additionally, the `TypeCheckProgramHost` now uses the original `ts.Program` to retrieve original instances of `ts.SourceFile`s where possible, preventing issues when a host would otherwise return fresh instances. Together, these fixes ensure that program reuse is as incremental as possible, and tests have been added to verify this for certain scenarios. An optimization was further added to prevent the creation of a type-checking `ts.Program` in the first place if no type-checking is necessary. PR Close #37641
2020-06-19 12:55:13 -07:00
// TODO(alxhub): Eventually the `wrapHost()` API will accept the old `ts.Program` (if one is
// available). When it does, its `ts.SourceFile`s need to be re-tagged to enable proper
// incremental compilation.
this.options = {...this.ngOptions, ...options} as NgCompilerOptions;
fix(compiler): switch to 'referencedFiles' for shim generation (#36211) Shim generation was built on a lie. Shims are files added to the program which aren't original files authored by the user, but files authored effectively by the compiler. These fall into two categories: files which will be generated (like the .ngfactory shims we generate for View Engine compatibility) as well as files used internally in compilation (like the __ng_typecheck__.ts file). Previously, shim generation was driven by the `rootFiles` passed to the compiler as input. These are effectively the `files` listed in the `tsconfig.json`. Each shim generator (e.g. the `FactoryGenerator`) would examine the `rootFiles` and produce a list of shim file names which it would be responsible for generating. These names would then be added to the `rootFiles` when the program was created. The fatal flaw here is that `rootFiles` does not always account for all of the files in the program. In fact, it's quite rare that it does. Users don't typically specify every file directly in `files`. Instead, they rely on TypeScript, during program creation, starting with a few root files and transitively discovering all of the files in the program. This happens, however, during `ts.createProgram`, which is too late to add new files to the `rootFiles` list. As a result, shim generation was only including shims for files actually listed in the `tsconfig.json` file, and not for the transitive set of files in the user's program as it should. This commit completely rewrites shim generation to use a different technique for adding files to the program, inspired by View Engine's shim generator. In this new technique, as the program is being created and `ts.SourceFile`s are being requested from the `NgCompilerHost`, shims for those files are generated and a reference to them is patched onto the original file's `ts.SourceFile.referencedFiles`. This causes TS to think that the original file references the shim, and causes the shim to be included in the program. The original `referencedFiles` array is saved and restored after program creation, hiding this little hack from the rest of the system. The new shim generation engine differentiates between two kinds of shims: top-level shims (such as the flat module entrypoint file and __ng_typecheck__.ts) and per-file shims such as ngfactory or ngsummary files. The former are included via `rootFiles` as before, the latter are included via the `referencedFiles` of their corresponding original files. As a result of this change, shims are now correctly generated for all files in the program, not just the ones named in `tsconfig.json`. A few mitigating factors prevented this bug from being realized until now: * in g3, `files` does include the transitive closure of files in the program * in CLI apps, shims are not really used This change also makes use of a novel technique for associating information with source files: the use of an `NgExtension` `Symbol` to patch the information directly onto the AST object. This is used in several circumstances: * For shims, metadata about a `ts.SourceFile`'s status as a shim and its origins are held in the extension data. * For original files, the original `referencedFiles` are stashed in the extension data for later restoration. The main benefit of this technique is a lot less bookkeeping around `Map`s of `ts.SourceFile`s to various kinds of data, which need to be tracked/ invalidated as part of incremental builds. This technique is based on designs used internally in the TypeScript compiler and is serving as a prototype of this design in ngtsc. If it works well, it could have benefits across the rest of the compiler. PR Close #36211
2020-02-26 16:12:39 -08:00
this.host = NgCompilerHost.wrap(host, inputFiles, this.options, /* oldProgram */ null);
return this.host;
}
setupCompilation(program: ts.Program, oldProgram?: ts.Program): {
ignoreForDiagnostics: Set<ts.SourceFile>,
ignoreForEmit: Set<ts.SourceFile>,
} {
if (this.host === null || this.options === null) {
throw new Error('Lifecycle error: setupCompilation() before wrapHost().');
}
perf(compiler-cli): fix regressions in incremental program reuse (#37641) Commit 4213e8d5 introduced shim reference tagging into the compiler, and changed how the `TypeCheckProgramHost` worked under the hood during the creation of a template type-checking program. This work enabled a more incremental flow for template type-checking, but unintentionally introduced several regressions in performance, caused by poor incrementality during `ts.Program` creation. 1. The `TypeCheckProgramHost` was made to rely on the `ts.CompilerHost` to retrieve instances of `ts.SourceFile`s from the original program. If the host does not return the original instance of such files, but instead creates new instances, this has two negative effects: it incurs additional parsing time, and it interferes with TypeScript's ability to reuse information about such files. 2. During the incremental creation of a `ts.Program`, TypeScript compares the `referencedFiles` of `ts.SourceFile` instances from the old program with those in the new program. If these arrays differ, TypeScript cannot fully reuse the old program. The implementation of reference tagging introduced in 4213e8d5 restores the original `referencedFiles` array after a `ts.Program` is created, which means that future incremental operations involving that program will always fail this comparison, effectively limiting the incrementality TypeScript can achieve. Problem 1 exacerbates problem 2: if a new `ts.SourceFile` is created by the host after shim generation has been disabled, it will have an untagged `referencedFiles` array even if the original file's `referencedFiles` was not restored, triggering problem 2 when creating the template type-checking program. To fix these issues, `referencedFiles` arrays are now restored on the old `ts.Program` prior to the creation of a new incremental program. This allows TypeScript to get the most out of reusing the old program's data. Additionally, the `TypeCheckProgramHost` now uses the original `ts.Program` to retrieve original instances of `ts.SourceFile`s where possible, preventing issues when a host would otherwise return fresh instances. Together, these fixes ensure that program reuse is as incremental as possible, and tests have been added to verify this for certain scenarios. An optimization was further added to prevent the creation of a type-checking `ts.Program` in the first place if no type-checking is necessary. PR Close #37641
2020-06-19 12:55:13 -07:00
this.host.postProgramCreationCleanup();
untagAllTsFiles(program);
perf(compiler-cli): split Ivy template type-checking into multiple files (#36211) As a performance optimization, this commit splits the single __ngtypecheck__.ts file which was previously added to the user's program as a container for all template type-checking code into multiple .ngtypecheck shim files, one for each original file in the user's program. In larger applications, the generation, parsing, and checking of this single type-checking file was a huge performance bottleneck, with the file often exceeding 1 MB in text content. Particularly in incremental builds, regenerating this single file for the entire application proved especially expensive. This commit introduces a new strategy for template type-checking code which makes use of a new interface, the `TypeCheckingProgramStrategy`. This interface abstracts the process of creating a new `ts.Program` to type-check a particular compilation, and allows the mechanism there to be kept separate from the more complex logic around dealing with multiple .ngtypecheck files. A new `TemplateTypeChecker` hosts that logic and interacts with the `TypeCheckingProgramStrategy` to actually generate and return diagnostics. The `TypeCheckContext` class, previously the workhorse of template type- checking, is now solely focused on collecting and generating type-checking file contents. A side effect of implementing the new `TypeCheckingProgramStrategy` in this way is that the API is designed to be suitable for use by the Angular Language Service as well. The LS also needs to type-check components, but has its own method for constructing a `ts.Program` with type-checking code. Note that this commit does not make the actual checking of templates at all _incremental_ just yet. That will happen in a future commit. PR Close #36211
2020-03-04 15:50:12 -08:00
const typeCheckStrategy = new ReusedProgramStrategy(
program, this.host, this.options, this.host.shimExtensionPrefixes);
this._compiler = new NgCompiler(
this.host, this.options, program, typeCheckStrategy,
new PatchedProgramIncrementalBuildStrategy(), oldProgram, NOOP_PERF_RECORDER);
return {
ignoreForDiagnostics: this._compiler.ignoreForDiagnostics,
ignoreForEmit: this._compiler.ignoreForEmit,
};
}
getDiagnostics(file?: ts.SourceFile): ts.Diagnostic[] {
return this.compiler.getDiagnostics(file);
}
getOptionDiagnostics(): ts.Diagnostic[] {
return this.compiler.getOptionDiagnostics();
}
getNextProgram(): ts.Program {
return this.compiler.getNextProgram();
}
createTransformers(): ts.CustomTransformers {
return this.compiler.prepareEmit().transformers;
}
}