When running in parallel mode, worker processes forward errors thrown
during task processing to the master process, which in turn exits with
an error.
However, there are cases where the error is not directly related to
processing the entry-point. One such case is when there is not enough
memory (for example, due to all the other tasks being processed
simultaneously).
Previously, an `ENOMEM` error thrown on a worker process would propagate
to the master process, eventually causing ngcc to exit with an error.
Example failure: https://circleci.com/gh/angular/angular/682198
This commit improves handling of these low-memory situations by
detecting `ENOMEM` errors and killing the worker process, thus allowing
the master process to decide how to handle that. The master process will
put the task back into the tasks queue and continue processing tasks
with the rest of the worker processes (and thus with lower memory
pressure).
PR Close#36626
With this commit, worker processes will notify the master process about
the transformed files they are about to write to disk before starting
writing them.
In a subsequent commit, this will be used to allow ngcc to recover when
a worker process crashes in the middle of processing a task.
PR Close#36626
This commit enhances the `CompileFn`, which is used to process each
entry-point, to support running a passed-in callback (and wait for it to
complete) before proceeding with writing the transformed files to disk.
This functionality is currently not used. In a subsequent commit, it
will be used for passing info from worker processes to the master
process that will allow ngcc to recover when a worker process crashes in
the middle of processing a task.
PR Close#36626
Moving the definition of the `onTaskCompleted` callback into `mainNgcc()`
allows it to be configured based on options passed in there more easily.
This will be the case when we want to configure whether to log or throw
an error for tasks that failed to be processed successfully.
This commit also creates two new folders and moves the code around a bit
to make it easier to navigate the code§:
* `execution/tasks`: specific helpers such as task completion handlers
* `execution/tasks/queues`: the `TaskQueue` implementations and helpers
PR Close#36083
`ngcc` supports both synchronous and asynchronous execution. The default
mode when using `ngcc` programmatically (which is how `@angular/cli` is
using it) is synchronous. When running `ngcc` from the command line
(i.e. via the `ivy-ngcc` script), it runs in async mode.
Previously, the work would be executed in the same way in both modes.
This commit improves the performance of `ngcc` in async mode by
processing tasks in parallel on multiple processes. It uses the Node.js
built-in [`cluster` module](https://nodejs.org/api/cluster.html) to
launch a cluster of Node.js processes and take advantage of multi-core
systems.
Preliminary comparisons indicate a 1.8x to 2.6x speed improvement when
processing the angular.io app (apparently depending on the OS, number of
available cores, system load, etc.). Further investigation is needed to
better understand these numbers and identify potential areas of
improvement.
Inspired by/Based on @alxhub's prototype: alxhub/angular@cb631bdb1
Original design doc: https://hackmd.io/uYG9CJrFQZ-6FtKqpnYJAA?view
Jira issue: [FW-1460](https://angular-team.atlassian.net/browse/FW-1460)
PR Close#32427
George Kalpakas
Pete Bacon Darwin
Alan Agius