With the refactoring from a Gulp task to a Bazel too, we tried switching
away from the hard-coded list of locales and aliases for the Closure
Locale file generation. After multiple attempts of landing this, it
turned out that Closure Compiler/Closure Library relies on locale
identifiers CLDR does not capture within it's `availableLocales.json`
or `aliases.json` data.
Closure Library does not use any unknown locale identifiers here. The
locale identifiers can be resolved within CLDR using the bundle lookup
algorithm that is specified as part of CLDR; instead the problem is that
the locale identifiers do not follow any reasonable pattern and
therefore it's extremely difficult to generate them automatically (it's
almost like we'd need to build up _all_ possible combinations). Instead
of doing that, we just use the hard-coded locales and aliases from the
old Closure Locale generation script.
PR Close#42230
Within Google, closure compiler is used for dealing with translations.
We generate a closure-compatible locale file that allows for
registration within Angular, so that Closure i18n works well together
with Angular applications. Closure compiler does not limit its
locales to BCP47-canonical locale identifiers. This commit updates
the generation logic so that we also support deprecated (but aliased)
locale identifiers, or other aliases which are likely used within
Closure. We use CLDR's alias supplemental data for this. It instructs
us to alias `iw` to `he` for example. `iw` is still supported in Closure.
Note that we do not manually extract all locales supported in Closure;
instead we only support the CLDR canonical locales (as done before) +
common aliases that CLDR provides data for. We are not aware of other
locale aliases within Closure that wouldn't be part of the CLDR aliases.
If there would be, then Angular/Closure would fail accordingly.
PR Close#42230
In the past, the closure file has been generated so that all individual
locale files were imported individually. This resulted in a huge
slow-down in g3 due to the large amount of imports.
With 90bd984ff7 this changed so that we
inline the locale data for the g3 closure locale file. Also the file
only contained data for locales being supported by Closure. For this a
list of locales has been extracted from Closure Compiler, as well as a
list of locale aliases.
This logic is prone to CLDR version updates, and also broke as part of
the Gulp -> Bazel migration where this logic has been slightly modified
but caused issues in G3. e.g. a locale `zh-Hant` was requested in g3,
but the locale data had the name of the alias locale that provided the
data at index zero (which represents the locale name). Note that the
locale names at index zero always could differentiate from the requested
`goog.LOCALE` due to the aliasing logic. This just didn't come up before.
We simplify this logic by generating a `goog.LOCALE` case for all
locales CLDR provides data for. We don't need to bother about aliasing
because with the refactorings to the CLDR generation tool, all locales
are built (which also captures the aliases), and we can generate the locale
file on the fly (which has not been done before).
PR Close#42230
The CLDR extraction tool has been reworked to run as part of Bazel.
This adds a initial readme explaining what the tool generates. It's
far from a detailed description but it can serve as foundation for more
detailed explanations.
PR Close#42230
Introduces a few Starlark macros for running the new Bazel
CLDR generation tool. Wires up the new tool so that locales
are generated properly. Also updates the existing
`closure-locale` file to match the new output generated by the Bazel tool.
This commit also re-adds a few locale files that aren't
generated by CLDR 37, but have been accidentally left in
the repository as the Gulp script never removed old locales
from previous CLDR versions. This problem is solved with the
Bazel generation of locale files, but for now we re-add these
old CLDR 33 locale files to not break developers relying on these
(even though the locale data indicies are incorrect; but there might
be users accessing the data directly)
PR Close#42230
Converts the CLDR locale extraction script to a Bazel tool.
This allows us to generate locale files within Bazel, so that
locales don't need to live as sources within the repo. Also
it allows us to get rid of the legacy Gulp tooling.
The migration of the Gulp script to a Bazel tool involved the
following things:
1. Basic conversion of the `extract.js` script to TypeScript.
This mostly was about adding explicit types. e.g. adding `locale:
string` or `localeData: CldrStatic`.
2. Split-up into separate files. Instead of keeping the large
`extract.js` file, the tool has been split into separate files.
The logic remains the same, just that code is more readable and
maintainable.
3. Introduction of a new `index.ts` file that is the entry-point
for the Bazel tool. Previously the Gulp tool just generated
all locale files, the default locale and base currency files
at once. The new entry-point accepts a mode to be passed as
first process argument. based on that argument, either locales
are generated into a specified directory, or the default locale,
base currencies or closure file is generated.
This allows us to generate files with a Bazel genrule where
we simply run the tool and specify the outputs. Note: It's
necessary to have multiple modes because files live in separate
locations. e.g. the default locale in `@angular/core`, but the
rest in `@angular/common`.
4. Removal of the `cldr-data-downloader` and custom CLDR resolution
logic. Within Bazel we cannot run a downloader using network.
We switch this to something more Bazel idiomatic with better
caching. For this a new repository rule is introduced that
downloads the CLDR JSON repository and extracts it. Within
that rule we determine the supported locales so that they
can be used to pre-declare outputs (for the locales) within
Bazel analysis phase. This allows us to add the generated locale
files to a `ts_library` (which we want to have for better testing,
and consistent JS transpilation).
Note that the removal of `cldr-data-downloader` also requires us to
add logic for detecting locales without data. The CLDR data
downloader overwrote the `availableLocales.json` file with a file
that only lists locales that CLDR provides data for. We use the
official `availableLocales` file CLDR provides, but filter out
locales for which no data is available. This is needed until we
update to CLDR 39 where data is available for all such locales
listed in `availableLocales.json`.
PR Close#42230
This is a pre-refactor commit allowing us to move
the CLDR locale generation to Bazel where files would
no longer be checked-in, except for the `closure-locale`
file that is synced into Google3.
PR Close#42230
Within Google, closure compiler is used for dealing with translations.
We generate a closure-compatible locale file that allows for
registration within Angular, so that Closure i18n works well together
with Angular applications. Closure compiler does not limit its
locales to BCP47-canonical locale identifiers. This commit updates
the generation logic so that we also support deprecated (but aliased)
locale identifiers, or other aliases which are likely used within
Closure. We use CLDR's alias supplemental data for this. It instructs
us to alias `iw` to `he` for example. `iw` is still supported in Closure.
Note that we do not manually extract all locales supported in Closure;
instead we only support the CLDR canonical locales (as done before) +
common aliases that CLDR provides data for. We are not aware of other
locale aliases within Closure that wouldn't be part of the CLDR aliases.
If there would be, then Angular/Closure would fail accordingly.
PR Close#42230
In the past, the closure file has been generated so that all individual
locale files were imported individually. This resulted in a huge
slow-down in g3 due to the large amount of imports.
With 90bd984ff7 this changed so that we
inline the locale data for the g3 closure locale file. Also the file
only contained data for locales being supported by Closure. For this a
list of locales has been extracted from Closure Compiler, as well as a
list of locale aliases.
This logic is prone to CLDR version updates, and also broke as part of
the Gulp -> Bazel migration where this logic has been slightly modified
but caused issues in G3. e.g. a locale `zh-Hant` was requested in g3,
but the locale data had the name of the alias locale that provided the
data at index zero (which represents the locale name). Note that the
locale names at index zero always could differentiate from the requested
`goog.LOCALE` due to the aliasing logic. This just didn't come up before.
We simplify this logic by generating a `goog.LOCALE` case for all
locales CLDR provides data for. We don't need to bother about aliasing
because with the refactorings to the CLDR generation tool, all locales
are built (which also captures the aliases), and we can generate the locale
file on the fly (which has not been done before).
PR Close#42230
The CLDR extraction tool has been reworked to run as part of Bazel.
This adds a initial readme explaining what the tool generates. It's
far from a detailed description but it can serve as foundation for more
detailed explanations.
PR Close#42230
Introduces a few Starlark macros for running the new Bazel
CLDR generation tool. Wires up the new tool so that locales
are generated properly. Also updates the existing
`closure-locale` file to match the new output generated by the Bazel tool.
This commit also re-adds a few locale files that aren't
generated by CLDR 37, but have been accidentally left in
the repository as the Gulp script never removed old locales
from previous CLDR versions. This problem is solved with the
Bazel generation of locale files, but for now we re-add these
old CLDR 33 locale files to not break developers relying on these
(even though the locale data indicies are incorrect; but there might
be users accessing the data directly)
PR Close#42230
Converts the CLDR locale extraction script to a Bazel tool.
This allows us to generate locale files within Bazel, so that
locales don't need to live as sources within the repo. Also
it allows us to get rid of the legacy Gulp tooling.
The migration of the Gulp script to a Bazel tool involved the
following things:
1. Basic conversion of the `extract.js` script to TypeScript.
This mostly was about adding explicit types. e.g. adding `locale:
string` or `localeData: CldrStatic`.
2. Split-up into separate files. Instead of keeping the large
`extract.js` file, the tool has been split into separate files.
The logic remains the same, just that code is more readable and
maintainable.
3. Introduction of a new `index.ts` file that is the entry-point
for the Bazel tool. Previously the Gulp tool just generated
all locale files, the default locale and base currency files
at once. The new entry-point accepts a mode to be passed as
first process argument. based on that argument, either locales
are generated into a specified directory, or the default locale,
base currencies or closure file is generated.
This allows us to generate files with a Bazel genrule where
we simply run the tool and specify the outputs. Note: It's
necessary to have multiple modes because files live in separate
locations. e.g. the default locale in `@angular/core`, but the
rest in `@angular/common`.
4. Removal of the `cldr-data-downloader` and custom CLDR resolution
logic. Within Bazel we cannot run a downloader using network.
We switch this to something more Bazel idiomatic with better
caching. For this a new repository rule is introduced that
downloads the CLDR JSON repository and extracts it. Within
that rule we determine the supported locales so that they
can be used to pre-declare outputs (for the locales) within
Bazel analysis phase. This allows us to add the generated locale
files to a `ts_library` (which we want to have for better testing,
and consistent JS transpilation).
Note that the removal of `cldr-data-downloader` also requires us to
add logic for detecting locales without data. The CLDR data
downloader overwrote the `availableLocales.json` file with a file
that only lists locales that CLDR provides data for. We use the
official `availableLocales` file CLDR provides, but filter out
locales for which no data is available. This is needed until we
update to CLDR 39 where data is available for all such locales
listed in `availableLocales.json`.
PR Close#42230
This is a pre-refactor commit allowing us to move
the CLDR locale generation to Bazel where files would
no longer be checked-in, except for the `closure-locale`
file that is synced into Google3.
PR Close#42230
In the past, the closure file has been generated so that all individual
locale files were imported individually. This resulted in a huge
slow-down in g3 due to the large amount of imports.
With 90bd984ff7 this changed so that we
inline the locale data for the g3 closure locale file. Also the file
only contained data for locales being supported by Closure. For this a
list of locales has been extracted from Closure Compiler, as well as a
list of locale aliases.
This logic is prone to CLDR version updates, and also broke as part of
the Gulp -> Bazel migration where this logic has been slightly modified
but caused issues in G3. e.g. a locale `zh-Hant` was requested in g3,
but the locale data had the name of the alias locale that provided the
data at index zero (which represents the locale name). Note that the
locale names at index zero always could differentiate from the requested
`goog.LOCALE` due to the aliasing logic. This just didn't come up before.
We simplify this logic by generating a `goog.LOCALE` case for all
locales CLDR provides data for. We don't need to bother about aliasing
because with the refactorings to the CLDR generation tool, all locales
are built (which also captures the aliases), and we can generate the locale
file on the fly (which has not been done before).
PR Close#42230
The CLDR extraction tool has been reworked to run as part of Bazel.
This adds a initial readme explaining what the tool generates. It's
far from a detailed description but it can serve as foundation for more
detailed explanations.
PR Close#42230
Introduces a few Starlark macros for running the new Bazel
CLDR generation tool. Wires up the new tool so that locales
are generated properly. Also updates the existing
`closure-locale` file to match the new output generated by the Bazel tool.
This commit also re-adds a few locale files that aren't
generated by CLDR 37, but have been accidentally left in
the repository as the Gulp script never removed old locales
from previous CLDR versions. This problem is solved with the
Bazel generation of locale files, but for now we re-add these
old CLDR 33 locale files to not break developers relying on these
(even though the locale data indicies are incorrect; but there might
be users accessing the data directly)
PR Close#42230
Converts the CLDR locale extraction script to a Bazel tool.
This allows us to generate locale files within Bazel, so that
locales don't need to live as sources within the repo. Also
it allows us to get rid of the legacy Gulp tooling.
The migration of the Gulp script to a Bazel tool involved the
following things:
1. Basic conversion of the `extract.js` script to TypeScript.
This mostly was about adding explicit types. e.g. adding `locale:
string` or `localeData: CldrStatic`.
2. Split-up into separate files. Instead of keeping the large
`extract.js` file, the tool has been split into separate files.
The logic remains the same, just that code is more readable and
maintainable.
3. Introduction of a new `index.ts` file that is the entry-point
for the Bazel tool. Previously the Gulp tool just generated
all locale files, the default locale and base currency files
at once. The new entry-point accepts a mode to be passed as
first process argument. based on that argument, either locales
are generated into a specified directory, or the default locale,
base currencies or closure file is generated.
This allows us to generate files with a Bazel genrule where
we simply run the tool and specify the outputs. Note: It's
necessary to have multiple modes because files live in separate
locations. e.g. the default locale in `@angular/core`, but the
rest in `@angular/common`.
4. Removal of the `cldr-data-downloader` and custom CLDR resolution
logic. Within Bazel we cannot run a downloader using network.
We switch this to something more Bazel idiomatic with better
caching. For this a new repository rule is introduced that
downloads the CLDR JSON repository and extracts it. Within
that rule we determine the supported locales so that they
can be used to pre-declare outputs (for the locales) within
Bazel analysis phase. This allows us to add the generated locale
files to a `ts_library` (which we want to have for better testing,
and consistent JS transpilation).
Note that the removal of `cldr-data-downloader` also requires us to
add logic for detecting locales without data. The CLDR data
downloader overwrote the `availableLocales.json` file with a file
that only lists locales that CLDR provides data for. We use the
official `availableLocales` file CLDR provides, but filter out
locales for which no data is available. This is needed until we
update to CLDR 39 where data is available for all such locales
listed in `availableLocales.json`.
PR Close#42230
This is a pre-refactor commit allowing us to move
the CLDR locale generation to Bazel where files would
no longer be checked-in, except for the `closure-locale`
file that is synced into Google3.
PR Close#42230
Update to the latest version of bazel.
`4.0.0` introduced a breaking change on unnecessary backslashes and these
instance are corrected in this change.
PR Close#40579
Historically files to be formatted were added to a listing (via matchers)
to be included in formatting. Instead, this change begins efforts to
instead include all files in format enforcement, relying instead on an
opt out methodology.
PR Close#36940
This reverts commit 078b0be4dc.
The original commit was a work around for a bug in CLI. That bug was fixed in the CLI, as a result this change is no longer needed and is being reverted.
PR Close#37074
Although this code has been part of Angular 9.x I only noticed this
error when upgrading to Angular 9.1.x because historically the source
locale data was not injected when localizing, but as of
angular/angular-cli#16394 (9.1.0) it is now included. This tipped me off
that my other bundles were not being built properly, and this change
allows me to build a valid ES5 bundle (I have also added a verification
step to my build pipeline to alert me if this error appears again in any
of my bundles).
I found the `locales/global/*.js` file paths being referenced by the
`I18nOptions` in
@angular-devkit/build-angular/src/utils/i18n-options.ts,
and following that it looks like it is actually loaded and used in
@angular-devkit/build-angular/src/utils/process-bundle.ts. I saw the
function `terserMangle` does appear that it is likely aware of the build
being ES5, but I'm not sure why this is not producing a valid ES5
bundle.
This change updates `tools/gulp-tasks/cldr/extract.js` to produce ES5
compliant `locales/global/*.js` and that fixes my issue. However, I am
not sure if @angular-devkit/build-angular should be modified to produce
a valid ES5 bundle instead or if the files could be TypeScript rather
than JavaScript files.
A test that a valid ES5 bundle is produced would be helpful, and I hope
this is reproducible and not some issue with my config.
PR Close#36342
The locale data extraction has been modified to include the default
currency code in the generated locale data. This commit updates these
generated files accordingly.
PR Close#32584
When directionality data was added to @angular/common locales, only
the externally used locales were updated. We need to additionally
update the closure locales which are synced into google3.
PR Close#34240
In order to support adding locales during compile-time
inlining of translations (i.e. after the TS build has completed),
we need to be able to attach the locale to the global scope.
This commit modifies CLDR extraction to emit additional "global"
locale files that appear in the `@angular/common/locales/global` folder.
These files are of the form:
```
(function() {
const root = typeof globalThis !== 'undefined' && globalThis ||
typeof global !== 'undefined' && global || typeof window !== 'undefined' && window;
root.ng = root.ng || {};
root.ng.common = root.ng.common || {};
root.ng.common.locale = root.ng.common.locale || {};
const u = undefined;
function plural(n) {
if (n === 1) return 1;
return 5;
}
root.ng.common.locale['xx-yy'] = [...];
})();
```
The IIFE will ensure that `ng.common.locale` exists and attach the
given locale (and its "extras") to it using it "normalized" locale
name.
* "extras": in the UMD module locale files the "extra" locale data,
currently the day period rules, and extended day period data, are
stored in separate files under the "common/locales/extra" folder.
* "normalized": Angular references locales using a normalized form,
which is lower case with `_` replaced by `-`. For example:
`en_UK` => `en-uk`.
PR Close#33523
Paul Gschwendtner
Alex Rickabaugh
Jessica Janiuk
Joey Perrott
Misko Hevery
Terence D. Honles
Greg Magolan
Pete Bacon Darwin
atscott
Joey Perrott
Feliks Khantsis
Alex Eagle
Wassim Chegham