9456eca7c5
Adds better caching for browser archives and their extraction. This is done because the archives are currently extracted as a build action and these are actions are invalidated frequently, causing flakiness on the CI and slow-down in local development. Here is an example flaky error on the CI (that surfaces often with RBE execution): ``` ERROR: /home/circleci/.cache/bazel/_bazel_circleci/9ce5c2144ecf75d11717c0aa41e45a8d/external/npm/@angular/dev-infra-private/bazel/browsers/chromium/BUILD.bazel:22:17: Extracting ../org_chromium_chromium_amd64/file/chrome-linux.zip failed: (Exit 34): extract.sh failed: error executing command external/io_bazel_rules_webtesting/web/internal/extract.sh external/org_chromium_chromium_amd64/file/chrome-linux.zip ... (remaining 2 argument(s) skipped). Note: Remote connection/protocol failed with: execution failed ``` We fix this by introducing a new rule that downloads a browser archive and unpacks it directly into a Bazel repository. Before this change, the archive would just be downloaded but extracted later as part of a build action. This is unnecessary and results in less efficient caching as build actions are invalidated more often, especially if developers run `bazel clean` in between. The root cause on why the extraction often fails in RBE containers is unclear. It's unclear why the extacted archive is not cached properly as part of a build action (most likely some hermeticity issue within `rules_webtesting`, but it seems more Bazel-idiomatic to unpack the archives as part of the repository anyway, and this solves the flakiness issue. PR Close #42814
81 lines
3.7 KiB
Python
81 lines
3.7 KiB
Python
"""Implementation of the `browser_archive` rule."""
|
|
|
|
def _browser_archive_impl(ctx):
|
|
ctx.report_progress("Downloading browser archive from: %s" % ctx.attr.url)
|
|
ctx.download_and_extract(
|
|
url = ctx.attr.url,
|
|
sha256 = ctx.attr.sha256,
|
|
)
|
|
|
|
# The browser archive has been downloaded and extracted. We now generate a repository
|
|
# `BUILD.bazel` file that exposes the archive files, together with the specified
|
|
# named files using the `browser_configure` rule.
|
|
ctx.file("BUILD.bazel", content = """
|
|
load("@angular//dev-infra/bazel/browsers:browser_configure.bzl", "browser_configure")
|
|
|
|
licenses(%s)
|
|
|
|
browser_configure(
|
|
name = "metadata",
|
|
files = glob(["**/*"]),
|
|
named_files = %s,
|
|
visibility = ["//visibility:public"],
|
|
)
|
|
""" % (str(ctx.attr.licenses), str(ctx.attr.named_files)))
|
|
|
|
"""
|
|
Rule that can be used to download and unpack a browser archive in a dedicated Bazel
|
|
repository. Additionally, files within the archive can be denoted with an unique name
|
|
so that web tests can access browser files in a platform-agnostic way, regardless of
|
|
which `browser_archive` repository is added as dependency.
|
|
|
|
As an example for the concept of denoting archive files with an unique name, consider a case
|
|
where a a web test decides conditionally based on the current exec platform which
|
|
`browser_archive` repository is used (e.g. mac, windows or linux). The archives are different
|
|
for each platform. The test usually would need to determine the current platform, and know how
|
|
each archive is structured in order to access the browser binary within the repository. By
|
|
defining named files though, the web test could just pull a named file called `BINARY` that
|
|
always resolves to the browser binary in a platform-agnostic way.
|
|
|
|
Note #1: This rule exists as an alternative to the `platform_http_file` concept
|
|
from `rules_webtesting` because the `platform_http_file` rule does not extract the archive
|
|
directly, but relies on later build actions to perform the unpacking. This results in less
|
|
efficient caching because build actions are invalidated more frequently (e.g. `bazel clean).
|
|
We also noticed that the extraction within RBE containers is rather unstable, and extracting
|
|
the archives as part of a Bazel repository mitigates this (as extractions happens on the host).
|
|
|
|
Note #2: Additionally `rules_webtesting` defines a single repository for all platforms,
|
|
where only an archive for the current host platform is pulled. This breaks cross-compilation
|
|
because the wrong platform archive would be used for web tests that run in the exec platform.
|
|
"""
|
|
browser_archive = repository_rule(
|
|
implementation = _browser_archive_impl,
|
|
attrs = {
|
|
"url": attr.string(
|
|
doc = "Browser archive to download and extract.",
|
|
mandatory = True,
|
|
),
|
|
"sha256": attr.string(
|
|
doc = "SHA256 checksum for the archive.",
|
|
mandatory = True,
|
|
),
|
|
"licenses": attr.string_list(
|
|
mandatory = True,
|
|
allow_empty = False,
|
|
doc = """
|
|
Licenses that apply to the archive. Will be passed to a `licenses` invocation
|
|
within the repository. https://docs.bazel.build/versions/0.24.0/be/functions.html#licenses.
|
|
""",
|
|
),
|
|
"named_files": attr.string_dict(
|
|
doc = """
|
|
Dictionary that maps files to unique identifiers. This is useful
|
|
if browser archives are different on different platforms and the web
|
|
tests would not want to care about archive-specific paths. e.g. targets
|
|
expect a `CHROMIUM` key to point to the Chromium browser binary.
|
|
""",
|
|
mandatory = True,
|
|
),
|
|
},
|
|
)
|