feat(dev-infra): better caching for browser archive contents (#42814)

Adds better caching for browser archives and their extraction.
This is done because the archives are currently extracted as a build
action and these are actions are invalidated frequently, causing
flakiness on the CI and slow-down in local development.

Here is an example flaky error on the CI (that surfaces often
with RBE execution):

```
ERROR:
/home/circleci/.cache/bazel/_bazel_circleci/9ce5c2144ecf75d11717c0aa41e45a8d/external/npm/@angular/dev-infra-private/bazel/browsers/chromium/BUILD.bazel:22:17:
Extracting ../org_chromium_chromium_amd64/file/chrome-linux.zip failed:
(Exit 34): extract.sh failed: error executing command
external/io_bazel_rules_webtesting/web/internal/extract.sh
external/org_chromium_chromium_amd64/file/chrome-linux.zip ...
(remaining 2 argument(s) skipped). Note: Remote connection/protocol
failed with: execution failed
```

We fix this by introducing a new rule that downloads a browser
archive and unpacks it directly into a Bazel repository. Before
this change, the archive would just be downloaded but extracted
later as part of a build action. This is unnecessary and results
in less efficient caching as build actions are invalidated more
often, especially if developers run `bazel clean` in between.

The root cause on why the extraction often fails in RBE containers
is unclear. It's unclear why the extacted archive is not cached
properly as part of a build action (most likely some hermeticity
issue within `rules_webtesting`, but it seems more Bazel-idiomatic
to unpack the archives as part of the repository anyway, and this solves
the flakiness issue.

PR Close #42814
This commit is contained in:
Paul Gschwendtner 2021-07-10 17:33:54 +02:00 committed by Andrew Kushnir
parent a524af15a5
commit 9456eca7c5
9 changed files with 229 additions and 176 deletions

View File

@ -57,6 +57,7 @@ pkg_npm(
substitutions = {
# angular/angular should not consume it's own packages, so we use
# substitutions to replace these in the published version of dev-infra.
"@angular//dev-infra/": "@npm//@angular/dev-infra-private/",
"//dev-infra/": "@npm//@angular/dev-infra-private/",
"//dev-infra:": "@npm//@angular/dev-infra-private:",

View File

@ -0,0 +1,80 @@
"""Implementation of the `browser_archive` rule."""
def _browser_archive_impl(ctx):
ctx.report_progress("Downloading browser archive from: %s" % ctx.attr.url)
ctx.download_and_extract(
url = ctx.attr.url,
sha256 = ctx.attr.sha256,
)
# The browser archive has been downloaded and extracted. We now generate a repository
# `BUILD.bazel` file that exposes the archive files, together with the specified
# named files using the `browser_configure` rule.
ctx.file("BUILD.bazel", content = """
load("@angular//dev-infra/bazel/browsers:browser_configure.bzl", "browser_configure")
licenses(%s)
browser_configure(
name = "metadata",
files = glob(["**/*"]),
named_files = %s,
visibility = ["//visibility:public"],
)
""" % (str(ctx.attr.licenses), str(ctx.attr.named_files)))
"""
Rule that can be used to download and unpack a browser archive in a dedicated Bazel
repository. Additionally, files within the archive can be denoted with an unique name
so that web tests can access browser files in a platform-agnostic way, regardless of
which `browser_archive` repository is added as dependency.
As an example for the concept of denoting archive files with an unique name, consider a case
where a a web test decides conditionally based on the current exec platform which
`browser_archive` repository is used (e.g. mac, windows or linux). The archives are different
for each platform. The test usually would need to determine the current platform, and know how
each archive is structured in order to access the browser binary within the repository. By
defining named files though, the web test could just pull a named file called `BINARY` that
always resolves to the browser binary in a platform-agnostic way.
Note #1: This rule exists as an alternative to the `platform_http_file` concept
from `rules_webtesting` because the `platform_http_file` rule does not extract the archive
directly, but relies on later build actions to perform the unpacking. This results in less
efficient caching because build actions are invalidated more frequently (e.g. `bazel clean).
We also noticed that the extraction within RBE containers is rather unstable, and extracting
the archives as part of a Bazel repository mitigates this (as extractions happens on the host).
Note #2: Additionally `rules_webtesting` defines a single repository for all platforms,
where only an archive for the current host platform is pulled. This breaks cross-compilation
because the wrong platform archive would be used for web tests that run in the exec platform.
"""
browser_archive = repository_rule(
implementation = _browser_archive_impl,
attrs = {
"url": attr.string(
doc = "Browser archive to download and extract.",
mandatory = True,
),
"sha256": attr.string(
doc = "SHA256 checksum for the archive.",
mandatory = True,
),
"licenses": attr.string_list(
mandatory = True,
allow_empty = False,
doc = """
Licenses that apply to the archive. Will be passed to a `licenses` invocation
within the repository. https://docs.bazel.build/versions/0.24.0/be/functions.html#licenses.
""",
),
"named_files": attr.string_dict(
doc = """
Dictionary that maps files to unique identifiers. This is useful
if browser archives are different on different platforms and the web
tests would not want to care about archive-specific paths. e.g. targets
expect a `CHROMIUM` key to point to the Chromium browser binary.
""",
mandatory = True,
),
},
)

View File

@ -0,0 +1,69 @@
load("@io_bazel_rules_webtesting//web/internal:metadata.bzl", "metadata")
load("@io_bazel_rules_webtesting//web/internal:provider.bzl", "WebTestInfo")
"""Converts the specified label to a manifest path"""
def _label_to_manifest_path(label):
if label.package != "":
return "%s/%s" % (label.workspace_name, label.package)
return label.workspace_name
"""Implementation of the `browser_configure` rule."""
def _browser_configure_impl(ctx):
named_files = {}
base_dir = _label_to_manifest_path(ctx.label)
# Update the named files to manifest paths that can be resolved
# with Bazel runfile resolution in web tests.
for n, p in ctx.attr.named_files.items():
named_files[n] = base_dir + "/" + p
# Create a web test metadata file that will be provided as part of
# the `WebTestInfo` provider.
metadata.create_file(
ctx = ctx,
output = ctx.outputs.web_test_metadata,
web_test_files = [
metadata.web_test_files(ctx = ctx, named_files = named_files),
],
)
return [
DefaultInfo(runfiles = ctx.runfiles(files = ctx.files.files)),
WebTestInfo(metadata = ctx.outputs.web_test_metadata),
]
"""
Rule that is used in combination with the `browser_archive` rule. It captures a set
of files which are needed for dealing with a browser. Additionally, specific files
for the browser can be denoted with an unique name so that web tests can access browser
files in a platform-agnostic way, regardless of which browser repository is selected.
The specified browser files are exposed as runfiles of the target defined through this
rule. The unique names with their associated files are captured within a metadata file
that is exposed through a `WebTestInfo` provider. Web tests will be able to deal with
this metadata file to resolve browser files in a platform-agnostic way.
More details on this can be found in the `browser_archive` rule.
"""
browser_configure = rule(
attrs = {
"files": attr.label_list(
mandatory = True,
allow_files = True,
doc = "List of files which are needed for the browser.",
),
"named_files": attr.string_dict(
doc = """
Dictionary that maps files to unique identifiers. This is useful
if browser archives are different on different platforms and the web
tests would not want to care about archive-specific paths. e.g. targets
expect a `CHROMIUM` key to point to the Chromium browser binary.
""",
mandatory = True,
),
},
outputs = {"web_test_metadata": "%{name}.gen.json"},
implementation = _browser_configure_impl,
)

View File

@ -1,17 +1,3 @@
# Copyright 2018 The Bazel Authors. All rights reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
"""Pinned browser versions.
This function is here to make browser repositories work with cross-platform RBE.

View File

@ -1,74 +1,26 @@
load("@io_bazel_rules_webtesting//web:web.bzl", "browser", "web_test_archive")
load("@io_bazel_rules_webtesting//web:web.bzl", "browser")
# Copyright Google LLC
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#
################################################################################
#
package(default_visibility = ["//visibility:public"])
# Override of chromium web_test_archive so that the archive is selected based on platform
web_test_archive(
name = "chromium_archive",
testonly = True,
archive = select({
"@io_bazel_rules_webtesting//common/conditions:linux": "@org_chromium_chromium_amd64//file",
"@io_bazel_rules_webtesting//common/conditions:mac": "@org_chromium_chromium_macos//file",
"@io_bazel_rules_webtesting//common/conditions:windows": "@org_chromium_chromium_windows//file",
}),
extract = "build",
named_files = select({
"@io_bazel_rules_webtesting//common/conditions:linux": {"CHROMIUM": "chrome-linux/chrome"},
"@io_bazel_rules_webtesting//common/conditions:mac": {"CHROMIUM": "chrome-mac/Chromium.app/Contents/MacOS/chromium"},
"@io_bazel_rules_webtesting//common/conditions:windows": {"CHROMIUM": "chrome-win/chrome.exe"},
}),
visibility = ["//dev-infra/bazel/browsers:__subpackages__"],
)
# Override of chromedriver web_test_archive so that the archive is selected based on platform
web_test_archive(
name = "chromedriver_archive",
testonly = True,
archive = select({
"@io_bazel_rules_webtesting//common/conditions:linux": "@org_chromium_chromedriver_amd64//file",
"@io_bazel_rules_webtesting//common/conditions:mac": "@org_chromium_chromedriver_macos//file",
"@io_bazel_rules_webtesting//common/conditions:windows": "@org_chromium_chromedriver_windows//file",
}),
extract = "build",
named_files = select({
"@io_bazel_rules_webtesting//common/conditions:linux": {
"CHROMEDRIVER": "chromedriver_linux64/chromedriver",
},
"@io_bazel_rules_webtesting//common/conditions:mac": {
"CHROMEDRIVER": "chromedriver_mac64/chromedriver",
},
"@io_bazel_rules_webtesting//common/conditions:windows": {
"CHROMEDRIVER": "chromedriver_win32/chromedriver.exe",
},
}),
visibility = ["//dev-infra/bazel/browsers:__subpackages__"],
)
browser(
name = "chromium",
metadata = "chromium.json",
visibility = ["//visibility:public"],
deps = [
":chromedriver_archive",
":chromium_archive",
"@io_bazel_rules_webtesting//go/wsl",
] + select({
"@io_bazel_rules_webtesting//common/conditions:linux": [
"@org_chromium_chromedriver_amd64//:metadata",
"@org_chromium_chromium_amd64//:metadata",
],
"@io_bazel_rules_webtesting//common/conditions:mac": [
"@org_chromium_chromedriver_macos//:metadata",
"@org_chromium_chromium_macos//:metadata",
],
"@io_bazel_rules_webtesting//common/conditions:windows": [
"@org_chromium_chromedriver_windows//:metadata",
"@org_chromium_chromium_windows//:metadata",
],
}),
)
# Make source files available for distribution via pkg_npm

View File

@ -1,4 +1,4 @@
load("//dev-infra/bazel/browsers:platform_http_file.bzl", "platform_http_file")
load("//dev-infra/bazel/browsers:browser_archive_repo.bzl", "browser_archive")
"""
Defines repositories for Chromium that can be used inside Karma unit tests
@ -9,50 +9,68 @@ def define_chromium_repositories():
# To update to a newer version of Chromium see instructions in
# https://github.com/angular/angular/blob/master/dev-infra/bazel/browsers/README.md.
platform_http_file(
browser_archive(
name = "org_chromium_chromium_amd64",
licenses = ["notice"], # BSD 3-clause (maybe more?)
sha256 = "36759ed6d151645d00a3a015200334edc70188b422eec51bcaa5790c8e906e27",
# 87.0.4280
urls = ["https://commondatastorage.googleapis.com/chromium-browser-snapshots/Linux_x64/812847/chrome-linux.zip"],
url = "https://commondatastorage.googleapis.com/chromium-browser-snapshots/Linux_x64/812847/chrome-linux.zip",
named_files = {
"CHROMIUM": "chrome-linux/chrome",
},
)
platform_http_file(
browser_archive(
name = "org_chromium_chromium_macos",
licenses = ["notice"], # BSD 3-clause (maybe more?)
sha256 = "e10533c84ef57232975d6bde9cd28fd0354371e9556dda85e01178e6dcd56b93",
# 87.0.4280
urls = ["https://commondatastorage.googleapis.com/chromium-browser-snapshots/Mac/812851/chrome-mac.zip"],
url = "https://commondatastorage.googleapis.com/chromium-browser-snapshots/Mac/812851/chrome-mac.zip",
named_files = {
"CHROMIUM": "chrome-mac/Chromium.app/Contents/MacOS/chromium",
},
)
platform_http_file(
browser_archive(
name = "org_chromium_chromium_windows",
licenses = ["notice"], # BSD 3-clause (maybe more?)
sha256 = "40d0dec1892d729db2f7d8f27feff762b070a02f04d4e14f4e37b97d6b7c3c8f",
# 87.0.4280
urls = ["https://commondatastorage.googleapis.com/chromium-browser-snapshots/Win/812822/chrome-win.zip"],
url = "https://commondatastorage.googleapis.com/chromium-browser-snapshots/Win/812822/chrome-win.zip",
named_files = {
"CHROMIUM": "chrome-win/chrome.exe",
},
)
platform_http_file(
browser_archive(
name = "org_chromium_chromedriver_amd64",
licenses = ["reciprocal"], # BSD 3-clause, ICU, MPL 1.1, libpng (BSD/MIT-like), Academic Free License v. 2.0, BSD 2-clause, MIT
sha256 = "d859f8ecb21e26d3ddaf3f229da695bc86512f4e6c9fe32533af7a8b36783ec5",
# 87.0.4280
urls = ["https://commondatastorage.googleapis.com/chromium-browser-snapshots/Linux_x64/812847/chromedriver_linux64.zip"],
url = "https://commondatastorage.googleapis.com/chromium-browser-snapshots/Linux_x64/812847/chromedriver_linux64.zip",
named_files = {
"CHROMEDRIVER": "chromedriver_linux64/chromedriver",
},
)
platform_http_file(
browser_archive(
name = "org_chromium_chromedriver_macos",
licenses = ["reciprocal"], # BSD 3-clause, ICU, MPL 1.1, libpng (BSD/MIT-like), Academic Free License v. 2.0, BSD 2-clause, MIT
sha256 = "aa7a99fa23287725d7108cc07baa94e6f0ef4171ff7b134018387a939a67d93d",
# 87.0.4280
urls = ["https://commondatastorage.googleapis.com/chromium-browser-snapshots/Mac/812851/chromedriver_mac64.zip"],
url = "https://commondatastorage.googleapis.com/chromium-browser-snapshots/Mac/812851/chromedriver_mac64.zip",
named_files = {
"CHROMEDRIVER": "chromedriver_mac64/chromedriver",
},
)
platform_http_file(
browser_archive(
name = "org_chromium_chromedriver_windows",
licenses = ["reciprocal"], # BSD 3-clause, ICU, MPL 1.1, libpng (BSD/MIT-like), Academic Free License v. 2.0, BSD 2-clause, MIT
sha256 = "826f2bd0c50b823e7642860ed08cacf69d3756002a71ac30cdd77c68f31d2d24",
# 87.0.4280
urls = ["https://commondatastorage.googleapis.com/chromium-browser-snapshots/Win/812822/chromedriver_win32.zip"],
url = "https://commondatastorage.googleapis.com/chromium-browser-snapshots/Win/812822/chromedriver_win32.zip",
named_files = {
"CHROMEDRIVER": "chromedriver_win32/chromedriver.exe",
},
)

View File

@ -1,52 +1,7 @@
load("@io_bazel_rules_webtesting//web:web.bzl", "browser", "web_test_archive")
load("@io_bazel_rules_webtesting//web:web.bzl", "browser")
# Copyright Google LLC
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#
################################################################################
#
package(default_visibility = ["//visibility:public"])
# Override of firefox web_test_archive so that the archive is selected based on platform.
web_test_archive(
name = "firefox_archive",
testonly = True,
archive = select({
"@io_bazel_rules_webtesting//common/conditions:linux": "@org_mozilla_firefox_amd64//file",
"@io_bazel_rules_webtesting//common/conditions:mac": "@org_mozilla_firefox_macos//file",
}),
extract = "build",
named_files = select({
"@io_bazel_rules_webtesting//common/conditions:linux": {"FIREFOX": "firefox/firefox"},
"@io_bazel_rules_webtesting//common/conditions:mac": {"FIREFOX": "Firefox.app/Contents/MacOS/firefox"},
}),
visibility = ["//dev-infra/bazel/browsers:__subpackages__"],
)
# Override of geckodriver web_test_archive so that the archive is selected based on platform.
web_test_archive(
name = "geckodriver_archive",
testonly = True,
archive = select({
"@io_bazel_rules_webtesting//common/conditions:linux": "@org_mozilla_geckodriver_amd64//file",
"@io_bazel_rules_webtesting//common/conditions:mac": "@org_mozilla_geckodriver_macos//file",
}),
extract = "build",
named_files = {"GECKODRIVER": "geckodriver"},
visibility = ["//dev-infra/bazel/browsers:__subpackages__"],
)
browser(
name = "firefox",
disabled = select({
@ -56,14 +11,18 @@ browser(
"//conditions:default": None,
}),
metadata = "firefox.json",
visibility = ["//visibility:public"],
deps = select({
"@io_bazel_rules_webtesting//common/conditions:windows": [],
"//conditions:default": [
deps = [
"@io_bazel_rules_webtesting//go/wsl",
":firefox_archive",
":geckodriver_archive",
] + select({
"@io_bazel_rules_webtesting//common/conditions:linux": [
"@org_mozilla_firefox_amd64//:metadata",
"@org_mozilla_geckodriver_amd64//:metadata",
],
"@io_bazel_rules_webtesting//common/conditions:mac": [
"@org_mozilla_firefox_macos//:metadata",
"@org_mozilla_geckodriver_macos//:metadata",
],
"@io_bazel_rules_webtesting//common/conditions:windows": [],
}),
)

View File

@ -1,4 +1,4 @@
load("//dev-infra/bazel/browsers:platform_http_file.bzl", "platform_http_file")
load("//dev-infra/bazel/browsers:browser_archive_repo.bzl", "browser_archive")
"""
Defines repositories for Firefox that can be used inside Karma unit tests
@ -9,34 +9,46 @@ def define_firefox_repositories():
# Instructions on updating the Firefox version can be found in the `README.md` file
# next to this file.
platform_http_file(
browser_archive(
name = "org_mozilla_firefox_amd64",
licenses = ["reciprocal"], # MPL 2.0
sha256 = "601e5a9a12ce680ecd82177c7887dae008d8f33690da43be1a690b76563cd992",
# Firefox v84.0
urls = ["https://ftp.mozilla.org/pub/firefox/releases/84.0/linux-x86_64/en-US/firefox-84.0.tar.bz2"],
url = "https://ftp.mozilla.org/pub/firefox/releases/84.0/linux-x86_64/en-US/firefox-84.0.tar.bz2",
named_files = {
"FIREFOX": "firefox/firefox",
},
)
platform_http_file(
browser_archive(
name = "org_mozilla_firefox_macos",
licenses = ["reciprocal"], # MPL 2.0
sha256 = "4c7bca050eb228f4f6f93a9895af0a87473e03c67401d1d2f1ba907faf87fefd",
# Firefox v84.0
urls = ["https://ftp.mozilla.org/pub/firefox/releases/84.0/mac/en-US/Firefox%2084.0.dmg"],
url = "https://ftp.mozilla.org/pub/firefox/releases/84.0/mac/en-US/Firefox%2084.0.dmg",
named_files = {
"FIREFOX": "Firefox.app/Contents/MacOS/firefox",
},
)
platform_http_file(
browser_archive(
name = "org_mozilla_geckodriver_amd64",
licenses = ["reciprocal"], # MPL 2.0
sha256 = "61bfc547a623d7305256611a81ecd24e6bf9dac555529ed6baeafcf8160900da",
# Geckodriver v0.28.0
urls = ["https://github.com/mozilla/geckodriver/releases/download/v0.28.0/geckodriver-v0.28.0-linux64.tar.gz"],
url = "https://github.com/mozilla/geckodriver/releases/download/v0.28.0/geckodriver-v0.28.0-linux64.tar.gz",
named_files = {
"GECKODRIVER": "geckodriver",
},
)
platform_http_file(
browser_archive(
name = "org_mozilla_geckodriver_macos",
licenses = ["reciprocal"], # MPL 2.0
sha256 = "c288ff6db39adfd5eea0e25b4c3e71bfd9fb383eccf521cdd65f67ea78eb1761",
# Geckodriver v0.28.0
urls = ["https://github.com/mozilla/geckodriver/releases/download/v0.28.0/geckodriver-v0.28.0-macos.tar.gz"],
url = "https://github.com/mozilla/geckodriver/releases/download/v0.28.0/geckodriver-v0.28.0-macos.tar.gz",
named_files = {
"GECKODRIVER": "geckodriver",
},
)

View File

@ -1,24 +0,0 @@
load(
"@io_bazel_rules_webtesting//web/internal:platform_http_file.bzl",
_platform_http_file = "platform_http_file",
)
def platform_http_file(name, licenses, sha256, urls):
"""Platform specific browser repository.
This works around a deficiency in io_bazel_rules_webtesting platform_http_file in that
it selects the platform when the repository rule is executed. This limits browsers
tests to run on the local user platform only. For cross-platform RBE we want a repository
to be defined per platform so the correct one can be selected.
"""
_platform_http_file(
name = name,
amd64_sha256 = sha256,
amd64_urls = urls,
licenses = licenses,
macos_sha256 = sha256,
macos_urls = urls,
windows_sha256 = sha256,
windows_urls = urls,
)