ci: enable bazel remote caching on CircleCI (#21784)

This should cause Bazel builds to be incremental, only re-building parts of Angular affected by changes since the last build.
It also fixes a potential version skew, where CI was running the Bazel linter binaries in the ngcontainer docker image, but developers built them using the versions in WORKSPACE

PR Close #21784

.circleci/bazel.rc

@@ -0,0 +1,25 @@
+# These options are enabled when running on CI
+# We do this by copying this file to /etc/bazel.bazelrc at the start of the build.
+# See remote cache documentation in /docs/BAZEL.md
+
+# Don't be spammy in the logs
+build --noshow_progress
+
+# Don't run manual tests
+test --test_tag_filters=-manual
+
+# Enable experimental CircleCI bazel remote cache proxy
+# See remote cache documentation in /docs/BAZEL.md
+build --experimental_remote_spawn_cache --remote_rest_cache=http://localhost:7643
+
+# Prevent unstable environment variables from tainting cache keys
+build --experimental_strict_action_env
+
+# Workaround https://github.com/bazelbuild/bazel/issues/3645
+# Bazel doesn't calculate the memory ceiling correctly when running under Docker.
+# Limit Bazel to consuming resources that fit in CircleCI "medium" class which is the default:
+# https://circleci.com/docs/2.0/configuration-reference/#resource_class
+build --local_resources=3072,2.0,1.0
+
+# Retry in the event of flakes, eg. https://circleci.com/gh/angular/angular/31309
+test --flaky_test_attempts=2

.circleci/config.yml

@@ -15,6 +15,13 @@
 var_1: &docker_image angular/ngcontainer:0.1.0
 var_2: &cache_key angular-{{ .Branch }}-{{ checksum "yarn.lock" }}-0.1.0
 
+# See remote cache documentation in /docs/BAZEL.md
+var_3: &setup-bazel-remote-cache
+  run:
+    name: Start up bazel remote cache proxy
+    command: ~/bazel-remote-proxy -backend circleci://
+    background: true
+
 # Settings common to each job
 anchor_1: &job_defaults
   working_directory: ~/ng
@@ -34,14 +41,16 @@ jobs:
     steps:
       - checkout:
           <<: *post_checkout
-      # Check BUILD.bazel formatting before we have a node_modules directory
-      # Then we don't need any exclude pattern to avoid checking those files
-      - run: 'buildifier -mode=check $(find . -type f \( -name BUILD.bazel -or -name BUILD \)) ||
-              (echo "BUILD files not formatted. Please run ''yarn buildifier''" ; exit 1)'
-      # Run the skylark linter to check our Bazel rules
-      - run: 'find . -type f -name "*.bzl" |
-              xargs java -jar /usr/local/bin/Skylint_deploy.jar ||
+      # See remote cache documentation in /docs/BAZEL.md
+      - run: .circleci/setup_cache.sh
+      - run: sudo cp .circleci/bazel.rc /etc/bazel.bazelrc
+      - *setup-bazel-remote-cache
+
+      - run: 'yarn buildifier -mode=check ||
+              (echo -e "\nBUILD files not formatted. Please run ''yarn buildifier''" ; exit 1)'
+      - run: 'yarn skylint ||
               (echo -e "\n.bzl files have lint errors. Please run ''yarn skylint''"; exit 1)'
+
       - restore_cache:
           key: *cache_key
 
@@ -54,6 +63,11 @@ jobs:
     steps:
       - checkout:
           <<: *post_checkout
+      # See remote cache documentation in /docs/BAZEL.md
+      - run: .circleci/setup_cache.sh
+      - run: sudo cp .circleci/bazel.rc /etc/bazel.bazelrc
+      - *setup-bazel-remote-cache
+
       - restore_cache:
           key: *cache_key
 
@@ -62,7 +76,7 @@ jobs:
       # Use bazel query so that we explicitly ask for all buildable targets to be built as well
       # This avoids waiting for a build command to finish before running the first test
       # See https://github.com/bazelbuild/bazel/issues/4257
-      - run: bazel query --output=label '//modules/... union //packages/... union //tools/...' | xargs bazel test --config=ci
+      - run: bazel query --output=label '//modules/... union //packages/... union //tools/...' | xargs bazel test
 
       # CircleCI will allow us to go back and view/download these artifacts from past builds.
       # Also we can use a service like https://buildsize.org/ to automatically track binary size of these artifacts.

.circleci/setup_cache.sh

@@ -0,0 +1,11 @@
+#!/bin/sh
+# Install bazel remote cache proxy
+# This is temporary until the feature is no longer experimental on CircleCI.
+# See remote cache documentation in /docs/BAZEL.md
+
+set -u -e
+
+readonly DOWNLOAD_URL="https://5-116431813-gh.circle-artifacts.com/0/pkg/bazel-remote-proxy-$(uname -s)_$(uname -m)"
+
+curl --fail -o ~/bazel-remote-proxy "$DOWNLOAD_URL"
+chmod +x ~/bazel-remote-proxy

docs/BAZEL.md

@@ -112,6 +112,30 @@ Apple+Shift+D on Mac) and click on the green play icon next to the configuration
 - Open chrome at: [http://localhost:9876/debug.html](http://localhost:9876/debug.html)
 - Open chrome inspector
 
+## Remote cache
+
+Bazel supports fetching action results from a cache, allowing a clean build to pick up artifacts from prior builds.
+This makes builds incremental, even on CI.
+It works because Bazel assigns a content-based hash to all action inputs, which is used as the cache key for the action outputs.
+Thanks the the hermeticity property, we can skip executing an action if the inputs hash is already present in the cache.
+
+Of course, non-hermeticity in an action can cause problems.
+At worst, you can fetch a broken artifact from the cache, making your build non-reproducible.
+For this reason, we are careful to implement our Bazel rules to depend only on their inputs.
+
+Currently we only use remote caching on CircleCI.
+We could enable it for developer builds as well, which would make initial builds much faster for developers by fetching already-built artifacts from the cache.
+
+This feature is experimental, and developed by the CircleCI team with guidance from Angular.
+Contact Alex Eagle with questions.
+
+*How it's configured*:
+
+1. In `.circleci/config.yml`, each CircleCI job downloads a proxy binary, which is built from https://github.com/notnoopci/bazel-remote-proxy. The download is done by running `.circleci/setup_cache.sh`. When the feature graduates from experimental, this proxy will be installed by default on every CircleCI worker, and this step will not be needed.
+1. Next, each job runs the `setup-bazel-remote-cache` anchor. This starts up the proxy running in the background. In the CircleCI UI, you'll see this step continues running while later steps run, and you can see logging from the proxy process.
+1. Bazel must be configured to connect to the proxy on a local port. This configuration lives in `.circleci/bazel.rc` and is enabled because we overwrite the system Bazel settings in /etc/bazel.bazelrc with this file.
+1. Each `bazel` command in `.circleci/config.yml` picks up and uses the caching flags.
+
 ## Known issues
 
 ### Xcode

tools/bazel.rc

@@ -41,11 +41,8 @@ test --test_output=errors
 build --experimental_ui
 test --experimental_ui
 
-# Don't be spammy in the continuous integration logs
-build:ci --noshow_progress
+################################
+# Settings for CircleCI        #
+################################
 
-# Don't run manual tests on CI
-test:ci --test_tag_filters=-manual
-
-# Retry in the event of flakes, eg. https://circleci.com/gh/angular/angular/31309
-test:ci --flaky_test_attempts=2
+# Bazel flags for CircleCI are in /.circleci/bazel.rc