Add markdown docs about RBE

docs/rbe/rbe.md

@@ -0,0 +1,161 @@
+# RBE for Flutter Engine Developers
+
+## Overview
+
+This is documentation on setting up RBE for building the Flutter engine. It is
+expected to work for Googlers only on corp Linux, macOS, and Windows hosts,
+including cloudtop instances.
+
+## Getting started
+
+The first step is to add an entry to the `.gclient` file. The entry to add is
+`"use_rbe": True` in the `custom_vars` section. It should look like this:
+
+After making this edit, you must be authenticated as a Googler by CIPD so that
+the RBE configurations can be downloaded from the `flutter_internal`
+[CIPD bucket](https://chrome-infra-packages.appspot.com/p/flutter_internal):
+
+After authentication successfully, run `gclient sync -D`.
+
+## Running an RBE build
+
+In the engine repo, all RBE builds must be initiated through the `et` tool whose
+entrypoint is the script `//flutter/bin/et`. This is so that the local RBE
+proxy is correctly initialized and shut down around invocations of `ninja`.
+
+### gcloud
+
+Before running an RBE build, you must be authenticated with the Google cloud
+project that owns the RBE worker pool. A gcloud SDK is checked out with the
+engine repo under e.g. `//flutter/buildtools/mac-arm64/gcloud`.
+(Replace `mac-arm64` with the directory name that is correct for your host
+platform.)
+
+The `gcloud` tool in this SDK must be on your path. The tool lives under
+`//flutter/buildtools/mac-arm64/gcloud/bin`, which is the path to add to your
+`PATH` environment variable. Alternatively, you can get the gcloud SDK on your
+path by installing it on your system by following the instructions at
+[https://cloud.google.com/sdk/docs/install]
+(https://cloud.google.com/sdk/docs/install).
+
+On macOS, before running the `gcloud` command ensure that `python3` is on your
+path, and does not come from e.g. homebrew. The command `which python3` should
+return `/usr/bin/python3`.
+
+### Listing builds
+
+The builds available to the `et` tool are those specified by the build
+configuration json files under `//flutter/ci/builders`. A list of builds
+suitable for local development can also be printed by running `et help build`.
+The list of all builds can be printed by running `et help build --verbose`.
+Builds in the verbose list prefixed with `ci/` or `ci\` are exactly the builds
+run on CI with the same GN flags and Ninja targets.
+
+### Running builds
+
+To run a build, pass the name of the configuration to the `et build` command:
+
+If RBE is working correctly, you should see logs like the following:
+
+To disable RBE in a build, pass the flag `--no-rbe` to the `build` command.
+
+Since LTO is slow and rarely useful in local builds, `et` disables it by default
+in all builds, even when it is specified by a build configuration. To enable
+it, pass the `--lto` flag to the `build` command.
+
+### Customizing builds
+
+Beyond enabling/disabling RBE and LTO, the `build` command does not currently
+support customizing builds at the command line.
+
+If you need custom GN flags or Ninja targets, then this can be achieved by
+making local edits to the build configuration json files in your checkout. In
+particular, the file `//flutter/ci/builders/local_engine.json` is intended to
+contain builds that are commonly used for local engine development.
+
+If a configuration does not exist there, and will be needed by multiple people
+over a long period of time, consider checking in the new configuration so that
+it will be built on CI, which will keep the RBE cache warm for fast local
+builds.
+
+On the other hand, if a build is intended to produce artifacts or run tests on
+CI, then it should _not_ go in `local_engine.json`. Instead, it should go in
+the json file appropriate for its purpose. There are many examples available,
+so following suit is likely your best bet, but if you are unsure, ask in the
+`hackers-engine` Discord.
+
+## Maintaining Compatibility with RBE
+
+RBE is sensitive to what paths in compile commands look like. In particular, all
+paths in compile commands must be relative paths, and those relative paths must
+all resolve to files within the engine source tree. In practice, this means
+that the GN function `rebase_path()` should only lack a second parameter when
+its result won’t be used in a compile or link command. There are unfortunately
+many examples in the tree where `rebase_path()` lacks a second parameter.
+However to be on the safe side a good rule of thumb is: do not add new usages
+of the single parameter `rebase_path()`.
+
+## Troubleshooting
+
+### Slow builds
+
+RBE builds can be slow for a few different reasons. The most common reason is
+likely to be that the remote caches are cold. When the caches are warm, a
+compile step consists only of downloading the compiled TU from the cache. When
+the caches are cold, the remote workers must run the compilation commands,
+which takes more time.
+
+RBE builds can also be slow if your network connection is bandwidth constrained.
+Anecdotally, even with a warm cache, I have noticed slow builds from home due
+to RBE saturating my low-tier Comcast Business connection.
+
+When RBE builds are slow, non-RBE builds may be faster, especially incremental
+builds.
+
+### Proxy status and debug logs
+
+The status of the local RBE proxy can be queried with the following command
+
+It will give output describing the number of actions completed and in progress,
+and the number of remote executions, local executions, and remote cache hits.
+
+The logs for RBE live under the system `/tmp` folder in the files `/tmp/reproxy.
+{INFO,WARNING,ERROR}` and `/tmp/bootstrap.{INFO,WARNING,ERROR}`.
+
+In CI runs, the RBE logs are also available by following the links as in the
+screenshot below under `teardown remote execution > collect rbe logs`:
+
+![LUCI logs links](luci.png "LUCI logs links")
+
+### Dependency analysis failures
+
+These logs can be useful to highlight failures in RBE’s dependency analysis. The
+dependency analysis can fail for a variety of reasons, but a common one during
+development is likely to be that the source file is really malformed somehow.
+This can be debugged by doing a local build with RBE turned off.
+
+
+## References
+
+* Code for RBE (i.e. reproxy, rewrapper, bootstrap, etc.) lives in [this GitHub
+  repository](https://github.com/bazelbuild/reclient). The tools are not
+  well-documented, so the source code is the source of truth for the command
+  line flags that they accept, for example.
+* Internal-facing RBE migration guide is [here](go/reclient-migration-guide).
+  (Mostly focused on Chrome and Android, so not all parts are relevant to
+  Flutter.)
+* The version of RBE for local development is set in the DEPS file [here]
+  (https://github.com/flutter/engine/blob/8578edf9c9393471ca9eab18e9154f0e6066dcb6/DEPS#L53).
+  It needs to be manually rolled occasionally.
+* The version of RBE used by CI is set in a LUCI recipe [here]
+  (https://flutter.googlesource.com/recipes/+/be12675150183af68223f5fbc6e0f888a1139e79/recipe_modules/rbe/api.py#16).
+  It also needs to be manually rolled occasionally.
+* Googler-only RBE configuration files live in the CIPD bucket [here]
+  (https://chrome-infra-packages.appspot.com/p/flutter_internal/rbe/reclient_cfgs).
+  They need to be updated when we roll clang to a new version as described
+  [here]
+  (https://github.com/flutter/engine/pull/52062#issuecomment-2050902282).
+* Flutter’s RBE worker pool is defined [here]
+  (https://source.corp.google.com/piper///depot/google3/configs/cloud/gong/services/flutter_rbe/modules/infra/prod/main.tf).
+* Using RBE for Engine clang-tidy is blocked on [b/326591374]
+  (http://b/326591374).