Reorganize and expand the testing chapters. (#1281)

* Reorganize and expand the testing chapters.

* Update tests chapters for review comments.

* Fix typo.

Author: ehuss

book.toml

@@ -24,3 +24,6 @@ follow-web-links = true
 exclude = [ "crates\\.io", "gcc\\.godbolt\\.org", "youtube\\.com", "youtu\\.be", "dl\\.acm\\.org", "cs\\.bgu\\.ac\\.il", "www\\.amazon\\.com", "www\\.rustaceans\\.org", "play\\.rust-lang\\.org" ]
 cache-timeout = 86400
 warning-policy = "error"
+
+[output.html.redirect]
+"/compiletest.html" = "tests/compiletest.html"

src/SUMMARY.md

@@ -15,10 +15,16 @@
     - [Documenting Compiler](./building/compiler-documenting.md)
     - [Rustdoc overview](./rustdoc.md)
     - [Adding a new target](./building/new-target.md)
-- [The compiler testing framework](./tests/intro.md)
+- [Testing the compiler](./tests/intro.md)
     - [Running tests](./tests/running.md)
+        - [Testing with Docker](./tests/docker.md)
+        - [Testing with CI](./tests/ci.md)
     - [Adding new tests](./tests/adding.md)
-    - [Using `compiletest` commands to control test execution](./compiletest.md)
+    - [Compiletest](./tests/compiletest.md)
+        - [UI tests](./tests/ui.md)
+        - [Test headers](./tests/headers.md)
+    - [Performance testing](./tests/perf.md)
+    - [Crater](./tests/crater.md)
 - [Debugging the Compiler](./compiler-debugging.md)
     - [Using the tracing/logging instrumentation](./tracing.md)
 - [Profiling the compiler](./profiling.md)

src/compiletest.md

@@ -4,11 +4,8 @@ This section talks about how to profile the compiler and find out where it spend
 
 Depending on what you're trying to measure, there are several different approaches:
 
-- If you want to see if a PR improves or regresses compiler performance:
-  - The [rustc-perf](https://github.com/rust-lang/rustc-perf) project makes this easy and can be triggered to run on a PR via the `@rust-timer` bot.
-    The `@bors try @rust-timer queue` command, in a comment on the PR, will queue a try build and a
-    benchmarking run.
-    Note: you need `try` privileges to be able to do this. More details are available in the [perf collector documentation](https://github.com/rust-lang/rustc-perf/blob/master/collector/README.md).
+- If you want to see if a PR improves or regresses compiler performance,
+  see the [rustc-perf chapter](tests/perf.md) for requesting a benchmarking run.
 
 - If you want a medium-to-high level overview of where `rustc` is spending its time:
   - The `-Z self-profile` flag and [measureme](https://github.com/rust-lang/measureme) tools offer a query-based approach to profiling.

src/profiling.md

@@ -0,0 +1,73 @@
+# Testing with CI
+
+## Testing infrastructure
+
+When a Pull Request is opened on GitHub, [GitHub Actions] will automatically
+launch a build that will run all tests on some configurations
+(x86_64-gnu-llvm-12 linux. x86_64-gnu-tools linux, mingw-check linux).
+In essence, each runs `./x.py test` with various different options.
+
+The integration bot [bors] is used for coordinating merges to the master branch.
+When a PR is approved, it goes into a [queue] where merges are tested one at a
+time on a wide set of platforms using GitHub Actions. Due to the limit on the
+number of parallel jobs, we run CI under the [rust-lang-ci] organization except
+for PRs.
+Most platforms only run the build steps, some run a restricted set of tests,
+only a subset run the full suite of tests (see Rust's [platform tiers]).
+
+If everything passes, then all of the distribution artifacts that were
+generated during the CI run are published.
+
+[GitHub Actions]: https://github.com/rust-lang/rust/actions
+[rust-lang-ci]: https://github.com/rust-lang-ci/rust/actions
+[bors]: https://github.com/servo/homu
+[queue]: https://bors.rust-lang.org/queue/rust
+[platform tiers]: https://forge.rust-lang.org/release/platform-support.html#rust-platform-support
+
+## Using CI to test
+
+In some cases, a PR may run into problems with running tests on a particular
+platform or configuration.
+If you can't run those tests locally, don't hesitate to use CI resources to
+try out a fix.
+
+As mentioned above, opening or updating a PR will only run on a small subset
+of configurations.
+Only when a PR is approved will it go through the full set of test configurations.
+However, you can try one of those configurations in your PR before it is approved.
+For example, if a Windows build fails, but you don't have access to a Windows
+machine, you can try running the Windows job that failed on CI within your PR
+after pushing a possible fix.
+
+To do this, you'll need to edit [`src/ci/github-actions/ci.yml`].
+The `jobs` section defines the jobs that will run.
+The `jobs.pr` section defines everything that will run in a push to a PR.
+The `jobs.auto` section defines the full set of tests that are run after a PR is approved.
+You can copy one of the definitions from the `auto` section up to the `pr` section.
+
+For example, the `x86_64-msvc-1` and `x86_64-msvc-2` jobs are responsible for
+running the 64-bit MSVC tests.
+You can copy those up to the `jobs.pr.strategy.matrix.include` section with
+the other jobs.
+
+The comment at the top of `ci.yml` will tell you to run this command:
+
+```sh
+./x.py run src/tools/expand-yaml-anchors
+````
+
+This will generate the true [`.github/workflows/ci.yml`] which is what GitHub
+Actions uses.
+
+Then, you can commit those two files and push to GitHub.
+GitHub Actions should launch the tests.
+
+After you have finished, don't forget to remove any changes you have made to `ci.yml`.
+
+Although you are welcome to use CI, just be conscientious that this is a shared
+resource with limited concurrency.
+Try not to enable too many jobs at once (one or two should be sufficient in
+most cases).
+
+[`src/ci/github-actions/ci.yml`]: https://github.com/rust-lang/rust/blob/master/src/ci/github-actions/ci.yml
+[`.github/workflows/ci.yml`]: https://github.com/rust-lang/rust/blob/master/.github/workflows/ci.yml#L1

src/tests/adding.md

@@ -0,0 +1,45 @@
+# Crater
+
+[Crater](https://github.com/rust-lang/crater) is a tool for compiling
+and running tests for _every_ crate on [crates.io](https://crates.io) (and a
+few on GitHub). It is mainly used for checking the extent of breakage when
+implementing potentially breaking changes and ensuring lack of breakage by
+running beta vs stable compiler versions.
+
+## When to run Crater
+
+You should request a crater run if your PR makes large changes to the compiler
+or could cause breakage. If you are unsure, feel free to ask your PR's reviewer.
+
+## Requesting Crater Runs
+
+The rust team maintains a few machines that can be used for running crater runs
+on the changes introduced by a PR. If your PR needs a crater run, leave a
+comment for the triage team in the PR thread. Please inform the team whether
+you require a "check-only" crater run, a "build only" crater run, or a
+"build-and-test" crater run. The difference is primarily in time; the
+conservative (if you're not sure) option is to go for the build-and-test run.
+If making changes that will only have an effect at compile-time (e.g.,
+implementing a new trait) then you only need a check run.
+
+Your PR will be enqueued by the triage team and the results will be posted when
+they are ready. Check runs will take around ~3-4 days, with the other two
+taking 5-6 days on average.
+
+While crater is really useful, it is also important to be aware of a few
+caveats:
+
+- Not all code is on crates.io! There is a lot of code in repos on GitHub and
+  elsewhere. Also, companies may not wish to publish their code. Thus, a
+  successful crater run is not a magically green light that there will be no
+  breakage; you still need to be careful.
+
+- Crater only runs Linux builds on x86_64. Thus, other architectures and
+  platforms are not tested. Critically, this includes Windows.
+
+- Many crates are not tested. This could be for a lot of reasons, including
+  that the crate doesn't compile any more (e.g. used old nightly features),
+  has broken or flaky tests, requires network access, or other reasons.
+
+- Before crater can be run, `@bors try` needs to succeed in building artifacts.
+  This means that if your code doesn't compile, you cannot run crater.

src/tests/ci.md

@@ -0,0 +1,52 @@
+# Testing with Docker
+
+The Rust tree includes [Docker] image definitions for the platforms used on
+GitHub Actions in [`src/ci/docker`].
+The script [`src/ci/docker/run.sh`] is used to build the Docker image, run it,
+build Rust within the image, and run the tests.
+
+You can run these images on your local development machine. This can be
+helpful to test environments different from your local system. First you will
+need to install Docker on a Linux, Windows, or macOS system (typically Linux
+will be much faster than Windows or macOS because the later use virtual
+machines to emulate a Linux environment). To enter interactive mode which will
+start a bash shell in the container, run `src/ci/docker/run.sh --dev <IMAGE>`
+where `<IMAGE>` is one of the directory names in `src/ci/docker` (for example
+`x86_64-gnu` is a fairly standard Ubuntu environment).
+
+The docker script will mount your local Rust source tree in read-only mode,
+and an `obj` directory in read-write mode. All of the compiler artifacts will
+be stored in the `obj` directory. The shell will start out in the `obj`
+directory. From there, you can run `../src/ci/run.sh` which will run the build
+as defined by the image.
+
+Alternatively, you can run individual commands to do specific tasks. For
+example, you can run `python3 ../x.py test src/test/ui` to just run UI tests.
+Note that there is some configuration in the [`src/ci/run.sh`] script that you
+may need to recreate. Particularly, set `submodules = false` in your
+`config.toml` so that it doesn't attempt to modify the read-only directory.
+
+Some additional notes about using the Docker images:
+
+- Some of the std tests require IPv6 support. Docker on Linux seems to have it
+  disabled by default. Run the commands in [`enable-docker-ipv6.sh`] to enable
+  IPv6 before creating the container. This only needs to be done once.
+- The container will be deleted automatically when you exit the shell, however
+  the build artifacts persist in the `obj` directory. If you are switching
+  between different Docker images, the artifacts from previous environments
+  stored in the `obj` directory may confuse the build system. Sometimes you
+  will need to delete parts or all of the `obj` directory before building
+  inside the container.
+- The container is bare-bones, with only a minimal set of packages. You may
+  want to install some things like `apt install less vim`.
+- You can open multiple shells in the container. First you need the container
+  name (a short hash), which is displayed in the shell prompt, or you can run
+  `docker container ls` outside of the container to list the available
+  containers. With the container name, run `docker exec -it <CONTAINER>
+  /bin/bash` where `<CONTAINER>` is the container name like `4ba195e95cef`.
+
+[Docker]: https://www.docker.com/
+[`src/ci/docker`]: https://github.com/rust-lang/rust/tree/master/src/ci/docker
+[`src/ci/docker/run.sh`]: https://github.com/rust-lang/rust/blob/master/src/ci/docker/run.sh
+[`src/ci/run.sh`]: https://github.com/rust-lang/rust/blob/master/src/ci/run.sh
+[`enable-docker-ipv6.sh`]: https://github.com/rust-lang/rust/blob/master/src/ci/scripts/enable-docker-ipv6.sh

src/tests/compiletest.md

@@ -0,0 +1,50 @@
+# Performance testing
+
+## rustc-perf
+
+A lot of work is put into improving the performance of the compiler and
+preventing performance regressions.
+The [rustc-perf](https://github.com/rust-lang/rustc-perf) project provides
+several services for testing and tracking performance.
+It provides hosted infrastructure for running benchmarks as a service.
+At this time, only `x86_64-unknown-linux-gnu` builds are tracked.
+
+A "perf run" is used to compare the performance of the compiler in different
+configurations for a large collection of popular crates.
+Different configurations include "fresh builds", builds with incremental compilation, etc.
+
+The result of a perf run is a comparison between two versions of the compiler
+(by their commit hashes).
+
+### Automatic perf runs
+
+After every PR is merged, a suite of benchmarks are run against the compiler.
+The results are tracked over time on the <https://perf.rust-lang.org/> website.
+Any changes are noted in a comment on the PR.
+
+### Manual perf runs
+
+Additionally, performance tests can be ran before a PR is merged on an as-needed basis.
+You should request a perf run if your PR may affect performance, especially if
+it can affect performance adversely.
+
+To evaluate the performance impact of a PR, write this comment on the PR:
+
+`@bors try @rust-timer queue`
+
+> **Note**: Only users authorized to do perf runs are allowed to post this comment.
+> Teams that are allowed to use it are tracked in the [Teams
+> repository](https://github.com/rust-lang/team) with the `perf = true` value
+> in the `[permissions]` section (and bors permissions are also required).
+> If you are not on one of those teams, feel free to ask for someone to post
+> it for you (either on Zulip or ask the assigned reviewer).
+
+This will first tell bors to do a "try" build which do a full release build
+for `x86_64-unknown-linux-gnu`.
+After the build finishes, it will place it in the queue to run the performance
+suite against it.
+After the performance tests finish, the bot will post a comment on the PR with
+a summary and a link to a full report.
+
+More details are available in the [perf collector
+documentation](https://github.com/rust-lang/rustc-perf/blob/master/collector/README.md).

src/tests/crater.md

@@ -18,8 +18,9 @@ I think is done, but rarely otherwise. -nmatsakis)
 The test results are cached and previously successful tests are
 `ignored` during testing. The stdout/stderr contents as well as a
 timestamp file for every test can be found under `build/ARCH/test/`.
-To force-rerun a test (e.g. in case the test runner fails to notice
-a change) you can simply remove the timestamp file.
+To force-rerun a test (e.g. in case the test runner fails to notice a change)
+you can simply remove the timestamp file, or use the `--force-rerun` CLI
+option.
 
 Note that some tests require a Python-enabled gdb. You can test if
 your gdb install supports Python by using the `python` command from
@@ -143,6 +144,18 @@ to automatically adjust the `.stderr`, `.stdout` or `.fixed` files of
 all tests. Of course you can also target just specific tests with the
 `--test-args your_test_name` flag, just like when running the tests.
 
+## Configuring test running
+
+There are a few options for running tests:
+
+* `config.toml` has the `rust.verbose-tests` option.
+  If `false`, each test will print a single dot (the default).
+  If `true`, the name of every test will be printed.
+  This is equivalent to the `--quiet` option in the [Rust test
+  harness](https://doc.rust-lang.org/rustc/tests/)
+* The environment variable `RUST_TEST_THREADS` can be set to the number of
+  concurrent threads to use for testing.
+
 ## Passing `--pass $mode`
 
 Pass UI tests now have three modes, `check-pass`, `build-pass` and
@@ -156,9 +169,8 @@ exists in the test file. For example, you can run all the tests in
 ```
 
 By passing `--pass $mode`, you can reduce the testing time. For each
-mode, please see [here][mode].
-
-[mode]: ./adding.md#tests-that-do-not-result-in-compile-errors
+mode, please see [Controlling pass/fail
+expectations](ui.md#controlling-passfail-expectations).
 
 ## Using incremental compilation
 
@@ -193,17 +205,7 @@ To run the UI test suite in NLL mode, one would use the following:
 ./x.py test src/test/ui --compare-mode=nll
 ```
 
-The possible compare modes are:
-
-* nll - currently nll is implemented in migrate mode, this option runs with true nll.
-* polonius
-* chalk
-* split-dwarf
-* split-dwarf-single
-
-Note that compare modes are separate to [revisions](./adding.html#revisions).
-All revisions are tested when running `./x.py test src/test/ui`,
-however compare-modes must be manually run individually via the `--compare-mode` flag.
+See [Compare modes](compiletest.md#compare-modes) for more details.
 
 ## Running tests manually
 
@@ -219,3 +221,105 @@ rustc +stage1 src/test/ui/issue-1234.rs
 This is much faster, but doesn't always work. For example, some tests
 include directives that specify specific compiler flags, or which rely
 on other crates, and they may not run the same without those options.
+
+
+## Running tests on a remote machine
+
+Tests may be run on a remote machine (e.g. to test builds for a different
+architecture). This is done using `remote-test-client` on the build machine
+to send test programs to `remote-test-server` running on the remote machine.
+`remote-test-server` executes the test programs and sends the results back to
+the build machine. `remote-test-server` provides *unauthenticated remote code
+execution* so be careful where it is used.
+
+To do this, first build `remote-test-server` for the remote
+machine, e.g. for RISC-V
+```sh
+./x.py build src/tools/remote-test-server --target riscv64gc-unknown-linux-gnu
+```
+
+The binary will be created at
+`./build/$HOST_ARCH/stage2-tools/$TARGET_ARCH/release/remote-test-server`. Copy
+this over to the remote machine.
+
+On the remote machine, run the `remote-test-server` with the `remote` argument
+(and optionally `-v` for verbose output). Output should look like this:
+```sh
+$ ./remote-test-server -v remote
+starting test server
+listening on 0.0.0.0:12345!
+```
+
+You can test if the `remote-test-server` is working by connecting to it and
+sending `ping\n`. It should reply `pong`:
+```sh
+$ nc $REMOTE_IP 12345
+ping
+pong
+```
+
+To run tests using the remote runner, set the `TEST_DEVICE_ADDR` environment
+variable then use `x.py` as usual. For example, to run `ui` tests for a RISC-V
+machine with the IP address `1.2.3.4` use
+```sh
+export TEST_DEVICE_ADDR="1.2.3.4:12345"
+./x.py test src/test/ui --target riscv64gc-unknown-linux-gnu
+```
+
+If `remote-test-server` was run with the verbose flag, output on the test machine
+may look something like
+```
+[...]
+run "/tmp/work/test1007/a"
+run "/tmp/work/test1008/a"
+run "/tmp/work/test1009/a"
+run "/tmp/work/test1010/a"
+run "/tmp/work/test1011/a"
+run "/tmp/work/test1012/a"
+run "/tmp/work/test1013/a"
+run "/tmp/work/test1014/a"
+run "/tmp/work/test1015/a"
+run "/tmp/work/test1016/a"
+run "/tmp/work/test1017/a"
+run "/tmp/work/test1018/a"
+[...]
+```
+
+Tests are built on the machine running `x.py` not on the remote machine. Tests
+which fail to build unexpectedly (or `ui` tests producing incorrect build
+output) may fail without ever running on the remote machine.
+
+## Testing on emulators
+
+Some platforms are tested via an emulator for architectures that aren't
+readily available. For architectures where the standard library is well
+supported and the host operating system supports TCP/IP networking, see the
+above instructions for testing on a remote machine (in this case the
+remote machine is emulated).
+
+There is also a set of tools for orchestrating running the
+tests within the emulator. Platforms such as `arm-android` and
+`arm-unknown-linux-gnueabihf` are set up to automatically run the tests under
+emulation on GitHub Actions. The following will take a look at how a target's tests
+are run under emulation.
+
+The Docker image for [armhf-gnu] includes [QEMU] to emulate the ARM CPU
+architecture. Included in the Rust tree are the tools [remote-test-client]
+and [remote-test-server] which are programs for sending test programs and
+libraries to the emulator, and running the tests within the emulator, and
+reading the results.  The Docker image is set up to launch
+`remote-test-server` and the build tools use `remote-test-client` to
+communicate with the server to coordinate running tests (see
+[src/bootstrap/test.rs]).
+
+> TODO:
+> Is there any support for using an iOS emulator?
+>
+> It's also unclear to me how the wasm or asm.js tests are run.
+
+[armhf-gnu]: https://github.com/rust-lang/rust/tree/master/src/ci/docker/host-x86_64/armhf-gnu/Dockerfile
+[QEMU]: https://www.qemu.org/
+[remote-test-client]: https://github.com/rust-lang/rust/tree/master/src/tools/remote-test-client
+[remote-test-server]: https://github.com/rust-lang/rust/tree/master/src/tools/remote-test-server
+[src/bootstrap/test.rs]: https://github.com/rust-lang/rust/tree/master/src/bootstrap/test.rs
+