From d847468768446168b596f721844193afaaf9d3f2 Mon Sep 17 00:00:00 2001 From: Ed Page Date: Tue, 14 Jun 2022 17:03:50 -0500 Subject: [PATCH 1/2] docs(contrib): Add documentation for ui tests This only adds information about snapshot testing using `snapbox` and keeps the functional testing documentation focused on the existing facilities. We can updated this as our use of `snapbox` matures. In writing this, I did notice that we define `cargo_test_support::compare::assert` but only use it for filesystem asserts and not binary asserts. We should probably add our own function that wraps `snapbox::cmd::Command::cargo()` and passes in `cargo_test_support::compare::assert`. I've left that out of this PR to keep things focused. --- src/doc/contrib/src/tests/writing.md | 218 ++++++++++++++++++++------- 1 file changed, 160 insertions(+), 58 deletions(-) diff --git a/src/doc/contrib/src/tests/writing.md b/src/doc/contrib/src/tests/writing.md index 3d9e1b26753..5d0c7e0f406 100644 --- a/src/doc/contrib/src/tests/writing.md +++ b/src/doc/contrib/src/tests/writing.md @@ -6,65 +6,69 @@ tests is also encouraged! ## Testsuite Cargo has a wide variety of integration tests that execute the `cargo` binary -and verify its behavior, located in the [`testsuite`] directory. The -[`support`] crate contains many helpers to make this process easy. +and verify its behavior, located in the [`testsuite`] directory. The +[`support`] crate and [`snapbox`] contain many helpers to make this process easy. + +There are two styles of tests that can roughly be categorized as +- functional tests + - The fixture is programmatically defined + - The assertions are regular string comparisons + - Easier to share in an issue as a code block is completely self-contained + - More resilient to insignificant changes though ui tests are easy to update when a change does occur +- ui tests + - The fixture is file-based + - The assertions use file-backed snapshots that can be updated with an env variable + - Easier to review the expected behavior of the command as more details are included + - Easier to get up and running from an existing project + - Easier to reason about as everything is just files in the repo These tests typically work by creating a temporary "project" with a `Cargo.toml` file, executing the `cargo` binary process, and checking the stdout and stderr output against the expected output. -### `cargo_test` attribute - -Cargo's tests use the `#[cargo_test]` attribute instead of `#[test]`. This -attribute injects some code which does some setup before starting the test, -creating the little "sandbox" described below. - -### Basic test structure - -The general form of a test involves creating a "project", running `cargo`, and -checking the result. Projects are created with the [`ProjectBuilder`] where -you specify some files to create. The general form looks like this: - -```rust,ignore -let p = project() - .file("src/main.rs", r#"fn main() { println!("hi!"); }"#) - .build(); -``` - -The project creates a mini sandbox under the "cargo integration test" -directory with each test getting a separate directory such as -`/path/to/cargo/target/cit/t123/`. Each project appears as a separate -directory. There is also an empty `home` directory created that will be used -as a home directory instead of your normal home directory. - -If you do not specify a `Cargo.toml` manifest using `file()`, one is -automatically created with a project name of `foo` using `basic_manifest()`. - -To run Cargo, call the `cargo` method and make assertions on the execution: +### Functional Tests +Generally, a functional test will be placed in `tests/testsuite/.rs` and will look roughly like: ```rust,ignore -p.cargo("run --bin foo") - .with_stderr( - "\ -[COMPILING] foo [..] -[FINISHED] [..] -[RUNNING] `target/debug/foo` -", - ) - .with_stdout("hi!") - .run(); +#[cargo_test] +fn () { + let p = project() + .file("src/main.rs", r#"fn main() { println!("hi!"); }"#) + .build(); + + p.cargo("run --bin foo") + .with_stderr( + "\ + [COMPILING] foo [..] + [FINISHED] [..] + [RUNNING] `target/debug/foo` + ", + ) + .with_stdout("hi!") + .run(); + } +} ``` -This uses the [`Execs`] struct to build up a command to execute, along with -the expected output. +`#[cargo_test]`: +- This is used in place of `#[test]` +- This attribute injects code which does some setup before starting the + test, creating a filesystem "sandbox" under the "cargo integration test" + directory for each test such as + `/path/to/cargo/target/cit/t123/` +- The sandbox will contain a `home` directory that will be used instead of your normal home directory -See [`support::compare`] for an explanation of the string pattern matching. -Patterns are used to make it easier to match against the expected output. +[`ProjectBuilder`] via `project()`: +- Each project is in a separate directory in the sandbox +- If you do not specify a `Cargo.toml` manifest using `file()`, one is + automatically created with a project name of `foo` using `basic_manifest()`. -Browse the `pub` functions and modules in the [`support`] crate for a variety -of other helpful utilities. +[`Execs`] via `p.cargo(...)`: +- This executes the command and evaluates different assertions + - See [`support::compare`] for an explanation of the string pattern matching. + Patterns are used to make it easier to match against the expected output. -### Testing Nightly Features +#### Testing Nightly Features If you are testing a Cargo feature that only works on "nightly" Cargo, then you need to call `masquerade_as_nightly_cargo` on the process builder like @@ -85,17 +89,7 @@ if !is_nightly() { } ``` -### Platform-specific Notes - -When checking output, use `/` for paths even on Windows: the actual output -of `\` on Windows will be replaced with `/`. - -Be careful when executing binaries on Windows. You should not rename, delete, -or overwrite a binary immediately after running it. Under some conditions -Windows will fail with errors like "directory not empty" or "failed to remove" -or "access is denied". - -### Specifying Dependencies +#### Specifying Dependencies You should not write any tests that use the network such as contacting crates.io. Typically, simple path dependencies are the easiest way to add a @@ -123,6 +117,110 @@ If you need to test with registry dependencies, see If you need to test git dependencies, see [`support::git`] to create a git dependency. +### UI Tests + +UI Tests are a bit more spread out and generally look like: + +`tests/testsuite//mod.rs`: +```rust,ignore +mod ; +``` + +`tests/testsuite///mod.rs`: +```rust,ignore +use cargo_test_support::prelude::*; +use cargo_test_support::compare::assert; +use cargo_test_support::Project; +use cargo_test_support::curr_dir; + +#[cargo_test] +fn () { + let project = Project::from_template(curr_dir!().join("in")); + let project_root = project.root(); + let cwd = &project_root; + + snapbox::cmd::Command::cargo() + .arg("run") + .arg_line("--bin foo") + .current_dir(cwd) + .assert() + .success() + .stdout_matches_path(curr_dir!().join("stdout.log")) + .stderr_matches_path(curr_dir!().join("stderr.log")); + + assert().subset_matches(curr_dir!().join("out"), &project_root); +} +``` + +Then populate +- `tests/testsuite///in` with the project's directory structure +- `tests/testsuite///out` with the files you want verified +- `tests/testsuite///stdout.log` with nothing +- `tests/testsuite///stderr.log` with nothing + +`#[cargo_test]`: +- This is used in place of `#[test]` +- This attribute injects code which does some setup before starting the + test, creating a filesystem "sandbox" under the "cargo integration test" + directory for each test such as + `/path/to/cargo/target/cit/t123/` +- The sandbox will contain a `home` directory that will be used instead of your normal home directory + +`Project`: +- The project is copied from a directory in the repo +- Each project is in a separate directory in the sandbox + +[`Command`] via `Command::cargo()`: +- Set up and run a command. + +[`OutputAssert`] via `Command::assert()`: +- Perform assertions on the result of the [`Command`] + +[`Assert`] via `assert()`: +- Verify the command modified the file system as expected + +#### Updating Snapshots + +The project, stdout, and stderr snapshots can be updated by running with the +`SNAPSHOTS=overwrite` environment variable, like: +```console +$ SNAPSHOTS=overwrite cargo test +``` + +Be sure to check the snapshots to make sure they make sense. + +#### Testing Nightly Features + +If you are testing a Cargo feature that only works on "nightly" Cargo, then +you need to call `masquerade_as_nightly_cargo` on the process builder like +this: + +```rust,ignore + snapbox::cmd::Command::cargo() + .masquerade_as_nightly_cargo() +``` + +If you are testing a feature that only works on *nightly rustc* (such as +benchmarks), then you should exit the test if it is not running with nightly +rust, like this: + +```rust,ignore +if !is_nightly() { + // Add a comment here explaining why this is necessary. + return; +} +``` + +### Platform-specific Notes + +When checking output, use `/` for paths even on Windows: the actual output +of `\` on Windows will be replaced with `/`. + +Be careful when executing binaries on Windows. You should not rename, delete, +or overwrite a binary immediately after running it. Under some conditions +Windows will fail with errors like "directory not empty" or "failed to remove" +or "access is denied". + ## Debugging tests In some cases, you may need to dig into a test that is not working as you @@ -159,3 +257,7 @@ environment. The general process is: [`support::registry::Package`]: https://github.com/rust-lang/cargo/blob/e4b65bdc80f2a293447f2f6a808fa7c84bf9a357/crates/cargo-test-support/src/registry.rs#L73-L149 [`support::git`]: https://github.com/rust-lang/cargo/blob/master/crates/cargo-test-support/src/git.rs [Running Cargo]: ../process/working-on-cargo.md#running-cargo +[`snapbox`]: https://docs.rs/snapbox/latest/snapbox/ +[`Command`]: https://docs.rs/snapbox/latest/snapbox/cmd/struct.Command.html +[`OutputAssert`]: https://docs.rs/snapbox/latest/snapbox/cmd/struct.OutputAssert.html +[`Assert`]: https://docs.rs/snapbox/latest/snapbox/struct.Assert.html From 619630c3ecb5bbe90d342ae1bf68a9ba533c68c1 Mon Sep 17 00:00:00 2001 From: Ed Page Date: Mon, 20 Jun 2022 09:35:54 -0500 Subject: [PATCH 2/2] docs(contrib): Update API item links --- src/doc/contrib/src/tests/writing.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/src/doc/contrib/src/tests/writing.md b/src/doc/contrib/src/tests/writing.md index 5d0c7e0f406..156ce2b594f 100644 --- a/src/doc/contrib/src/tests/writing.md +++ b/src/doc/contrib/src/tests/writing.md @@ -250,11 +250,11 @@ environment. The general process is: 3. Run with arguments: `r check` [`testsuite`]: https://github.com/rust-lang/cargo/tree/master/tests/testsuite/ -[`ProjectBuilder`]: https://github.com/rust-lang/cargo/blob/e4b65bdc80f2a293447f2f6a808fa7c84bf9a357/crates/cargo-test-support/src/lib.rs#L225-L231 -[`Execs`]: https://github.com/rust-lang/cargo/blob/e4b65bdc80f2a293447f2f6a808fa7c84bf9a357/crates/cargo-test-support/src/lib.rs#L558-L579 +[`ProjectBuilder`]: https://github.com/rust-lang/cargo/blob/d847468768446168b596f721844193afaaf9d3f2/crates/cargo-test-support/src/lib.rs#L196-L202 +[`Execs`]: https://github.com/rust-lang/cargo/blob/d847468768446168b596f721844193afaaf9d3f2/crates/cargo-test-support/src/lib.rs#L531-L550 [`support`]: https://github.com/rust-lang/cargo/blob/master/crates/cargo-test-support/src/lib.rs [`support::compare`]: https://github.com/rust-lang/cargo/blob/master/crates/cargo-test-support/src/compare.rs -[`support::registry::Package`]: https://github.com/rust-lang/cargo/blob/e4b65bdc80f2a293447f2f6a808fa7c84bf9a357/crates/cargo-test-support/src/registry.rs#L73-L149 +[`support::registry::Package`]: https://github.com/rust-lang/cargo/blob/d847468768446168b596f721844193afaaf9d3f2/crates/cargo-test-support/src/registry.rs#L311-L389 [`support::git`]: https://github.com/rust-lang/cargo/blob/master/crates/cargo-test-support/src/git.rs [Running Cargo]: ../process/working-on-cargo.md#running-cargo [`snapbox`]: https://docs.rs/snapbox/latest/snapbox/