cargo test --all-targets also runs binaries marked as test=false

[Ekleog]

Problem
Running cargo test --all-targets apparently runs binaries even when marked test=false. This is a problem in particular with cargo fuzz targets, that do some linker dark magic and start trying to fuzz during the test session, failing as it wasn't built for it.

Steps
First, provision the following files:

Cargo.toml:

[package]
name = "foobar"
version = "0.1.0"
authors = ["Léo Gaspard <leo@gaspard.io>"]
edition = "2018"

# See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html

[dependencies]
libfuzzer-sys = "0.3.2"

[[bin]]
name = "example-bin"
path = "example-bin/main.rs"
test = false

src/lib.rs:

#[cfg(test)]
mod tests {
    #[test]
    fn it_works() {
        assert_eq!(2 + 2, 4);
    }
}

example-bin/main.rs:

#![no_main]

use libfuzzer_sys::fuzz_target;

fuzz_target!(|_foo: &[u8]| {});

Then, running cargo test --all-targets should result in a message like the following:

   Compiling foobar v0.1.0 (/tmp/foobar)
    Finished test [unoptimized + debuginfo] target(s) in 0.64s
     Running target/debug/deps/foobar-a99608cf2788c81d

running 1 test
test tests::it_works ... ok

test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out

     Running target/debug/deps/example_bin-045b781c1c0562c0
WARNING: Failed to find function "__sanitizer_acquire_crash_state".
WARNING: Failed to find function "__sanitizer_print_stack_trace".
WARNING: Failed to find function "__sanitizer_set_death_callback".
INFO: Seed: 2622528302
INFO: -max_len is not provided; libFuzzer will not generate inputs larger than 4096 bytes
INFO: A corpus is not provided, starting from an empty corpus
#2	INITED exec/s: 0 rss: 24Mb
ERROR: no interesting inputs were found. Is the code instrumented for coverage? Exiting.
error: test failed, to rerun pass '--bin example-bin'

Possible Solution(s)
Maybe it'd be possible to just not call binaries that are tagged test=false, even when ran with --all-targets, seeing as test=false should clearly stipulate that there are no tests to be found here?

Notes

Output of cargo version:
cargo 1.41.0

Anyway, thank you for all you do on cargo! It's a great piece of software, and I just wanted to report this small hiccup :)

[kpreid]

I just ran into essentially this problem with my own fuzz targets, but not with --all-targets:

cargo test foo (test filtering) seems to also ignore test = false (even though the otherwise-equivalent cargo test -- foo doesn't). This seems to contradict the documentation, which says that (emphasis mine):

Setting targets to test = false will stop them from being tested by default. Target selection options that take a target by name ignore the test flag and will always test the given target.

--bins also has this behavior, even though one might expect it to narrow the set of targets, not widen them.

[epage]

Related

• When running cargo test, cargo also re-builds all examples #6675
• Cargo test quicker by not building untested examples when filtered #6683
• Cargo test ignores test flags when testing specific keyword #10268
• do not compile test for bins flagged as test = false #10305