Skip to content

Cargo runs unit tests when they are disabled and --lib is passed #13668

@RalfJung

Description

@RalfJung
Member

Problem

When a crate has no unit tests (via test = false in its manifest), I would expect cargo to never ever try to run unit tests for that crate.

And yet it turns out that cargo test --lib will run the unit tests.

Steps

  1. Run cargo test --lib --bins --examples --tests --benches (in an attempt to run "everything except for doc tests"), or just cargo test --lib
  2. Note how even if the library has unit tests are disabled, cargo will run them
     Running unittests src/lib.rs (build/x86_64-unknown-linux-gnu/stage0-std/x86_64-unknown-linux-gnu/release/deps/core-04ceb2ced8c36de5)

running 0 tests


test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 7.54µs

Possible Solution(s)

--lib should just be a filter, removing from the usual test set everything that's not a "library unit test". It should not add things that were not present to begin with.

Notes

This is related to but different from #10936

Version

No response

Activity

added
C-bugCategory: bug
S-triageStatus: This issue is waiting on initial triage.
on Mar 29, 2024
RalfJung

RalfJung commented on Mar 29, 2024

@RalfJung
MemberAuthor

Something similar seems to happen with benchmarks. With a manifest like this

[[bench]]
name = "allocbenches"
path = "benches/lib.rs"
test = true

[[bench]]
name = "vec_deque_append_bench"
path = "benches/vec_deque_append.rs"
harness = false

Running cargo test will only run the first benchmark, but running cargo test --lib --bins --examples --tests --benches will run both -- even though it's not a test and shouldn't be run as a test.

weihanglo

weihanglo commented on Mar 29, 2024

@weihanglo
Member

This seems to be covered by rust-lang/testing-devex-team#1> Once we have a new JSON output format, Cargo could get more controls over tests than relying on libtest.

added
S-blocked-externalStatus: ❌ blocked on something out of the direct control of the Cargo project, e.g., upstream fix
and removed
C-bugCategory: bug
S-triageStatus: This issue is waiting on initial triage.
on Mar 29, 2024
added
C-feature-requestCategory: proposal for a feature. Before PR, ping rust-lang/cargo if this is not `Feature accepted`
on Mar 29, 2024
RalfJung

RalfJung commented on Mar 29, 2024

@RalfJung
MemberAuthor

I don't think this issue has anything to do with handling the output from the tests. It's all about which tests cargo decides to run.

weihanglo

weihanglo commented on Mar 29, 2024

@weihanglo
Member

I see. Sorry I understood it wrong.

--lib is a flag for user to explicitly requests tests in library target to be run, so that is expected. So does other flags like --test, --bench, etc.

test = false controls only the default set of tests being run. See the target selection section.

Setting targets to test = false will stop them from being tested by default. Target selection options that take a target by name (such as --example foo) ignore the test flag and will always test the given target.

weihanglo

weihanglo commented on Mar 29, 2024

@weihanglo
Member

--lib should just be a filter, removing from the usual test set everything that's not a "library unit test". It should not add things that were not present to begin with.

The proposed solution can only be solved with rust-lang/testing-devex-team#1, from my understanding, because Cargo needs to run libtest to know if there is any unit test defined.

RalfJung

RalfJung commented on Mar 29, 2024

@RalfJung
MemberAuthor

test = false controls only the default set of tests being run. See the target selection section.

That's very surprising... but it explains a lot, thanks.

added a commit that references this issue on Mar 30, 2024
added a commit that references this issue on Mar 31, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    C-feature-requestCategory: proposal for a feature. Before PR, ping rust-lang/cargo if this is not `Feature accepted`Command-testS-blocked-externalStatus: ❌ blocked on something out of the direct control of the Cargo project, e.g., upstream fix

    Type

    No type

    Projects

    Status

    No status

    Milestone

    No milestone

    Relationships

    None yet

      Development

      No branches or pull requests

        Participants

        @RalfJung@weihanglo

        Issue actions

          Cargo runs unit tests when they are disabled and `--lib` is passed · Issue #13668 · rust-lang/cargo