Doctest slowdown with large target directory

[ehuss]

I experienced a problem while working on a project where trivial doctests were taking multiple seconds to run as my target directory got larger. I think the problem is amplified on macOS due to using unpacked debuginfo.

Reproduction

This reproduction shows a sample project, filling up the target directory, and seeing how long a doctest takes to run.

git clone https://github.com/rust-lang/mdbook.git
cd mdbook
git checkout 3236ec0aea1fb520fe60ea15acb74d7261fb5505
# Warm up the target cache
cargo build
cargo test -p mdbook-core --doc
# Check the disk usage. (~1.2G for me)
du -sh target
# Let's see how long just running rustdoc takes as a baseline. (0.329s for me)
time cargo test -p mdbook-core --doc
# Let's fill up our target directory with lots of files. 
# This simulates the impact of doing lots of development (changing features, compiler versions, and other changes).
for i in {1..20}; do echo number $i; CARGO_PROFILE_DEV_SPLIT_DEBUGINFO=unpacked RUSTFLAGS="$(yes -- "-Aunused " | head -n $i)" cargo build --quiet; done
# How big is it now? (~19G for me)
du -sh target
# Let's see how a full target directory affects our doctest. (1.225s for me, 3.7x slower)
time cargo test -p mdbook-core --doc

-L duplicates

The problem isn't so much that the target directory was very large (although that is a problem). The problem is that rustdoc is passing -L target/debug/deps to rustc multiple times (one for each --extern).

From the log output, we can see that rustdoc is essentially running:

RUSTC_BOOTSTRAP="1" rustc "--edition" "2024" "--crate-type=bin" \
    "-o" "doctests/merged_doctest_2024_0/rust_out" \
    "-L" "target/debug/deps" \
    "-L" "target/debug/deps" \
    "-L" "target/debug/deps" \
    "-L" "target/debug/deps" \
    "-L" "target/debug/deps" \
    "-L" "target/debug/deps" \
    "-L" "target/debug/deps" \
    "-L" "target/debug/deps" \
    "--extern=doctest_bundle_2024=doctests/merged_doctest_2024_0/libdoctest_bundle_2024.rlib" \
    "doctests/merged_doctest_2024_0/doctest_runner_2024.rs"

Let's try that command with just one -L: 0m0.754s
Let's try that command with 20 -L: 0m7.546s ❗

Possible solution

I think there are roughly two issues that could be fixed:

• rustdoc should not be passing duplicate -L flags
• rustc could be de-duplicating the list of -L directories (not sure if that would complicate any CLI order sensitivity?)

Notes

My target/debug/deps directory for real-world development (not involving RUSTFLAGS at all, maybe 3 or so versions of rustc) was 14G, 299727 files (2003 rmeta, 861 rlib, 294129 o files).

rustc 1.91.0-nightly (1ebbd87a6 2025-08-11)
binary: rustc
commit-hash: 1ebbd87a62ce96a72b22da61b7c2c43893534842
commit-date: 2025-08-11
host: aarch64-apple-darwin
release: 1.91.0-nightly
LLVM version: 21.1.0