Cargo build hangs

[Emilgardis]

originally posted in zulip

Hi! I've found a regression in compilation introduced in #85499,

searched nightlies: from nightly-2021-08-17 to nightly-2021-09-10
regressed nightly: nightly-2021-08-26
searched commits: from b03ccac to 0afc208
regressed commit: 0afc208

bisected with cargo-bisect-rustc v0.6.0

Host triple: x86_64-pc-windows-msvc
Reproduce with:

# bisect-script.bat
pwsh.exe -noexit -Command "stop-job -Name cargobuild_bisect; if ((start-job -Name cargobuild_bisect { cargo build -p server 2>&1 } | wait-job -timeout 600).State -ne 'Completed') {receive-job -Name cargobuild_bisect; receive-job -Name cargobuild_bisect | remove-job -Force; exit 10} else {receive-job -Name cargobuild_bisect; receive-job -Name cargobuild_bisect | remove-job -Force; exit 0}"

cargo bisect-rustc --start=2021-08-17 --script .\bisect-script.bat

The issue is that compilation succeeds in a acceptable time before this change, but now it's seemingly taking forever, I have not tried this on linux, only windows so far. I'm not sure how I should report this because I'm not able to reproduce it outside my large workspace where I bisected this. Does anyone have an idea to what this could be or have a good way to figure out what the cause is?

some notable dependencies: actix-web, sqlx, serde, chrono, ring, tokio, tracing, reqwest,anyhow,eyre,async-trait and many more

I've tried

$ export CARGO_LOG="info"
$ export RUSTC_LOG="info" # to stop massive console spam I use this instead = "info,rustc_trait_selection=warn,rustc_codegen_ssa=warn,rustc_metadata=warn,rustc_mir=warn,rustc_query_system=warn,rustc_const_eval=warn"
$ cargo build -p server

which outputs at the end

┐rustc_query_system::dep_graph::serialized::encode_node node=NodeInfo { node: variances_of(7b5e03ed597c3c33-573328205f4d7f81), fingerprint: Fingerprint(12749739837320991735, 10040035723298659737), edges: [2870] }
┘
┐rustc_query_system::dep_graph::serialized::encode_node node=NodeInfo { node: variances_of(7b5e03ed597c3c33-6f459134164ff719), fingerprint: Fingerprint(8361987967435134197, 5478714612052799554), edges: [2870] }
┘
[2021-09-11T16:54:37Z INFO  cargo::core::compiler::job_queue] tokens in use: 0, rustc_tokens: [], waiting_rustcs: [] (events this tick: 0)
┐rustc_query_system::dep_graph::serialized::encode_node node=NodeInfo { node: variances_of(7b5e03ed597c3c33-82cb26faded76d4), fingerprint: Fingerprint(12749739837320991735, 10040035723298659737), edges: [2870] }
[2021-09-11T16:54:40Z INFO  cargo::core::compiler::job_queue] tokens in use: 0, rustc_tokens: [], waiting_rustcs: [] (events this tick: 1)
┘
[2021-09-11T16:54:40Z INFO  cargo::core::compiler::job_queue] tokens in use: 0, rustc_tokens: [], waiting_rustcs: [] (events this tick: 0)
┐rustc_query_system::dep_graph::serialized::encode_node node=NodeInfo { node: variances_of(7b5e03ed597c3c33-7c40a2b5f026df66), fingerprint: Fingerprint(12526569172406770563, 17126563881511896260), edges: [2870] }
[2021-09-11T16:54:40Z INFO  cargo::core::compiler::job_queue] tokens in use: 0, rustc_tokens: [], waiting_rustcs: [] (events this tick: 1)
┘
[2021-09-11T16:54:40Z INFO  cargo::core::compiler::job_queue] tokens in use: 0, rustc_tokens: [], waiting_rustcs: [] (events this tick: 0)
┐rustc_query_system::dep_graph::serialized::encode_node node=NodeInfo { node: variances_of(7b5e03ed597c3c33-c7d9e80976e93c5), fingerprint: Fingerprint(12526569172406770563, 17126563881511896260), edges: [2870] }
[2021-09-11T16:54:40Z INFO  cargo::core::compiler::job_queue] tokens in use: 0, rustc_tokens: [], waiting_rustcs: [] (events this tick: 1)
┘
[2021-09-11T16:54:40Z INFO  cargo::core::compiler::job_queue] tokens in use: 0, rustc_tokens: [], waiting_rustcs: [] (events this tick: 0)
    Building [=======================> ] 445/446: server(bin)

and then just hangs.

How do I proceed?

[camelid]

cc @jackh726

[camelid]

One way to minimize could be to see which crate in your workspace is triggering the hang. E.g., run cargo build -p my_crate for each of the crates in the workspace and find the one with the fewest dependencies (in-workspace dependencies, or external). Then you can try to minimize that crate further by deleting different parts of the code until the hang disappears.

[Emilgardis]

The crate/step which is failing is the final one. I managed to delete parts of the code to make the hang disappear but I'm not certain I've actually pinpointed it exactly yet.

I let it run and seems it failed on STATUS_STACK_BUFFER_OVERRUN
https://gist.github.com/Emilgardis/954c2aeefc0cca270a421a90ecc5067a#file-output-cargo-issue88862-log-L988

[Emilgardis]

ran with

[profile.dev]
lto = "off"
panic = "abort"
incremental = false

and the build completed after a while. The memory usage jumps up dramatically, from 138 MB to 7GB to 20GB, and then down again to repeat.

Same behaviour without those flags.

This is probably not a hang, just increased complexity causing longer run-time.

EDIT: On older nightly, the max memory usage is around 1.5GB

[Emilgardis]

Managed to minimize it somewhat. Was not able to remove all dependencies but most of them.

❯ cargo -Vv                                           
cargo 1.56.0-nightly (e96bdb0c3 2021-08-17)
release: 1.56.0
commit-hash: e96bdb0c3d0a418e7fcd7fbd69be08abf830b4bc
commit-date: 2021-08-17

takes 40-45 seconds to compile from clean state.

❯ cargo -Vv                                           
cargo 1.56.0-nightly (18751dd3f 2021-09-01)
release: 1.56.0
commit-hash: 18751dd3f238d94d384a7fe967abfac06cbfe0b9
commit-date: 2021-09-01

takes 125-500 seconds to compile from clean state. (and uses 5-7GB of memory sometimes)

I believe the issue is to do with the client traits and impls and tracing instrumentation. Replacing upsert_token with a todo!() makes compilation sane again.

https://gist.github.com/Emilgardis/194812fa2e73ff6839d9942163329887

[camelid]

@rustbot ping cleanup

I think it would be useful to further minimize the above Gist.

[rustbot]

Hey Cleanup Crew ICE-breakers! This bug has been identified as a good
"Cleanup ICE-breaking candidate". In case it's useful, here are some
instructions for tackling these sorts of bugs. Maybe take a look?
Thanks! <3

cc @AminArria @camelid @chrissimpkins @contrun @DutchGhost @elshize @h-michael @HallerPatrick @hdhoang @hellow554 @henryboisdequin @imtsuki @JamesPatrickGill @kanru @KarlK90 @MAdrianMattocks @matheus-consoli @mental32 @nmccarty @Noah-Kennedy @pard68 @PeytonT @pierreN @Redblueflame @RobbieClarken @RobertoSnap @robjtede @SarthakSingh31 @shekohex @sinato @smmalis37 @steffahn @Stupremee @tamuhey @turboladen @woshilapin @yerke