Enforce in bootstrap that dist and install must have stage at least 1 #145472

Kobzol · 2025-08-15T19:16:46Z

This is a continuation of my efforts to fix staging in bootstrap after the stage0 redesign. This PR gets rid of all compiler_for usages in the dist steps 🎉 The only remaining usages are in the test steps, which are my next goal, and which will be the final boss.. both because they are quite tricky, and because most of the the rest of the steps have been already fixed.

The changes are relatively straightforward, x dist defaults to stage 2, so in that case we want to use a stage 1 build compiler for everything (except stdlib, but we usually use it even there because of uplifting).

What I didn't fix yet are the docs steps, because they are very implicit right now, and fixing them deserves its own PR.

The first commit reverts the tests that I accidentally removed in #145340 🤦‍♂️ I did it in this PR to avoid further git conflicts..

Best reviewed commit-by-commit.

r? @jieyouxu

try-job: dist-i686-linux
try-job: dist-armhf-linux
try-job: dist-x86_64-linux
try-job: dist-x86_64-msvc
try-job: dist-apple-various
try-job: dist-x86_64-apple
try-job: x86_64-msvc-2

rustbot · 2025-08-15T19:16:54Z

This PR modifies src/bootstrap/src/core/config.

If appropriate, please update CONFIG_CHANGE_HISTORY in src/bootstrap/src/utils/change_tracker.rs.

Kobzol · 2025-08-20T07:03:21Z

@bors try

Enforce in bootstrap that dist and install must have stage at least 1 try-job: dist-i686-linux try-job: dist-armhf-linux try-job: dist-x86_64-linux try-job: dist-x86_64-msvc try-job: dist-apple-various try-job: dist-x86_64-apple

rust-bors · 2025-08-20T10:07:18Z

☀️ Try build successful (CI)
Build commit: f9464d8 (f9464d83262ce8f495c129d9e1059a8d108f3df9, parent: f605b57042ffeb320d7ae44490113a827139b766)

jieyouxu

Thanks, the other changes I'm on board with. The only bit I'm not sure is the rust-std build optimization, IMO it's not worth the hack to introduce a special case. WDYT?

jieyouxu · 2025-08-20T12:01:44Z

src/bootstrap/src/core/build_steps/dist.rs

        let sysroot = self.sysroot;
        let dst = sysroot.join("lib/rustlib/etc");
        t!(fs::create_dir_all(&dst));
        let cp_debugger_script = |file: &str| {
            builder.install(&builder.src.join("src/etc/").join(file), &dst, FileType::Regular);
        };
-        if host.contains("windows-msvc") {
+        if target.contains("windows-msvc") {


Remark: I really hope in the future we have some way to share an in-tree structured target spec with rustc... Target tuple string matching feels bad every time I see it 😔

jieyouxu · 2025-08-20T12:09:07Z

src/bootstrap/src/core/build_steps/dist.rs

+                "Note: stage {top_stage} library for `{}` would be uplifted from stage 1, so stage was downgraded from {top_stage} to 1 to avoid needless compiler build(s)",
+                run.target


Suggestion: maybe say

so build compiler stage was downgraded from {top_stage} to 1 to avoid needless compiler build(s)

?

jieyouxu · 2025-08-20T12:19:30Z

src/bootstrap/src/core/build_steps/dist.rs

+        // This is a build time optimization for running just `x dist rust-std` (without
+        // `x dist rustc`).
+        // If we know that we will be uplifting a stage2+ library from stage 1 anyway,
+        // there is no point in building a stage2 rustc, which will then not do anything (because
+        // the stdlib will be uplifted).


Discussion:

Who's actually using ./x dist rust-std specifically? Usually, we would be distributing a toolchain together, right? Or at least both compiler and std?

I feel uneasy about doing this build optimization, it feels rather "special" (hacky) especially when this is the dist flow. Could we get away with just doing the straightforward ™️ build? I feel like we should just do the "useless" stage 2 rustc build, and then use the same stage 2 std uplifting logic, instead of doing this special casing.

There are some CI jobs that do ./x dist --host='' --target i686-unknown-linux-gnu,aarch64-unknown-linux-gnu. But I confirmed it produced the same steps with/without this "optimization".

I agree, let's get rid of it.

src/bootstrap/src/core/build_steps/dist.rs

jieyouxu · 2025-08-20T13:04:14Z

@rustbot author

Kobzol · 2025-08-20T13:20:07Z

Agreed, removed the special case, and fixed one doc bug that I found when testing the x dist cross-compilation scenarios (the docs dist steps just invoke all default doc steps, which I'm planning to change, so it exposed this issue. Not sure why we didn't hit it on CI...).

Kobzol · 2025-08-20T13:35:17Z

@rustbot ready

jieyouxu

Thanks!

src/bootstrap/src/core/build_steps/doc.rs

jieyouxu · 2025-08-20T14:19:35Z

@bors r+ rollup=never

bors · 2025-08-20T14:19:38Z

📌 Commit 7da375e has been approved by jieyouxu

It is now in the queue for this repository.

samueltardieu · 2025-08-22T18:05:31Z

I can't seem to create rollups right now, so prefer this rollup=never over the next rollup=iffy that would run alone.

@bors p=1

Enforce in bootstrap that dist and install must have stage at least 1 This is a continuation of my efforts to fix staging in bootstrap after the stage0 redesign. This PR gets rid of all `compiler_for` usages in the `dist` steps 🎉 The only remaining usages are in the `test` steps, which are my next goal, and which will be the final boss.. both because they are quite tricky, and because most of the the rest of the steps have been already fixed. The changes are relatively straightforward, `x dist` defaults to stage 2, so in that case we want to use a stage 1 build compiler for everything (except stdlib, but we usually use it even there because of uplifting). What I didn't fix yet are the docs steps, because they are *very* implicit right now, and fixing them deserves its own PR. The first commit reverts the tests that I accidentally removed in #145340 🤦‍♂️ I did it in this PR to avoid further git conflicts.. Best reviewed commit-by-commit. r? `@jieyouxu` try-job: dist-i686-linux try-job: dist-armhf-linux try-job: dist-x86_64-linux try-job: dist-x86_64-msvc try-job: dist-apple-various try-job: dist-x86_64-apple

rustbot · 2025-08-23T09:17:56Z

This PR was rebased onto a different master commit. Here's a range-diff highlighting what actually changed.

Rebasing is a normal part of keeping PRs up to date, so no action is needed—this note is just to help reviewers.

Kobzol · 2025-08-23T09:19:08Z

@bors try

Enforce in bootstrap that dist and install must have stage at least 1 try-job: dist-i686-linux try-job: dist-armhf-linux try-job: dist-x86_64-linux try-job: dist-x86_64-msvc try-job: dist-apple-various try-job: dist-x86_64-apple try-job: x86_64-msvc-2

rust-bors · 2025-08-23T11:08:17Z

💔 Test for ba8e5de failed: CI. Failed jobs:

try - dist-armhf-linux (web logs, extended logs)
try - dist-i686-linux (web logs, extended logs)
try - dist-x86_64-apple (web logs, extended logs)
try - dist-x86_64-linux (web logs, extended logs)
(and 2 other jobs)

…e same arguments

Kobzol · 2025-08-23T12:31:22Z

Well this was a detective hunt.. Luckily I could reproduce it locally. Because of the changes of this PR, the dist::Std step is now invoked twice during x dist (once with the stage1 build_compiler, and once with the stage2 build_compiler). The actual std build is reused, but the tarball is generated and compressed twice.

Since recently, bootstrap is caching all command executions by default. Since we were caching the rust-installer invocation, which was extracting the rust-std tarball twice, it wasn't executed for the second time, and nothing was thus created on disk, which triggered assertions in bootstrap.

I removed caching for all tool invocations.

Kobzol · 2025-08-23T12:31:29Z

@bors try

Enforce in bootstrap that dist and install must have stage at least 1 try-job: dist-i686-linux try-job: dist-armhf-linux try-job: dist-x86_64-linux try-job: dist-x86_64-msvc try-job: dist-apple-various try-job: dist-x86_64-apple try-job: x86_64-msvc-2

rust-bors · 2025-08-23T15:36:30Z

☀️ Try build successful (CI)
Build commit: 7fcf301 (7fcf30178209c10ea8218e69a3c2f2d02fd368dc, parent: 5b6ceb58f800079f8c59f5194b5db08924693927)

Kobzol · 2025-08-23T19:35:21Z

@rustbot ready

jieyouxu

Mostly looks good, just two questions

View changes since this review

jieyouxu · 2025-08-24T11:40:55Z

src/bootstrap/src/core/build_steps/dist.rs

+        // It's possible that std was uplifted and thus built with a different build compiler
+        // So we need to read the stamp that was actually generated when std was built
+        let stamp =
+            builder.std(build_compiler, target).expect("Standard library has to be built for dist");


Remark: sneaky...

jieyouxu · 2025-08-24T11:50:27Z

src/bootstrap/src/core/build_steps/dist.rs

+            // We currently always ship a stage 2 rustc-dev component, so we build it with the
+            // stage 1 compiler. This might change in the future.
+            // The precise stage used here is important, so we hard-code it.
+            build_compiler: run.builder.compiler(1, run.builder.config.host_target),


Question: hm, does this mean that ./x dist rustc-dev --stage=1 silently does the wrong thing without it being a hard error?

Yes. rustc-dev only works for stage 2, nothing else, currently. I can add a hard error if you want.

I think, at least for the moment, we should hard error instead of ignoring an explicitly requested stage for this.

Though that makes the overall ./x dist awkward... because what if someone writes ./x dist --stage=1?

jieyouxu · 2025-08-24T11:51:49Z

src/bootstrap/src/core/build_steps/dist.rs

-                run.builder.config.host_target,
-                run.target,
-            ),
+            build_compiler: run.builder.compiler(1, run.builder.config.host_target),


Question: same thing, does this silently do the wrong thing if someone specifies ./x dist analysis --stage=1? Not that this does much in the first place...

jieyouxu · 2025-08-24T11:56:52Z

src/bootstrap/src/core/build_steps/tool.rs

+        // Do not cache tool invocations, as they can have side effects
+        cmd.do_not_cache();


Remark: hmm, this is rather sneaky...

When we implemented command caching, I was worried about bugs being caused by it, but I had no "proof". This PR showed that such bugs can indeed happen, and that it is VERY annoying to debug. This would be a permanent footgun, so I will instead change the default and make caching opt-in, not opt-out.

jieyouxu · 2025-08-24T12:02:17Z

@rustbot author

rustbot assigned jieyouxu Aug 15, 2025

This comment has been minimized.

Sign in to view

This comment was marked as outdated.

Sign in to view

Kobzol force-pushed the bootstrap-dist branch from 7c6b798 to ad8384e Compare August 20, 2025 07:00

This comment has been minimized.

Sign in to view

jieyouxu approved these changes Aug 20, 2025

View reviewed changes

rustbot added S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Aug 20, 2025

rustbot added S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. and removed S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. labels Aug 20, 2025

jieyouxu approved these changes Aug 20, 2025

View reviewed changes

src/bootstrap/src/core/build_steps/doc.rs Show resolved Hide resolved

bors added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Aug 20, 2025

Kobzol added 11 commits August 23, 2025 11:00

Remove compiler_for from dist::Extended

e68c93b

Cleanup dist::Bootstrap and add a simple test for it

699e5fe

Cleanup the last few dist steps

e8af236

Forbid running install steps on stage 0

2074e13

Fix staging in x install

3ec2abc

Add change tracker entry

0579f05

Fix doc link

ba1bc64

Don't try to cross-document bootstrap tools

3303770

Remove the x dist rust-std built optimization special-case

0470945

Fix install test on Windows

21c30ea

Handle uplifting in libstd dist step

a62488b

Kobzol force-pushed the bootstrap-dist branch from 9a629ec to a62488b Compare August 23, 2025 09:17

This comment has been minimized.

Sign in to view

Kobzol added 2 commits August 23, 2025 14:17

Use correct mode when printing a build message for std documentation

600d9bf

Allow running rust-installer (and other tools) multiple times with th…

412734d

…e same arguments

This comment has been minimized.

Sign in to view

jieyouxu reviewed Aug 24, 2025

View reviewed changes

rustbot added S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Aug 24, 2025

		"Note: stage {top_stage} library for `{}` would be uplifted from stage 1, so stage was downgraded from {top_stage} to 1 to avoid needless compiler build(s)",
		run.target

		// Do not cache tool invocations, as they can have side effects
		cmd.do_not_cache();

Enforce in bootstrap that dist and install must have stage at least 1 #145472

Are you sure you want to change the base?

Enforce in bootstrap that dist and install must have stage at least 1 #145472

Conversation

Kobzol commented Aug 15, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

rustbot commented Aug 15, 2025

Uh oh!

This comment has been minimized.

This comment has been minimized.

This comment was marked as outdated.

This comment has been minimized.

Kobzol commented Aug 20, 2025

Uh oh!

This comment has been minimized.

rust-bors bot commented Aug 20, 2025

Uh oh!

jieyouxu left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

jieyouxu commented Aug 20, 2025

Uh oh!

Kobzol commented Aug 20, 2025

Uh oh!

Kobzol commented Aug 20, 2025

Uh oh!

jieyouxu left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

jieyouxu commented Aug 20, 2025

Uh oh!

bors commented Aug 20, 2025

Uh oh!

samueltardieu commented Aug 22, 2025

Uh oh!

rustbot commented Aug 23, 2025

Uh oh!

Kobzol commented Aug 23, 2025

Uh oh!

This comment has been minimized.

This comment has been minimized.

rust-bors bot commented Aug 23, 2025

Uh oh!

Kobzol commented Aug 23, 2025

Uh oh!

Kobzol commented Aug 23, 2025

Uh oh!

This comment has been minimized.

rust-bors bot commented Aug 23, 2025

Uh oh!

Kobzol commented Aug 23, 2025

Uh oh!

jieyouxu left a comment • edited by rustbot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jieyouxu Aug 24, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Kobzol commented Aug 15, 2025 •

edited

Loading

jieyouxu left a comment •

edited

Loading

jieyouxu left a comment •

edited by rustbot

Loading

jieyouxu Aug 24, 2025 •

edited

Loading