Skip to content

rust-src does not include workspace Cargo.toml #95736

@jonhoo

Description

@jonhoo
Contributor

The rust-src component that gets installed by Rustup to ~/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/src/rust includes a Cargo.lock that spans the standard library crates, but does not include a corresponding Cargo.toml (essentially this Cargo.toml).

Cargo's -Zbuild-std feature makes this work anyway by constructing a virtual workspace for the standard library, but this doesn't work for editors that want to provide users with completion and analysis all the way into std/alloc/core. They end up running cargo metadata directly in the various standard library crates (library/std for example) (IntelliJ, rust-analyzer). However, since there's no workspace-level Cargo.toml next to the Cargo.lock, they do not pick up on the locked versions at all, and instead use whatever the latest versions of the standard library's dependencies are. This in turn means that go-to-definition and similar mechanisms do not go to the actual versions used by the standard library the user is actually using.

In chatting to @Mark-Simulacrum about this, I think the editors are arguably doing the right thing here by running cargo metadata inside the individual crates (like std), and Cargo is doing the right thing not looking at the Cargo.lock since there isn't a workspace there. And it feels like the right thing to do is to synthesize a workspace Cargo.toml for inclusion with rust-src. But if we explicitly won't (or can't) include a Cargo.toml in rust-src's root, we have a couple of options:

  • We could maybe special-case Cargo to recognize when it's run on a crate from within the standard library.
  • cargo metadata could have something like --with-lockfile Cargo.lock so that you can run it in arbitrary places on disk while pointing to (for example) a checked out repositories Cargo.lock.
  • We could change Cargo to always include std + deps in everyone's Cargo.lock as some kind of special dep kind.
  • We could create cargo metadata --include-std.

All of these feel brittle to one extent or another though, and it's not clear that they are sufficient for what editors may need (cc @matklad), so I think we should push for including Cargo.toml if we can.

Meta

rustc --version --verbose:

rustc 1.59.0 (9d1b2106e 2022-02-23)
binary: rustc
commit-hash: 9d1b2106e23b1abd32fce1f17267604a5102f57a
commit-date: 2022-02-23
host: x86_64-unknown-linux-gnu
release: 1.59.0
LLVM version: 13.0.0

Activity

jonhoo

jonhoo commented on Apr 6, 2022

@jonhoo
ContributorAuthor

A simple way to reproduce this is:

$ cd ~/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/
$ cargo metadata --locked
warning: please specify `--format-version` flag explicitly to avoid compatibility problems
    Updating crates.io index
error: the lock file ~/.rustup/toolchains/stable-x86_64-apple-darwin/lib/rustlib/src/rust/library/std/Cargo.lock needs to be updated but --locked was passed to prevent this
If you want to try to generate the lock file without accessing the network, remove the --locked flag and use --offline instead.
jonhoo

jonhoo commented on Apr 6, 2022

@jonhoo
ContributorAuthor

This also seems like a regression from #44076

matklad

matklad commented on Apr 6, 2022

@matklad
Contributor

The absolute best for the editors would be if /library was a separate workspace with dedicated Cargo.toml, Cargo.lock, and vendored sourced of crates.io deps (such that cargo-metadata works without net).

Practically, this would allow us to not worry about "large" compiler codebase and only deal with the small set of library crates. Theoretically, it still feels to me that "compiler" and "library" actually exist in separate dimensions, and using a single Cargo.lock for them seems accidental.

Obviously, that's "perfect is the enemy of the good" kind of thing, just restoring Cargo.toml/.lock would be helpful

jonhoo

jonhoo commented on Apr 6, 2022

@jonhoo
ContributorAuthor

Another thing worth adding to this is that cargo metadata only accidentally works in rustlib/src/rust/library/* because empty versions of crates like rustc-std-workspace-alloc exist on crates.io. Without the workspace Cargo.toml to pick up the local path, that means the editors end up going to fetch the crates.io versions of these even though the right versions are sitting in rustlib/src/rust/library.

RalfJung

RalfJung commented on Apr 10, 2022

@RalfJung
Member

FWIW, the lock file is included because I needed it to make Miri more reliable, and then just added it. I think a PR to simply add Cargo.toml to the list of shipped files would probably go through in a similar vein.

However, it is not clear to me whether you are saying the original Cargo.toml should be shipped, or whether it should be somehow modified before shipping. The latter will be more tricky, of course -- is it even clear what the "right" way to adjust the toml file is?

RalfJung

RalfJung commented on Apr 10, 2022

@RalfJung
Member

Oh, you actually linked to my old PR that added the Cargo.lock file. 😂 Looks like it originally intended to also add Cargo.toml but that part got bailed out before landing.

This is where I had to remove the Cargo.toml because it caused trouble with distcheck that I didn't have the patience to debug.

Another thing worth adding to this is that cargo metadata only accidentally works in rustlib/src/rust/library/* because empty versions of crates like rustc-std-workspace-alloc exist on crates.io.

FWIW, those crates existing on crates.io is quite deliberate and a key part to their function.

eddyb

eddyb commented on Apr 11, 2022

@eddyb
Member

Cross-posting (potentially relevant) #78790 (comment) (note that that PR was reverted):

The implementation is a bit hacky, hiding the workspace and copying in the lockfile. So perhaps some other things need to be pulled in in a similar manner. That said, I would hope that --offline Just Works here.

Not sure where to post this but AFAICT this is fixable by generating a workspace Cargo.toml for the rust-src dist/vendor step, with a [patch.crates-io] section that contains:

rust/Cargo.toml

Lines 129 to 133 in 7941b3f

# See comments in `library/rustc-std-workspace-core/README.md` for what's going on
# here
rustc-std-workspace-core = { path = 'library/rustc-std-workspace-core' }
rustc-std-workspace-alloc = { path = 'library/rustc-std-workspace-alloc' }
rustc-std-workspace-std = { path = 'library/rustc-std-workspace-std' }

This could potentially even allow Cargo to stop hardcoding them - if we did ship that Cargo.toml, at least.

bjorn3

bjorn3 commented on Apr 11, 2022

@bjorn3
Member

Not sure where to post this but AFAICT this is fixable by generating a workspace Cargo.toml for the rust-src dist/vendor step, with a [patch.crates-io] section that contains:

I think we should just split the sysroot into a new workspace and use it as part of bootstrapping instead of generating a new workspace in the dist step. That may also simplify the tidy license check by not requiring to check if a crate is a dependency of the sysroot to apply stricter license checks. Instead it can apply them to the entire contents of library/Cargo.lock.

jonhoo

jonhoo commented on Apr 14, 2022

@jonhoo
ContributorAuthor

One more observation: this problem is exacerbated by the fact that when an editor runs cargo metadata in, say, <rust-src>/library/std, it will end up generating a Cargo.lock. The same will happen when it's run in alloc, core, etc. This means you end up with one Cargo.lock in every sub-crate, plus one at the root, all of which can contain different versions from one another since all the sub-crate ones contain whatever was the latest version at the time cargo metadata was run.

added a commit that references this issue on May 22, 2022
c7c5980
lf-

lf- commented on Oct 10, 2022

@lf-
Contributor

This winds up extra fun under Nix using $RUST_SRC_PATH where the rust sources are literally on a readonly file system, which is the r-a issue here: rust-lang/rust-analyzer#13393

The correct fix for this issue is to add the lockfile and manifest to rust-src, I think, so cargo doesn't try to relock it.

RalfJung

RalfJung commented on Oct 10, 2022

@RalfJung
Member

IMO it's a bug in cargo metadata that it doesn't work on a read-only file system (rust-lang/cargo#10096).

added
T-libsRelevant to the library team, which will review and decide on the PR/issue.
and removed on Jul 8, 2024
Veykril

Veykril commented on Aug 5, 2024

@Veykril
Member

I believe #128534 essentially fixes this?

bjorn3

bjorn3 commented on Aug 5, 2024

@bjorn3
Member

It does. I would have linked this issue from that PR if I had remembered this issue existed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    C-bugCategory: This is a bug.T-libsRelevant to the library team, which will review and decide on the PR/issue.

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

      Development

      No branches or pull requests

        Participants

        @eddyb@Enselic@jonhoo@RalfJung@matklad

        Issue actions

          rust-src does not include workspace Cargo.toml · Issue #95736 · rust-lang/rust