MSRV resolver has surprising behaviour in workspaces with mixed MSRV

[VorpalBlade]

Problem

As state by @epage in #13873 (comment) , a new issue was preferred for this. The issue is that the new MSRV aware resolver leads to surprising and suboptimal behaviour in a workspace with mixed MSRV:

• Call for Testing: MSRV-aware resolver #13873 (comment)
• Call for Testing: MSRV-aware resolver #13873 (comment)
• Call for Testing: MSRV-aware resolver #13873 (comment)

The issue is that (for whatever reason) you have a workspace where one (or more) of you crates is on a much older MSRV than the rest. Then the MSRV aware resolver will attempt to create a lockfile for all of the workspace based on that oldest version even if the dependency in question isn't even used by that old-MSRV crate. In certain cases this can cause there to be no possible solutions to the dependency resolution (as detailed by the comment linked in the third bullet point above).

There are valid reasons to have mixed MSRV, such as the different crates having different MSRV policy. or just that some crates naturally have solver development and don't need to be bumped to a newer MSRV as often as other crates.

I will here describe my use case where I run into this (I cannot speak for the cases where others run into this as what @Darksonn described):

I have a workspace I use to develop a command line program. Having it all the related crates (both the main binary and supporting libraries) all in one workspace makes development much easier, especially during the early phases when everything is evolving rapidly. Some of these crates are generally useful to others as well (file format parsers in my case).

I also have an agressive MSRV policy: only N-2 is guaranteed (but I won't bump MSRV gratuitously just for the sake of it, so actual MSRV can be much further back).

Eventually some of these crates will start to stabilise, especially the "leaf node" supporting crates with simple and clearly defined scope. There is only so many things you need in a library that parses a file format. That means it won't get breaking changes nor get published to crates.io as often. And also that there will be no reason to bump MSRV.

Now the problem here is that this leaf node dependency with older MSRV will now drag down the versions of dependencies that get used for my main crate of the workspace (the "flagship" binary crate).

Steps

1. Have a (virtual?, I don't know if it affects non-virtual as well) workspace with mixed MSRV
2. CARGO_RESOLVER_INCOMPATIBLE_RUST_VERSIONS=fallback cargo +nightly -Zmsrv-policy generate-lockfile

Possible Solution(s)

Possible solutions (and why they are suboptimal):

• I could bump the MSRV and publish releases to crates.io that are nothing but "bump MSRV". This leads to churn and "pointless" releases from the point of view of everyone else.
• I could selectively publish single crates from the workspace (update the MSRV of the dependency crates but don't publish them). It would be error prone:
  • I would need to ensure that I didn't start depending on any "unreleased commits". Sounds like I need a "cargo-reverse-semver-checks"?
  • This would be complicated since I automate publishing from CI using release-plz. Partial publishing is as far as I understand not supported with the automated CI/PR workflow, only when publishing locally from your own computer (which is bad practise).

The better option would seem to me that cargo would consider the entire dependency graph when writing the lock file and look at what dependencies are reachable from what crates, and compute the minimum needed MSRV for each dependency.

Notes

No response

Version

❯ cargo +nightly version --verbose
cargo 1.82.0-nightly (2f738d617 2024-08-13)
release: 1.82.0-nightly
commit-hash: 2f738d617c6ead388f899802dd1a7fd66858a691
commit-date: 2024-08-13
host: x86_64-unknown-linux-gnu
libgit2: 1.8.1 (sys:0.19.0 vendored)
libcurl: 8.9.0-DEV (sys:0.4.74+curl-8.9.0 vendored ssl:OpenSSL/1.1.1w)
ssl: OpenSSL 1.1.1w  11 Sep 2023
os: Arch Linux Rolling Release [64-bit]

[epage]

btw Cargo is a multi-MSRV workspace but we only have two and one of those is "N" so we avoid most of this problem.

[epage]

So another way to put this is that when you are in a mixed-MSRV workspace, you get proper handling of your lowest MSRV but for all other MSRVs you get one of

• Older dependencies than necessary
• No MSRV-aware resolver

Depending on the dependency and how "wide" of a version requirement you have.

For older dependencies, the answer is "just pick a higher version req" but then that puts you in the second situation unless you make your MSRV "N"

[epage]

As this puts this use caae in the "No MSRV-aware resolver" situation, this is no worse off than we are today except for a communication / expectations issue.

The MSRV-aware resolver not being path aware is a fundamental limitation of the "fallback" strategy. Quoting from #13873 (comment)

When we are evaluating a version requirement, we don't know what all version requirements might apply. We filter and sort to find a version we pick and move on. If a chosen version is filtered out for another version req, we backtrack. fallback applies to the sorting step, not the filtering step, and never gets re-evaluated unless something else causes enough of a backtrack. If we sorted per version requirement, what you got would be dependent on the order the dependency tree was walked.

Originally we had considered incompatible-rust-versions = "deny" as the policy. The quality of the error messages from the current resolver is a blocker to that policy and stopped us from looking any further into this. In discussing fallback, other workflows came up that wouldn't work with deny, like no-MSRV-dev-dependencies or per-feature MSRVs.

The fallback mechanism was a compromise to get something that works now.

Once we have the PugGrub resolver, we'll also need to figure out how to map package versions back to workspace members back to know which MSRVs apply.

Neither of these facts mean we shouldn't consider ways of improving things.

[epage]

So far, the only idea I have is that we collect all MSRVs in the workspace and sort the version candidates into MSRV buckets. If a version req disallows versions in the lowest MSRV bucket, consider the next MSRV bucket on up.

So long as your version requirements are MSRV compatible, you'll get an MSRV-aware resolve. You just might get older versions than you absolutely need for MSRVs sake but you did say you are compatible with it, so I'm not too worried without mitigating context being brought up.

I just checked cargo add and it checks the MSRV of the package being added to so that will play nicely with this scheme though it may need tweaks for #10608.

[epage]

Before I forget, it would be very easy for dependency unification to cause a package with a high MSRV to make it so you can't verify a package with a low MSRV, so the mixed-MSRV workflow is already a bit "here be dragons" (which is why we need to be careful with #10608).

[VorpalBlade]

A third possible workaround would be to split out "stabilised crates" from your workspace into their own separate git repos. That is slightly non-trivial though (you need to use some arcane git commands if you want to keep the history, you need to set up CI anew, generate new and separate publishing secrets, possibly add contributors again if there are other contributors, etc etc). Doable, but there is a definite barrier to entry. Plus if you go back to a active development phase later on again you won't have the benefits. Monorepos are in many ways a lot nicer to work with.

Frankly it is a bit more work than I have the energy to deal with right now, but I might consider it. Bonus: It would also simplify MSRV CI testing (that is a bit complex in a multi-MSRV repo),

So far, the only idea I have is that we collect all MSRVs in the workspace and sort the version candidates into MSRV buckets. If a version req disallows versions in the lowest MSRV bucket, consider the next MSRV bucket on up.

This sounds similar to my graph suggestion (I guess it depends on what your internal representation is). If it is a full blown graph you could traverse the graph from the workspace crates, and mark each crate with the oldest MSRV it needs to support. Then you have your answer there. Though a complication is that an older/newer version of a dependency might have different dependencies transitive dependencies I guess.

Before I forget, it would be very easy for dependency unification to cause a package with a high MSRV to make it so you can't verify a package with a low MSRV, so the mixed-MSRV workflow is already a bit "here be dragons" (which is why we need to be careful with #10608).

I do put all dependencies (and features that aren't tied to feature flags of my own crates) in the workspace currently. This seems to help a bit with feature unification (without going to a full blown workspace hack, which I have been considering, though the publishing story for that seems complex).

I think it is fine if you get a (somewhat reasonable) error message for these tricky situations though. Unexpected behaviour is worse, but a "Can't resolve tokio due to conflicting MSRV requirements from crate_a and crate_b" is perfectly fine.

[epage]

A third possible workaround would be to split out "stabilised crates" from your workspace into their own separate git repos. That is slightly non-trivial though (you need to use some arcane git commands if you want to keep the history, you need to set up CI anew, generate new and separate publishing secrets, possibly add contributors again if there are other contributors, etc etc). Doable, but there is a definite barrier to entry. Plus if you go back to a active development phase later on again you won't have the benefits. Monorepos are in many ways a lot nicer to work with.

You don't have to go so far, you can have multiple workspaces in one repo. Still not great.

Without worrying about transient state like "stabilization level" (unless you coupled MSRV to that), you could break up your workspace by MSRV. I've been tempted to do that for cargo.

This sounds similar to my graph suggestion (I guess it depends on what your internal representation is). If it is a full blown graph you could traverse the graph from the workspace crates, and mark each crate with the oldest MSRV it needs to support. Then you have your answer there. Though a complication is that an older/newer version of a dependency might have different dependencies transitive dependencies I guess.

If I'm understanding, you are suggesting we traverse the transitive dependency graph, marking packages with the MSRV that they need to uphold. The problem is we are in the process of building that graph as we go and there can be multiple paths to a single dependency and by the time we walk down the second path to it, it is already selected. We'd need a solution that allows us to then reject that dependency and try again, like when you resolve for foo = "1.0" and foo = "1.1" with -Zminimal-versions. We first select 1.0.0 but then find it doesn't work, so back track and try again with. Worse, we'd then need to re-traverse foos dependencies and make sure they also satisfy the new MSRV. This only works with incompatible-rust-versions = "deny" which is blocked at this time.

[VorpalBlade]

Oh right, you need each graph node to be a set of possible versions (with associated MSRV versions). Or do search with backtracking (which likely won't be performant, it seldom is). And yes indirect dependencies (that may contain subgraphs that are also reachable via other paths, this is a DAG not a tree after all) makes this really annoying.

[Darksonn]

I would find it really useful if there was a flag to tell cargo what version to use as MSRV when resolving packages. That way, I can at least use that in cases that are too complicated for the tool to handle automatically. I also have a case where I want to resolve using a different version than the MSRV of any crate in the workspace. Specifically, I'd want to resolve using whichever rustc was the latest release when I last used the branch.

[epage]

That way, I can at least use that in cases that are too complicated for the tool to handle automatically

Say we move forward with my earlier proposal, I assume these use cases would still exist? Could you describe what the use case if? Is this something that would need to block any MSRV-aware resolver from being released and, if so, why?

I also have a case where I want to resolve using a different version than the MSRV of any crate in the workspace. Specifically, I'd want to resolve using whichever rustc was the latest release when I last used the branch.

Could you describe the use case and the motivation behind it?

Most likely, we'll want to split the conversation about manually specifying the rust-version to resolve to as a separate issue and it will be important to understand the problems that are being solved with it to know how to prioritize it. My inclination at this moment would be to not adjust the status from "future possibility".

[VorpalBlade]

Manually specifying the MSRV to use for resolution could be a viable workaround for me, since the crates that become stable for me tend to have fairly few and slow moving dependencies.

It could however be risky, should one of those dependencies bump MSRV if I don't notice it. In my case CI would notice that, but that isn't a given for everyone.