RFC: Extended Standard Library (ESL)

[bdaehlie]

This RFC describes an Extended Standard Library (ESL) for Rust.

Please see this document for additional information about fundraising.

I’d like to thank Dirkjan Ochtman (@djc), Adolfo Ochagavía (@aochagavia), Alex Gaynor (@alex), James Munns (OneVariable GmbH, @jamesmunns), Andrew Gallant (@BurntSushi), Marco Ieni (@marcoieni), and many others for their feedback and assistance while working on this. My thanks does not necessarily imply their endorsement.

Rendered

[Manishearth]

implementation effort is ongoing. I don't think this ideal is difficult at all.

I'd argue that rustlang::foobar (or rust_esl::foobar) for ESL crates is a pretty good use of that feature, and is a good place to put funding if things need to be expedited.

I don't see another easy way to have this be clear from the names in a way that's not easily spoofed.

[bdaehlie]

If we could pull it off that would be ideal. It's clearly better. But it depends on namespacing being implemented and having the relevant maintainers on board with renames.

[Manishearth]

It seems like renames are necessary here anyway? Maybe I missed something but I think a very important facet that isn't being really discussed in depth this RFC is how ESL membership is signified.

[bdaehlie]

Right now the RFC does not require renaming crates, only moving them to the ESL GitHub organization:

"As such, pre-existing crates that join the ESL will keep their names unless there is a compelling reason to make a change" ... "Crate users will have to confirm ESL membership by checking the GitHub organization."

The ability to confirm membership is important. Originally my draft RFC did require RFC 3243 implementation and renames into an ESL namespace in order to provide clarity about functionality and membership. Concerns were raised about the difficulty of renaming existing popular crates, so this compromise was made. If the community thought renaming crates into an ESL namespace was feasible that would be a better outcome and I'd be happy to update the RFC as such. I'd want to hear from potential ESL crate maintainers first though.

[bdaehlie]

Also, I don't know if you got there yet, but the Naming and Namespaces section of "Future Possibilities" further discusses crate naming and indicating membership.

[bdaehlie]

I agree that it would make the result stronger and I would be more than happy to reintroduce the namespace requirement but I'd love to know beforehand how some potential ESL crate maintainers feel about it. We can't do much if the potential ESL crate maintainers are not on board, which is why I erred towards easing the requirements on them.

The concern pointed out to me about renames was not so much about literally renaming or redirecting crates, but about the volume of extant documentation, both formal and information (e.g. blog posts, stack overflow, that sort of stuff).

[epage]

This is organizational namespacing which goes against the spirit of that RFC.

I was going to raise a separate topic that naming should be out of scope.

[Manishearth]

@epage Yeah though if I recall correctly I wasn't against the RFC being used for org namsepacing, it was just not primarily designed for that.

There's a fuzzy boundary there: one could easily say that everything is just part of a "extended stdlib" metapackage.

I agree that it would make the result stronger and I would be more than happy to reintroduce the namespace requirement but I'd love to know beforehand how some potential ESL crate maintainers feel about it. We can't do much if the potential ESL crate maintainers are not on board, which is why I erred towards easing the requirements on them.

My point here is less about reintroducing the namespace requirement and more about bringing more focus onto how we make this trusted relationship clear to users. There are many mechanisms to do so, namespacing is one of them.

[epage]

Yeah though if I recall correctly I wasn't against the RFC being used for org namsepacing, it was just not primarily designed for that.

There's a fuzzy boundary there: one could easily say that everything is just part of a "extended stdlib" metapackage.

There are trade offs to using this outside of its intended purpose. Our example of doing so will be noticed much more than any caveats we put in the documentation around this feature.

[kornelski]

Namespacing is only a proxy for, and only one of many possible ways of helping users discover and identify the standard crates.

For example, crates.io could highlight ESL crates in search results, and add clearly visible indicators on crate pages. This can be done right away, without being blocked on having namespaces implemented at the protocol/cargo/rustc level.

Tools like cargo-vet don't really need namespaces integrated with the language either. To identify and trust the ESL crates it only needs an audit.toml list hosted somewhere official.

Besides, the hard parts here are bringing these crates up to the expected level of supply chain security, and solving the social problems of moving crates from being BDFL-owned to a shared org.

[ChrisDenton]

I feel like a lot of the "Tranche 1" functionality should be in the standard library, at least in some form. Maybe not doing everything that the relevant crates do but at solve 90% of use cases or so.

[JohnDowson]

Some, perhaps, even a language feature. Bitfields probably being the least controversial candidate.

[ssokolow]

I feel like a lot of the "Tranche 1" functionality should be in the standard library, at least in some form. Maybe not doing everything that the relevant crates do but at solve 90% of use cases or so.

As someone who's been here since about 2013, it's been my observation that Tranche 1 stuff that isn't in the standard library had very good reason to not be in the standard library back then, and the question is whether they've managed to stabilize their APIs enough in the intervening 12 years for that to no longer be an issue.

I can literally name, off the top of my head, at least one (sometimes two or more) formerly-leading implementations of every item on the latter half of that list which were, at the time, the leading solution:

• Random number generation (rand has gone through multiple API breaks)
• Serialization/deserialization (Serde supplanted rustc_serialize, which was once the leading solution)
• Data structure serialization (Same thing. We'd have been stuck with rustc_serialize if we moved earlier.)
• Date/time support (We're still seeing churn in this space, with time, chrono, and the name eludes me, but I recently saw a third one start to crop up in my dependency bumps.)
  Error handling support (error-chain to failure, failure to anyhow and thiserror, with eyre around 1/7th of anyhow's popularity and climbing... and that doesn't count the ecosystem's "Yeah, no." reaction to a team member's idea of what a good error-handling API should look like in fehler.)

It may be slow, but things do make it into std in good time, as demonstrated by the incorporation of a variation on once-cell as it became apparent that it was both the superior API and slowly but surely winning out over lazy-static.

...but, at the same time, despite the team's best efforts, things like the two deprecated methods on the Error trait do still exist.

I will admit that I haven't noticed much churn in the top half of the list, but those are also niches I don't pay as much attention to.

[Sky2042]

We're still seeing churn in this space, with time, chrono, and the name eludes me, but I recently saw a third one start to crop up in my dependency bumps.

Likely jiff by burntsushi.

[ssokolow]

We're still seeing churn in this space, with time, chrono, and the name eludes me, but I recently saw a third one start to crop up in my dependency bumps.

Likely jiff by burntsushi.

Ahh, yeah. That sounds/looks right.

I tend to have burntsushi stuff that I'm not already in the habit of using slip my mind because using crates.io is just so much less comfortable than using lib.rs.

[jhpratt]

Even if the current crate maintainers were fine with it

I will state for the record that I am not. It is my understanding that, among time, chrono, and jiff, functionality (including via third party extension crates) is substantially similar and that the crate maintainers have acknowledged this. There is no "one size fits all" approach to date-times handling; my recent attempt to create a baseline level of interoperability didn't go very far.

[jhpratt]

And that's not to mention requiring separate reviews for code. If that were required, time would get approximately zero done. As far as I'm aware no one has reviewed the code I write thoroughly since I initially took the crate over in 2019.

[BurntSushi]

I will state for the record that I am not. It is my understanding that, among time, chrono, and jiff, functionality (including via third party extension crates) is substantially similar and that the crate maintainers have acknowledged this.

To be clear, I have acknowledged that this is the case for chrono and time. But not for Jiff. :-) I do not think that jiff is substantially similar to chrono or time in terms of its capabilities. Happy to chat more about that elsewhere though if you want to dig into it.

[jhpratt]

Sure. Admittedly my point was more that there's no clear, unambiguous "winner".

[kornelski]

The biggest advantage of having an extended "standard library" as separate crates, as opposed to a monolithic language built-in, is that the preferred crates can come and go without breaking anyone. The official blessing is still chilling for competition, but doesn't prevent it. For example, crossbeam-channel won over the original std::mpsc (and crossbeam-channel still has a better API despite std literally using it internally, because std is cursed by lack of versioning).

If ESL blessed error-chain as a crate, it could deprecate it and bless anyhow later. OTOH if the magic hardcoded unversioned std merged in error-chain into the std namespace, we'd be stuck with it forever. It would keep being a burden for Rust, since all current and future implementations would be forced to keep maintaining it for backwards compatibility.

Normal crates are versioned separately from std, which is a massive advantage. An ESL-blessed rand can keep evolving, and wouldn't be hamstrung by being in the std. OTOH if a rand implementation had been frozen in the shared std namespace, it couldn't redesign its API (having std::rand2, std::rand3...std::rand9 wouldn't be acceptable).

All of the upsides of being in std, like trust and discoverability, can be achieved without paying the never-ending cost of actually merging code into the technically-problematic std crate. Moving code into the std crate means having a huge responsibility of preserving backwards-compatibility for the Rust language. This severely limits evolution of the API and its implementation, leading to sub-par APIs and worse implementations long term.

Being in std forces supporting the same APIs on all Rust-supported platforms, which has already been problematic. The std::fs is completely useless for browser WASM, even in browsers that have a filesystem API! OTOH it's still too much of a compromised lowest common denominator, and lack of handles for directories makes it vulnerable to TOCTOU issues. std::fs should have been a crate!

std has advantages of the official blessing and convenience of being bundled with the Rust distribution, but these advantages were given to the std crate, they don't come from code being literally in the std crate. These advantages can be given to other crates, without the compromises, risks, and burden of dumping all code into a magic one-size-fits-all unversioned monolith.

[Lokathor]

Do you have any numbers or other indications to demonstrate that the reliability of software has been hurt by the status quo? I simply don't accept the basic claim that decentralized code is bad and that we can solve things via centralizing. If anything, the rust project is one giant indication that centralization is a quick way to bury a small amount of people under a massive amount of work, until it all slows to a crawl because too many things are looked at by too few eyes.

[JohnDowson]

But is this RFC about centralisation of development, or is it about

1. ) giving users a seal of quality for some third party crates
2. ) providing funding to the maintainers of such approved crates

To me it reads like it's more of the latter.

[Lokathor]

The entire thing is about moving approved crates into an approved github org so that only approved people can access any of it. The goal is a stamp of approval, the mechanism is centralization.

[bdaehlie]

The maintainers of the crates that move into the ESL would continue to be the maintainers. Anyone could contribute to the crates with PRs, same as before.

The GitHub org would enforce some best practices, particularly with regards to security, and additional resources would be available to help with maintenance. I think this is important for the ecosystem's most commonly used crates. Having them differ in terms of compliance with best practices, security policy and resources, and implementation of infrastructure and controls across many different GitHub orgs is not a good thing for Rust security, and it's probably more work in aggregate.

[Lokathor]

So everything is the same, except when it's not the same? That seems incoherent.

I don't ask all this totally out of nothing. tinyvec is a crate that's available in the rust playground. I take this fact as a burden to be extremely careful of. How, concretely, does me moving my repo from my GitHub acocunt to your theoretical org meaningfully change anything for the better? Can you point to actual security situations that have occurred with rust that would have been solved if this RFC had been implemented before the event?

[the8472]

I meant this one: rust-lang/crates.io#11131
It's basically outsourcing the process management of creating a trusted release to github (and other trusted publishing platforms). Then the org can limit publishing to bots instead of humans and the bots will only publish after the appropriate reviews.

What imo would still be missing then is to compare what's on git and in the tarball so that nothing surprising snuck in due to a subverted CI process.

[epage]

Slippery slopes...

"Trusted Publishing"s name isn't meant to say other forms of publishing aren't trusted but to secure specifically focusing on the CI publishing process. The misleading name already caught on though.

There are trade offs with the current state of bot-based publishing such that we should not require only that. Without requiring bot-based publishing, you then need to allow direct pushes unless you make the release process onerous.

[matthieu-m]

What imo would still be missing then is to compare what's on git and in the tarball so that nothing surprising snuck in due to a subverted CI process.

I personally would wish for quorums, here.

Automate the publication to crates.io, certainly, but then leave the crate in quarantine mode until other maintainers or dedicated auditors have vetted the published version.

Even small quorums (1 single independent auditor) would massively raise the bar compared to the statu quo.

(On projects where a single maintainer could also trigger a release by a bot, which is likely, you'd want a minimum quorum of 2, obviously)

[the8472]

A) a trusted publishing process may already have a quorum on the CI side (github deployment environments can require multiple approvers), so reimplementing the same functionality on cratesio seems redundant.

B) If we're involving independent authors this seems like out of the hands of the maintainers, in that case "quarantine" would be the wrong framing imo. I think users being able to filter crates on cargo add and cargo update by whatever security policies they want to apply would be more appropriate than making it a global decision.
AIUI big orgs already run their own crate mirrors where they filter which crates can be pulled.

[epage]

Depending on how its done, quarantining packages could cause major problems for workspace maintainers. It would help if we had a whole new publish API but that has been talked about for years. Even once you have that, there can be reason to publish a workspace in fragments.

[clarfonthey]

I don't think that this RFC is a good idea for a couple reasons.

The first is, quite frankly, I don't believe that this project will actually improve the situation, but in fact, make it worse. I do not believe that the Rust Foundation will be able to secure more funding to develop the necessary crates than these crates will ultimately be able to get on their own, which I'll get into later. I also believe that by "spreading thin," changes to these crates will ultimately be reviewed less often and lest thoroughly, meaning that it only increases the risk of "supply chain" issues it purports to prevent.

The second is that, flat out, this does not seem to understand the primary reasons why "supply chain" attacks happen, and thus will fail catastrophically to prevent them. The "secret sauce" involves some combination of the below ingredients:

• Widespread use
• High demand for changes
• A small number of maintainers
• A large amount of work per maintainer

Rust mitigates some of these issues, but not really. While unsafe code is heavily scrutinised, a large portion of the Rust ecosystem still relies heavily on external code without such promises. Even the Rust compiler itself, despite strongly working toward "oxidising" its internal dependencies (Rustup recently moved to being RusTLS only, for example), it still relies on massive unsafe code bases like LLVM and Git. While the number of these will decrease over time, I don't expect this to ever fully go away; even if all of the code Rust itself uses is written in Rust, it can still call down to the OS which may not be. And this doesn't even begin to approach exploits which don't even touch unsafe code.

It also particularly points out the XZ vulnerability, which I think is a perfect example that could happen in the Rust ecosystem right now, given the right circumstances. Right now, we have a small number of maintainers on each team which are expected to sign off on a very large number of changes created by anyone, and the Rust foundation has approximately 12 employees at the time of writing, with fewer people actually managing changes to the code itself.

But don't worry! The power of Rust is that many corporations find it in their best interest to hire people full-time as maintainers too!

Do you trust Microsoft, or Amazon, or Google to always hold the community's best interests at heart and not once, never, ever potentially exploit it? Do you think that this does not put them in an excellent position to leverage their community trust for exploiting any number of things, ever, in perpetuity? Do you trust every person in a position like this to properly back out and tell the project what's going on instead of exploiting their position like their employer wants?

Ultimately, there isn't a great solution for "supply chain" attacks that the tech industry wants, because its misdoings are the reason why such attacks exist in the first place. Moving crates into central control and telling people that the foundation has solved the supply chain problem, however, is actively harmful, because it hasn't solved the problem and can't.

I also would love to see more of these crates become supported and maintained by the entire community, but that's not going to happen by just waving a bureaucratic wand. And I think that pretending that the problem is solved will only make Rust a ripe target for these attacks.

[hawkw]

The drawbacks elaborated in this section are likely to be significant challenges facing such an effort. I feel the current discussion of these difficulties furnished here doesn't sufficiently discuss how these challenges will be handled. In particular, I have to say that simply stating "we will make the difficult decisions" and "we will reach consensus while keeping analysis paralysis at bay" does not inspire confidence in the ESL effort. These are serious issues, and the ways in which they will be addressed is, in my opinion, the most important part of a serious proposal to establish something like the ESL. Similar issues of community conflict and opaque decisionmaking processes have caused substantial problems for the Rust community in the past, and a proposal to establish "blessed" crates that doesn't anticipate such issues and propose a detailed plan for how they will be addressed seems likely to cause more conflict than it resolves.

I think these issues deserve substantially more discussion and forethought than is currently presented in this RFC. They cannot just be handwaved away by saying "we will make the right decisions".

[bdaehlie]

This text is only pointing out that difficult decisions will need to be made in order for the ESL to succeed. The Rust Project's governance already has decision making processes in place, and I don't think this proposal is the the right place to try to change those, or to repeat what they are. If those processes can successfully make difficult decisions that's great, if they cannot then it will impact the ESL's ability to move forward.

[Manishearth]

It's worth noting that previous such attempts for a blessed "extended stdlib" died on similar rocks. We have had tremendous conflict in the past when it comes to Rust "picking winners", even with situations where there's just the perception of Rust picking a winner. Folks have quit the community due to this type of thing.

There's some number of libraries where Rust chose to tip the scales (libc, regex, etc) by making them Official Libs Team projects, and that's mostly cases where the ecosystem mostly settled on a very clear "winner" (like regex), or where it's the type of thing that really should be managed by Rust (libc). Arguably, syn could fall in that category.

There's also a worry of interop: even if you bless regex as a winner, codebases using regex can easily interoperate with codebases using other regex engines because it's just a library you call. This is different from, say, picking a serialization library or async runtime, where the decision can be much more viral.

(syn is an interesting example because it's a massive API surface which tends to be a problem for ecosystem interop, but on the other hand 99% of syn users use it as an internal dependency, so it ends up mostly being okay. mostly)

I think it's possible to come up with a set of principles for the types of crates where it is acceptable to pick winners. I think it'll take significant effort, but it's doable. I think that to be a prerequisite for such an RFC; otherwise we risk significant drama and hurt feelings in the future with bad decisions.

At the end of the day, making a decision is important even if it is not "perfect" by a given set of standards.

I strongly disagree with this. It is not important to uplift a library for any given function. It's a nice thing to have. The downsides here are tremendous, and their associated costs are measured in people.

[hawkw]

This text is only pointing out that difficult decisions will need to be made in order for the ESL to succeed. The Rust Project's governance already has decision making processes in place, and I don't think this proposal is the the right place to try to change those, or to repeat what they are. If those processes can successfully make difficult decisions that's great, if they cannot then it will impact the ESL's ability to move forward.

If the intent is to use existing decisionmaking processes within the Rust foundation here, I think it would be worthwhile to explicitly state that, at the very least. Ideally there would be some discussion of how these processes apply to the decisions necessary to establish the ESL.

Fundamentally though, I agree strongly with the points in @Manishearth's comment. What you are proposing is something that has the potential to cause a great deal of conflict in the community, and previous efforts to do similar things have failed largely due to that conflict. How are you going to solve those problems this time?

If the goal is to allow users to confidently depend on the crates in the ESL, then it is necessary to establish community confidence in the process through which those crates are selected and through which technical decisions are made in the course of their maintenance. What you want to do here requires getting both maintainers and users of crates on board with the proposal. I think that the right time to sow the seeds of such a sense of confidence is now. Failure to convince the community that the processes used to make these decisions are fair and robust means that the best-case outcome for the ESL is a future where most users prefer to continue depending on independently-maintained crates due to higher levels of confidence in their decision-making processes than those of the crates in the ESL.

[Manishearth]

If those processes can successfully make difficult decisions that's great, if they cannot then it will impact the ESL's ability to move forward.

Those processes have historically been unable to successfully make difficult decisions around elevating ecosystem projects. Really, anything involving people and feelings over code is super tricky and is not what the Rust decisionmaking process is designed for.

It's understandably not obvious from the history, but I'd say figuring this out is the core prerequisite for any such proposal.

[hawkw]

For the record, I want to make it clear that I am not expressing opposition to the idea of having something like the ESL. Instead, I'm pointing out that there is a reason that there are so many previous attempts to do something like this — if it had worked in the past, we wouldn't have to be proposing it now. If this attempt is to succeed, it must address the issues that have led to the failures of past attempts. Perhaps it will do so successfully, but the text of this RFC doesn't give me a reason to believe that.

[the8472]

Specifically picking an async runtime is likely to be highly controversial. Interop already is difficult and yet there are multiple competing runtimes, some of them specialized. E.g. embassy focuses on embedded and wouldn't be suitable for server applications.

[epage]

Async, cli parsers, and many other cases involve trade offs and selecting which trade offs to bless will be difficult.

Some ideas

• Bless more than one, like blessed.rs sometimes does
• Bless the most minimal one (std-like)
• Bless the most flexible (full batteries)

[ssokolow]

Specifically picking an async runtime is likely to be highly controversial. Interop already is difficult and yet there are multiple competing runtimes, some of them specialized. E.g. embassy focuses on embedded and wouldn't be suitable for server applications.

Honestly, I think it's premature to bless any async runtime before the "tell a complete async story" process is far enough along to start abstracting the APIs that tie projects to specific runtimes and adding those to the standard library as traits.

It wasn't that long ago that we got the "async fn in trait" MVP.

[the8472]

We have git as a tool to share code. We have github as a community platform to make the code-sharing and communication around it public.
We have cargo vet (and crev) as tools. What we're lacking is a visible platform for aggregating reviews, the social dynamics around 3rd-party code reviews. It's not a publishable good.

So I agree it doesn't go far enough. But I think rather than centralizing a few critical pieces a more scalable solution would be to record the figurative eyeballs that went over all the code, and the lack of eyeballs too.

Netflix, Huawei and the sovereign tech fund assigned engineers to review this code is useful information.
10000 people use this library but nobody ever reviewed it is useful information too.

If everyone (and every company) reviews in silence we repeat efforts and don't know who else already did the thing. It feels very much like a coordination problem. Nobody reviews a thing because reviewing everything is too much work.

This could start with surfacing such information on crates.io.
And "review crate of the day" kinds of efforts, make it a vscode plugin, and...

Otherwise I think the issue will be that any hypothetical supply-chain attacker will move their focus to "widespread use, but just outside the ESL scope" crates, which will remain unreviewed...

[epage]

Would love to integrate cargo vet like features into the cargo/crates.io workflow.

There is that and other tacticts which we should work towards to improve supply chain management. I worry this might distract us from that larger problem because this can't cover everyone's needs but people will likely be optimistic about this effort

[Manishearth]

What we're lacking is a visible platform for aggregating reviews

n.b. libs.rs does show vet. as does deps.dev.

My comment below suggests a way we could make great use of cargo vet for attaining the goals in this RFC.

Would love to integrate cargo vet like features into the cargo/crates.io workflow.

Yes, 100%.

[Manishearth]

I think this might be selling cargo vet short. The design is not for one to trust random audits; it was thus designed precisely because trust is multidimensional and contextual. It is a framework upon which to build a trust model. cargo-vet doesn't care about a project's security policies because cargo vet shouldn't be making that decision: instead, it lets one define their own criteria for what constitutes "good security policies" and filter based on that.

A very feasible way of achieving many (not all) of the goals of this RFC would be to define a set of criteria, and have some blessed audit repository for those criteria¹. It could be automated such that crates whose maintainers are trusted to follow these criteria automatically get audits published alongside their crate publishes. Sites like lib.rs could be petitioned to elevate such audits with a badge.

"Trust the entity that produced the code in the first place" is definitely a possible thing to represent within cargo-vet's framework.

(Potentially crates.io as well, though that's the threshold where you would need an RFC, up to this point this can be done entirely as a community effort).

Footnotes

1. audits retain their source: it is possible to distinguish a Google audit from a Mozilla audit from a rustlang audit. ↩

[epage]

This should be out of scope. Renames are a breaking change and requiring one to add/remove branding is a bad idea.

[Manishearth]

In theory cargo/crates.io could support "redirect" renames. It's a thing that was briefly discussed during the namespacing RFC, it didn't end up happening, but it's ... something that could be proposed.

I agree that without such a change it's probably not going to work.

[epage]

We should not count on being able to do so though as I expect some tricky problems in the resolver and Cargo's ideals around ‘Cargo.lock` files.

[epage]

How many projects, even fundamental ones, have the maintainer capacity to do code reviews for maintainers?

I understand this list is meant to give an idea and not specify exact policies but I worry this list is either assuming more than can be expected or will give people a false sense of expectations for the result.

[epage]

(I have similar concerns about several others of these ideals)

[the8472]

I would even go a bit further, I think calling these requirements to be trustworthy is corporate-biased and mildly offensive to individual maintainers.
Saying that you're not trustworthy because you can't guarantee to maintain something well into the future or because you don't want to deal with the security stress and just want soundness bugs treated like any other bug doesn't make a dependency un-trustworthy. It just might make it fall below the level of unpaid support that some corporations might want.

But well

THE SOFTWARE IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.

To me trustworthiness in something that's provided for free only means non-malicious and not having a track-record of bad decisions. Everything else sounds more like paid support or nice-to-haves.

Some things don't require eternal vigilance, e.g. hex hasn't seen a release in 4 years and imo this doesn't impinge on its trustworthiness, it might eventually become a problem for its usefulness.

[jieyouxu]

How many projects, even fundamental ones, have the maintainer capacity to do code reviews for maintainers?

I understand this list is meant to give an idea and not specify exact policies but I worry this list is either assuming more than can be expected or will give people a false sense of expectations for the result.

To add to this, even the compiler, standard library, and various rust-lang/* crates like cc or libc have limited reviewer bandwidth, especially in specialized areas of the codebases.

• Has sufficient maintainer bandwidth to publish security updates in a reasonable timeframe
• Has reasonable code review and testing standards

Are really, really hard problems. I fully echo the

assuming more than can be expected or will give people a false sense of expectations for the result

concern.

[jswrenn]

How would removal work in practice? If the name of the crate is what conveys its membership in the ESL, what happens when that crate is removed from the ESL? Are its versions yanked?

[jswrenn]

How will the ESL ensure that existing maintainers are non-malicious? How will the ESL ensure that new maintainers of ESL crates are non-malicious?

[frewsxcv]

And similarly, who decides whom is qualified?

[BurntSushi]

I feel like this is largely a non-issue. Who decides who is on libs? Or the lang team? How do we know everyone on the teams is not malicious? How do we know they are qualified?

[frewsxcv]

If it's not important and up for debate, then what is this line for?

[BurntSushi]

It seems like it's elaborating on an important goal to me.

[epage]

This is a real tension where I have to decide between

• Being blessed or fading into obscurity
• The policies if they are too heavy

[jswrenn]

The drawback of picking winners isn't that folks "could be upset", it's the (good) reasons for which they would be made upset. If you maintain a crate whose competitor is selected for inclusion in the ESL, there will begin to be tremendous pressure to avoid your crate. If you have an idea for an alternative approach to a problem "solved" by the ESL, there will be tremendous pressure to avoid your crate. The ESL has the potential to stifle innovation in the crate ecosystem.

Had the ESL existed in 2018, I don't think I'd be paying my bills by maintaining one of the crates I maintain today; there simply would have been too much pressure to use the then-dominant alternative.

[BurntSushi]

I somewhat feel like this is already true today. I think the fundamental difference between the status quo and this proposal is that this proposal has a (presumably small) group of people making a determination about the winner as compared today where a winner tends to manifest via inertia or other pressures. But we certainly don't need an ESL to have the problem of it being almost impossible to unseat an existing dominant solution.

[Lokathor]

But I think we can definitely avoid making it worse at the least

[hawkw]

There's also the issue that, in many cases, there may be multiple crates for a given purpose which make very different design decisions. Often, these design decisions are equally "correct", and the right choice depends on the particular use case. In a situation like this where two or more crates that have a comparable level of maintenance quality make substantially different and incompatible design choices, what would the ESL do? Would both alternatives be added to the ESL? Or would the ESL just choose one such crate and leave users whose particular use case prefers the other design without an ESL-blessed option?

[fintelia]

Also worth calling out that choosing only one competitor means that the other ESL crates are forced to make the same pick. Basically...

We'd love for your crate to be part of ESL, but of course you'd have to switch from fastrand to rand. I'm sure your users won't mind!

[hawkw]

Here's a fun question that it would be nice to have an answer to: what happens if an ESL crate depends on a non-ESL crate? Now, you have a crate that's part of the trusted base of vetted libraries which, nonetheless, has the potential to "get xz'd" by its non-ESL dependency. Would ESL crates be forbidden from depending on non-ESL code? Would adopting a crate into the ESL require ESLing its entire dependency tree?

[epage]

I worry this will be too restrictive. I'm splitting up toml. If it was in the ESL, I'd have to get those lower level crates to be blessed to a similar level which also unduely raises their visibility withinea blessed list.

I worry this can also be stagnating with other dependencies.

[the8472]

we already have a policy distinguishing between intentional artifacts vs. internal use libs.
https://forge.rust-lang.org/policies/crate-ownership.html

[epage]

Another example: I'm working on a replacement for libtest. It has its own cli parser and soon a json writer to meet very specific design requirements. Abandoning those requirements would be rough blessing each part would also.

In both examples, some of the packages are also being developed on an experimental basis. And the libtest project just left rust-lang org because of how strict the requirements are, even for experiments.

[Lokathor]

It's unfortunately demonstrably true that, until we have multi-threaded rustc, it's faster for builds to split crates into multiple sub-crates as often as possible. We probably want to encourage large crates used by many people to break up their code as much as possible, rather than encouraging single large crates.

[epage]

we already have a policy distinguishing between intentional artifacts vs. internal use libs.

The Project has that but this is separate. That policy is also very strict in that something can only be Intentionally provided by the Project or its solely Internal to the project. Some of the cases I'm referring to would be Intention for the ESL or non-blessed within the ESL but still Intentionally for end-users.

[epage]

Have you talked with them for what lessons they have just doing a fraction of what this RFC does?

[oxalica]

Is there a possibility of blessing blessed.rs itself, making it into a semi-or-fully-official list? I think it's another alternative solution worth mentioning.

This would avoid the cost of maintaining crates themselves, bless multiple crates for different design goals. Newborn crates competing existing crates (with new design, new goals, new algorithm or etc) can also be less painful (as mentioned by @jswrenn https://github.com/rust-lang/rfcs/pull/3810/files#r2080592616) and they can be added to (not replace!) the list later if some conditions are met.

[jieyouxu]

Is there a possibility of blessing blessed.rs itself, making it into a semi-or-fully-official list? I think it's another alternative solution worth mentioning.

I don't see how this makes sense -- what does a "officially maintained but non-official blessed list of crates" even mean to end users?

[teor2345]

In practice it's a suggestion to end users, like we suggest tokio or various other crates in official documentation.

[epage]

As noted in https://github.com/rust-lang/rfcs/pull/3810/files#r2080548607 this runs counter to that RFC as that is about a cohesive API while this is about an organization.

[epage]

What lessons can we learn from these prior art?

[erickt]

Should this RFC discuss if it will be possible to remove crates from the ESL, or would we allow for duplicate functionally? I recall stdx was hoping to support removing crates, but I imagine it could be very disruptive to the community.

[epage]

This would be longer support than we off Rust itself

(yes, this is an example)

[comex]

Not really. The text is talking about how long to support an old SemVer-major version after releasing a new one. Rust itself never has SemVer-major updates, so the issue never arises. Theoretically at least.

[epage]

While this is an example, it highlights the assumed maturity for ESL.

Is there a place for younger crates that fill a core need within ESL? I'm again thinking of my libtest work once its further along.

[ssokolow]

...and what role does rust-lang-nursery play in this vision? I'm not sure it's even easy for people to learn what membership in it means now.

[epage]

Not to debate policy but just so I don't forget, there might want to be policies on this. In most cases, there is little difference between minor and update. When supporting for 2 years, there are major differences

• The more majors, the more work to backport, patch, and release
• Without majors, you can't patch. MSRV bumps and "risky" changes (major refactors) should necesisitate a minor

[Lokathor]

I think we should not accept pre-1.0 crates into the ESL. I'm not even joking.

[riking]

Being on a 0.x.x version hampers so much of your ability to perform semver updates; imposing a technical requirement for a non-0.x version does seem justified.

[epage]

Not to debate policy but just so I don't forget, consider the MSRV applying to the package and not every version.

What I mean is that a previous but supported minor satisfying the MSRV should count, even if the latest version doesn't. Maintainers will have flexibility to bump MSRV while having an incentive to not do it too often because it means more versions to support, creating more work, see https://github.com/rust-lang/rfcs/pull/3810/files#r2080681065

[epage]

Similarly about not wanting to forget, consider whether "N versions backs" actually meets people's needs. That means the MSRV can change every 6 weeks. Those with a direct MSRV requirement usually need predictable, stable stuff. Calendar based approaches might work better.

[Lokathor]

If were legislating MSRV we'd need to actually pick a policy on cargo features affecting MSRV, which means the cargo resolver should support that logic as well. Not that we can't do any of that, but it's complicated

[epage]

From where I am currently out, I see per-feature MSRVs to be highly unlikely.

[Lokathor]

If that can't be supported in some way, then "two years of msrv" is effectively a joke proposal.

[Lokathor]

epage was just taking notes on what the ESL org would have to cover eventually, I'm mostly just adding to the notes pile.

and i know that RFCs aren't always implemented exactly as first written, but if there's not really a remotely realistic way to implement what the RFC says, then the RFC needs to say something else. The current RFC text currently states a pretty specific policy.

[BurntSushi]

Okay.

[epage]

Along the lines of MSRVs is the conflict between long MSRVs and build times which is also another strong care about. I was analyzing someones dependencies recently and there were a lot of deps that exist solely for MSRV. Some could be handled through polyfills while others will require cfg(version) / cfg(accessible) to be stabilized. This is another reason to consider MSRV to be about the project as a whole and not apply to every major or minor version.

[nicoburns]

no_std support is another major source of otherwise unnecessary dependencies. Many projects unconditionally depend on libraries like libm and hashbrown because there is no ergonomic way to specify dependencies that are only enabled when not(feature = "std"). So you either have to add something like a no_std or libm feature (which is inconvenient for consumers of your crate) or unconditionally depend on the necessary crates.

[epage]

I wonder what all cases there are like this and if there are some high leverage changes that could be made to avoid this, like getting HashMap into alloc with a no-std hasher available.

[epage]

Not to debate policy but just so I don't forget, we should consider a little more nuance here.

Problems with releasing new major versions for why they should be throttled / avoided

• Work to upgrade
• Interop when in a public API of another package
• Compile times from multiple copies

Not everything is big and not everything is meant to be in a public API. As for being disruptive, we should probably encourage (but not require) breaking releases to be removal of deprecated functionality as discussed on matklad's blog and as I try to do with clap and winnow as much as possible.

[Lokathor]

I don't think I understand what you're trying to get at here. When you say "major" do you mean the first number in the three number sequence? cargo's vision of versions is to treat the first non-zero version as the "breaking change" number, and the next numbers as compatible, right? So how is 0.x.y somehow better than x.y.z? An incompatible version is just incompatible.

[epage]

The topic I'm replying to and my comment and "major" to refer to the relative position in the version number and not the absolute position (wish we had established names for these)

My comment is speaking to how to manage breaking releases.

[epage]

I think I saw where some of the confusion came from and made an edit that I hope helps.

[epage]

What about non-malicious but unfavored changes?

This can range from

• A maintainer making the API more non-ergonic for purity purposes (highly debateable on whether this is a problem)
• A maintainer shipping pre-built binaries (we saw how that went)

Flipping the perspective on this, I know with clap I've appreciated having WG-CLI as a sounding board for things I thought might be a good idea but I was unsure how they would be received.

[ssokolow]

*nod* APIs becoming less ergonomic is definitely a concern.

With clap's current API, I find it to be so heavily abstracted and the documentation for processing OsString paths (the only reason I choose it over gumdrop's offering of a derive API with faster build times as I do in cases where I can just say "non-UTF-8 paths not supported" like actix-web www roots and config file paths) so opaque last time I looked at it that I've built a habit of just rg-ing my way to one of my existing projects which contains a suitable OsString-processing pipeline and copy-pasting the relevant snippet to the new project without even trying to remember why it works anymore.

Clap 3's "vendored StructOpt" API? Comprehensible. Clap 4's overindulgence in rustdoc-hostile, rustc-error-message-hostile type-level dispatch obfuscated by weaknesses in the concept of derive macros? Not really.

For me, clap's API has become a bag of incantations with my prime use-case for it achieved through a piece of tribal knowledge that you blessed me with in some comment on the issue tracker that I forgot to bookmark... and I'm one of those people who makes an active effort to understand APIs before I use them and to maintain that understanding. (eg. I "make things harder than necessary" when implementing custom Serde work by trying to save the brute-force "just impl a whole new Serialize" option as a last resort until I've proven to myself that there's no higher-level way you're meant to be able to do something.)

[epage]

With clap's current API, I find it to be so heavily abstracted and the documentation for processing OsString paths

While a bit off topic for this RFC, OsString paths "just work" which was a large reason for the clap v4 design as before where you had to know they weren't supported natively and find the right incantation to get them to work.

[ssokolow]

They "just work(ed)" if you "just want an OsString". Last time I wasn't just copy-pasting a previous successful use, they were a frustrating and under-documented type-system puzzle-box if you wanted to plumb in some custom validation and end up with a PathBuf field on your struct. (But, as soon as you told me the magic incantation, they did indeed "just work".)

With StructOpt and Clap v3, I had no trouble understanding from the docs how to specify that I wanted the argument ingested as an OsString, passed to a validator function, and then ending up in a PathBuf... even if the APIs were suboptimal.

[oxalica]

What about non-malicious but unfavored changes?

There is already an example in the ecosystem,though not quite in this RFC's scope, gluon-lang/lsp-types#284. Some downstream users are so unfavor the changes and start creating forks. I think it's a lose-lose for the ecosystem. The problem will be worse if the crate in question is in ESL.

[Kijewski]

Maybe you could indent the content under a bullet point to make it more readble?

1. **Title**

   Description

   More description

2. **Next point**

   It has a description, too.

1. Title

   Description

   More description

2. Next point

   It has a description, too.

[nicoburns]

I'm glad to see someone trying to address the problem of supply chain attacks. I think it's an important issue which is currently very difficult to mitigate against.

However I do not think that the presented solution of centralising maintenance of key crates is a good one. I agree with @clarfonthey that the constraint which creates the opportunity for supply chain attacks is lack of available maintainer resources, and that this proposal is likely to reduce the availability of such resources not increase them.

I think the philosophy behind Cargo Vet where vetting/auditing is decoupled from maintaining/owning is the right one, and if there is funding available, I would love to see this model expanded via things like:

• An official, well-maintained version of diff.rs. This website allows people to 1. easily read the code actually published to crates.io (not just the code checked into git which may be entirely different) and 2. see a diff between versions of that published code. To me, these are foundational capabilities for enabling audits.
• Vetting/auditing functionality built into this "diff.rs" UI (e.g. something similar to Github review UI) such that people can easily create audits/reviews via the UI. These would then be saved to a central database.
• Metadata attached to audits with things like standardised categories which denote the depth of review that has been undertaken and the reviewer's confidence level in their review.
• Tooling built-in to cargo which allows users to specify criteria such an "N reviewers (with an optional allowlist of specific reviewers or organisations) must have approved" and/or "there must be less than N negative audits" (perhaps with an additional "warn" outcome that prompts the user to go and read the audits and/or audit the code themselves).
• Pushing crate authors towards MFA
• Expanded support for (emphasis on) signed crate releases (e.g. support for allowlisting certain crates only if new releases are signed by a trusted signing key and prompting for manual verification if a new key is used)
• Directly funding trusted people to spend their time producing audits

Trusting specific crates/publishers is all well and good, but the supply chain attacks I would be most concerned about are when crate maintenance changes hands or a users credentials are compromised. To mitigate against these kind of attacks one really needs each published version to be verified, not just the crate or it's publishers. Given how many users of the really key crates there are, it should be easy to get enough reviews if making one is easy enough.

I think lists of approved crates/authors/auditors/organisations would be useful. But I see no reason why there needs to be a single list. We could perhaps have an "official" rust foundation list that is quite conservative. But also allow other organisations to publish lists which people might decide to either trust wholesale and pick and choose pieces of to form their own lists (e.g. corporations like Mozilla, Google, Amazon, Dropbox, etc; open source collectives like Servo, Linebender, Bevy, etc; and even individuals that may be trusted by the community).

[abgros]

Will the extended standard library be automatically downloaded by Rustup?

[arielb1]

Moving crates to a central github org is pointless and creates a single point of failure.

Giving maintainers of popular crates the option to put their code in a Github repository with certifiably high security standards and coupling the code on git to the code on crates.io somehow (while letting the maintainer have complete control over the code except in "actual security breach" situations. I mean, Github owns the git repository and the Rust Foundation owns the crates.io index even today and they might act even today in the case of an actual security breach) rather than having maintainers needing to figure out by themselves how to secure github.com/jsmith123/my-crate, sounds to me like a good enough idea if it can be done in a reasonable way.

Of course, if maintainers already have some sort of git repository that they know how to secure, they might not need this help, and the Rust Project should not have a "let us own your repo OR ELSE you won't get a pretty badge" system.

[joshka]

Giving maintainers of popular crates the option to put their code in a Github repository with certifiably high security standards ...

The core benefit of that is that certification is measurable. It would be significantly more useful to be able to verify that those standards are met. Putting the source into a GitHub organization is not the only way to make that happen.

[Diggsey]

Giving maintainers of popular crates the option

Yeah I'm not opposed to giving the option, as long as it's not a requirement.

[burdges]

Non-malicious is subjective: You can be upstanding and well intentioned, but your employer maybe considered inherently malicious, ala government contractors, blockchain companies, advertising companies. It's likely that some nationalities get treated with suspicion by specific governments too, maybe requiring more ongoing auditing or whatever. It's likely better if these problems get addressed behind closed doors by the worried parties, nd the wider community mostly ignores them.

What might make more sense is if crates put "Integration notes" into their repos an/dor issues on what they've done, or plan to do, to worke better with other crates, or what they wish others would do.

[burdges]

I suppose "non-malicious" could be replaced by simply some disclosure policy: The absolute minimum would be that crates should disclose if their maintainers worked for millitary, intelligence, or law enforcement contractors, since those are the parties most likely to exploit an undisclosed exploit.

Auditor reports would be another standard here, but that gets expensive.

[teor2345]

There needs to be a (more) objective criteria here. Otherwise, people, topics, or groups can simply be declared malicious, even though a dependency on the crates they develop is unlikely to intentionally cause security issues.

This is particularly important for a project that seeks to be the One True Source for the One True Crate for X. You don't want to get caught up in arguments about the value or purpose of crates. Those issues are important to discuss and act on, but they need to be out of scope for this process.

(There are already contentious examples in this RFC discussion.)

[burdges]

As a rule, rust code outside the standard library winds up more "opinionated" than the standard library, like even the getrandom crate in rand, and/or more more featurefull. In particular, the standard library tends more towards a simple baseline targeted at most devs, but it absolutely doesn't care if your niche use case require that you fork one of the stc functions.

Those are natural explorations, but you really do notice a big stylaistic difference between std and other crates.

[TimTheBig]

Suggested change

	`itertools` equivalent

[jswrenn]

[itertools maintainer hat on] We make breaking changes much more frequently than the guidelines of this RFC suggest is appropriate for an ESL crate. Our types don't typically appear in public APIs, so this isn't typically problematic for our users. Itertools has long been informally referred to as part of Rust's extended standard library (and gadgets from itertools are regularly uplifted into the standard library), but it doesn't meet the stability and support criteria outlined by this RFC.

[TimTheBig]

Good point, maybe a stable subset of itertools could be part of the ESL

[jswrenn]

Maybe, but if such a subset exists, then why not make it part of the standard library? Again, elements of itertools are regularly uplifted into the standard library.

[ssokolow]

Good point, maybe a stable subset of itertools could be part of the ESL

Wasn't the point that "the stable subset of itertools is 'and gadgets from itertools are regularly uplifted into the standard library'"?

...as in depending on itertools is an alternative take on cargo +nightly build and things being in it instead of std is an indication that more work is needed before they'll be ready to be that API stable?

Premature stabilization can't be forced. Trying just results in "Python 2.x stdlib has urllib and urllib2 and everyone tells you to ignore them in favour of Requests, which contains a urllib3 that will never become part of stdlib". Induction into a "standard library" that results in successful stabilization is an acknowledgement that stabilization has happened organically, paired with a little pruning away of vestigial organs.

[joshka]

This is a perfect example of why focusing on standards may be (at least in some situations) a better idea than focusing on a standard library.

[TimTheBig]

Fair, it would be great to have stuff like ExactSizeIter in std it would be great for performance. That's why ESL as a step between crates and std. Some kind of robust process to uplift things from ESL to std would be great.

[SOF3]

Would we have less bikeshedding over preferences if we call this "officially endorsed libraries" instead of "extended standard library"? The term ESL gives the following (possibly untruthful) impressions:

• that ESL will be shipped with rustup
• that ESL correlates with msrv
• that ESL is maintained by T-libs or equivalent
• that ESL is an official recommendation over non-ESL libraries
• that the barrier for a v2 in an ESL library is very unlikely

[programmerjake]

Would we have less bikeshedding over preferences if we call this "officially endorsed libraries" instead of "extended standard library"?

• that ESL is an official recommendation over non-ESL libraries

for me, "officially endorsed libraries" gives that impression too, so if you're trying to avoid it, you'll want a different term.

[programmerjake]

maybe "well-known libraries"? that doesn't imply Rust is officially maintaining them, but that Rust is recognizing them as a good option, if they suit your purposes.

or something like "officially verified libraries", kinda inspired by what the blue check was on twitter before musk bought it.

[Lokathor]

If it's just about being well known, then there's already ways to search that information up.

[programmerjake]

If it's just about being well known, then there's already ways to search that information up.

maybe "officially well-known libraries", since I think the idea is that Rust is endorsing that the library is fine to use in your dependencies, but not trying to imply that Rust itself is maintaining it or that you must use it over any other libraries if it doesn't quite do what you want.

[tgross35]

(nit) The use cases for quote and cfg-if are likely to be covered by std at some point in the hopefully-not-too-distant future: rust-lang/rust#115585 rust-lang/rust#54722

[TimTheBig]

I think encoding/file formats should not be included as it's very specific to a use case, with the exception of those needed for http.

[Lokathor]

http is as specific a use case as "open a png file". if you're going to do one esl thing there's no reason not to do both esl things.

not that i think this esl thing is a good idea

[TimTheBig]

I feel that this should be part of lib proc_macro, I have never seen a proc_macro written without it or a library that depends on it. This the most downloaded crate with over 870,000,000 downloads.

[programmerjake]

I disagree that syn should be part of proc_macro specifically because rust's syntax is still changing and you want to be able to have updated AST structures that support the new syntax. this is why there's syn 2.0, and i expect there will be a 3.0 in the next few years or sooner.

[TimTheBig]

not that i think this esl thing is a good idea

This whole thing could be an organization that audits popular crates, maybe trying to reduce their dependeces

[ssokolow]

not that i think this esl thing is a good idea

This whole thing could be an organization that audits popular crates, maybe trying to reduce their dependeces

I could get behind giving https://github.com/rust-secure-code/safety-dance a friend.

[TimTheBig]

not that i think this esl thing is a good idea

This whole thing could be an organization that audits popular crates, maybe trying to reduce their dependeces

I could get behind giving https://github.com/rust-secure-code/safety-dance a friend.

Auditing crates and making some kind of standard process that any can use before a release would be a lot cheaper too.

[teor2345]

There should be an exception here for platforms where the functionality isn't needed or can't be implemented. We might also want exceptions where it would take a large amount of (unfunded) effort to make a crate work.

How would the relationship between Rust tier approvals and the ESL work?
Would approvals be gated on the ESL, or would the ESL have to adapt to new approvals?

[teor2345]

I don't understand why choosing a single crate for each purpose is important here. Or maybe I've misunderstood the RFC, and "which one should be preferred" is much more granular, and "not duplicating" is narrowly defined.

For example, could there be multiple async executors (for different use cases), such as performance, embedded, and ergonomics?
Or multiple CLI parsers with different feature sets?

[teor2345]

I don't see a clear rationale for combining the three motivations of this RFC into a single process/organisation. Either way, it's important to mention the alternative: tackling them each separately.

It feels like this rationale is mainly about motivation 2: the blessing of the One True Dependency for each purpose.
What if that was a separate, narrowly-scoped RFC?
Would we come up with a different set of tradeoffs?
Would there be much connection to ecosystem security at all? Or would it mainly be about discoverability?

Similarly, we could treat motivation 1 as a separate task, codify a set of security policies, assess crates, and list/badge those that pass. This would have a longer-term impact on ecosystem security.

And we could provide security support to all popular crates, regardless of affiliation (motivation 3). This would have the greatest immediate impact on the security of the ecosystem. Pay people to audit crates, and provide security fixes. Pay them to mentor interested security volunteers.

Given the existing discussions on this RFC, it seems like splitting it would be more likely to achieve (some of) its goals.

[fintelia]

I don't think the RFC ever actually justifies why there can't be duplicate functionality between ESL crates. The closest is pointing out that having a single endorsed crate making it easier for users to pick. But that doesn't seem like a sufficient argument to justify the significant downsides (discussed elsewhere in the thread). It wouldn't completely eliminate the concerns around "picking winners" but not having a fixed quota would at least slightly mitigate them.

Really it feels like there's two pieces of this proposal. One is a set of suggestions around funding core ecosystem crates, trying to align support expectations (MSRV, etc.), and combining effort spent on infrastructure. That all seems completely virtuous and worthwhile to me. But there's also another piece that an uncharitable reader could interpret as trying to co-opt the crates.io ecosystem: Picking a single winner for each category, coercing maintainers to join with an implied threat of forking their crates, and the underlying goal of sharply decreasing the usage of non-ESL crates

[kornelski]

Please define some reliable machine-readable way of easily identifying whether a crate belongs to ESL. It doesn't have to be a namespace, it could be metadata in the registry index or even a list hosted at a well-known URL.

Being in some official GitHub repo/org is not easy to verify, since repository field in Cargo.toml is unverified and any crate can claim to be in an official repo. A robust check requires cloning and scanning the repos, and that's slow and fiddly.

[cosmicexplorer]

This is largely because the Rust standard library is intentionally limited in scope.

It is very strange to me that this current paradigm of the Rust standard library forms the foundation of this RFC's motivating argument, and then is never mentioned again. Package namespacing is mentioned several times--why isn't the limited scope of the actual standard library?

Motivating discussion: Rust's I/O rewrite 11 years ago

To the best of my understanding as I type this, the standard library's limited scope of support for e.g. libc wrappers seems to stem from the process of rewriting the language's I/O runtime from scratch 11 years ago: https://github.com/aturon/rfcs/blob/io-env/text/0517-io-os-reform.md#what-cross-platform-means (please correct me here). That RFC describes a Rust ecosystem which was heavily shaped by the constraints of the libuv runtime, and describes this mantra for the new era of I/O and OS interactions:

The APIs should only expose a service or a configuration if it is supported on all platforms, and if the semantics on those platforms is or can be made loosely equivalent.

I became aware of this intentional approach from the stdlib a few hours ago, because I was very confused about the docstring for std::env::args_os(), which mentions "security purposes" but doesn't clarify or follow up.

It turns out that that phrasing has been unchanged since the method was first added to Rust 11 years ago, as part of the complete rewrite of the entire I/O and OS interface for the language. Apparently that phrasing has confused quite a few Rust programmers during that time! If I had just rewritten the language's entire I/O layer to use a fundamentally different model, I would also write a docstring like that to remind users not to trust the first iteration too much!

The problem here is that: a language like Rust, with so many experts invested in its safety, can't rely upon heroic individual efforts like that at this phase. I think this is one among many stark examples that the language's standard library needs engineering headcount devoted to its development and maintenance. If users can't rely on the stdlib, that's a much bigger (solvable) structural problem. Focusing on third-party crates is understandable, but I think it's a distraction.

Confusingly, when this RFC mentions how providing functionality in the standard library is the ideal scenario, it mentions not rust, but "golang":

The theoretically ideal minimum is zero additional trusted parties, which is the situation when dependencies come from the same organization as the compiler already being trusted (e.g. the Golang standard library).

The stdlib is exactly the place I would think deserves further investment, by this exact logic!

How to expand the existing guarantees around platform-specific code

So far, this RFC seems to be making a good argument for the Rust stdlib to revisit its conservative approach to new functionality. The vision for platform-specific opt-in described in the I/O RFC from 11 years ago has this to say:

For example, the os::unix module could provide a stat function that takes a standard Path and yields a custom struct. More interestingly, os::linux might include an epoll function that could operate directly on many io types (e.g. various socket types), without any explicit conversion to a file descriptor; that's what "seamless" means.

This description is quite similar to how Python's os module currently works: even non-standard APIs like os.splice() from Linux are able to use standard types like a file descriptor. Python achieves this by encoding platform support in the stdlib, and not through a separate web of crates. Here's how splice() support landed in the Python stdlib in 2020: python/cpython#85791 (comment)

This RFC mentions the golang standard library earlier--I also admire go's standard library for (like Python) including functions like pread(). I had to implement a Rust wrapper of pread() for the zip crate myself: zip-rs/zip2#236. That's more unsafe code that I'll be spreading throughout the "web of trust", because the charter of the Rust stdlib currently abdicates the responsibility of providing safe wrappers over OS abstractions. Let's change that!

(pread() is especially cool because it can use a non-mutable & reference as a file handle.)

libc case study: readdir() and getdents()

But let's take a step back from the stdlib for a moment. "Platform bindings (libc)" was fourth on this RFC's list of base functionality. In fact, the Rust stdlib consumes the libc crate itself, so in some ways it's even deeper down the stack than rustc. But libc has its own issues, such as an incredibly unreliable CI suite.

(Side note: I think it's a little ridiculous that the libc repo, which is so central to the Rust ecosystem, and has so much activity, doesn't have larger runners from github actions. That would be one easy way to make the whole Rust ecosystem more efficient!)

I have been looking through the issues the Rust libc maintainers have posted (e.g. rust-lang/libc#4650), and I'm hopeful about the test situation improving. But I want to present a case study: my attempt to get the Rust stdlib using a newer and faster directory traversal function (still in progress).

Background: fs::read_dir() and its haze of #[cfg(...)] guards

• The Rust stdlib fs::read_dir() implementation is riddled with very lengthy and uncommented #[cfg(...)] guards, which are not typechecked and may hide completely broken code without thorough testing.
• Each time a Rust program attempts to read one more directory entry, the stdlib performs a readdir() syscall in a loop to ensure thread safety. It then has to interpret the result of readdir() to read a null-terminated string of unknown length safely, which requires using the very unsafe &raw mechanism.

That's currently what the stdlib performs so users don't have to. But in fact, the Rust stdlib is very out of date, because it hasn't received sufficient investment! POSIX standardized an improvement over readdir() last year, because it's been in linux and bsd libc for at least a decade. The new(er) directory iteration API getdents() drastically reduces syscall count and thread safety with a user-allocated buffer.

I have been making a filesystem traversal crate which uses getdents(), which of course means more unsafe code spreading through the web of trust. But this time I also had to provide platform support, since it wasn't in the libc crate yet. I made a prototype, it seemed to work, cool.

So I then tried submitting this to the Rust libc crate, hoping to then propose a change to the stdlib. But this ran into further issues:

• libc expects PR submitters to run their testing before submission, but the tests fail with incomprehensible errors even on Tier 1 supported platforms like x86_64-unknown-linux-gnu.
  • It currently has a Tier 2 with host tools target failing in its CI (freebsd-13).
• More importantly, the maintainers are working very hard on it but have very little free time to assess new submissions.
  • It was very difficult for me to believe that the libc crate was on a shoestring budget, and I expressed some frustration about this yesterday add musl and glibc wrappers for getdents{,64} libc#4522 (comment), until I realized that in fact everyone was doing their best and they were not ignoring me personally.

Comparison of stdlib stability guarantees

If the Rust stdlib is just severely understaffed, it begins to make more sense why e.g. coroutines haven't been stabilized yet, which are also very useful for local filesystem I/O. Or why the allocator API remains unstable despite receiving so much interest it spawned a separate working group.

I personally hooked up the pants build tool's task engine (https://www.pantsbuild.org/stable/docs/writing-plugins/the-rules-api/concepts) into the Python async model, because Python includes coroutines in its "batteries included" approach. There are third-party async engines like Trio, but the fundamental framework for non-blocking computation is provided in the base language, and standardized via PEPs. Here's an in-progress PEP I really like that actually links together sync and async execution models through Python's separate context manager protocol (https://peps.python.org/pep-0806/).

In the time Rust has failed to stabilize the allocator API, C++ has added, deprecated, and removed multiple features of its own allocator API: https://en.cppreference.com/w/cpp/memory/allocator.html. C++ having stable coroutines remains one of the biggest reasons I started looking into using C++ over Rust for a regex engine I'm working on.

Lack of stability does more harm than an imperfect API

In pants, we (like everyone else) use tokio's async executor for our async code. Tokio has actually built up its own mirrored implementation of much of std::io, e.g. tokio::io::AsyncWrite. Note how it faithfully mirrors the std::io::Write interface, including the is_write_vectored() introspection API......which is another perpetually-unstable API back in stdlib world. I gave a specific use case from the zip crate in the discussion thread a whole year ago: rust-lang/rust#69941 (comment). My first interaction on a rust-lang repo was describing my use case for this in a third-party crate, with no response.

getdents(), pread(), and splice() all required me to write unsafe code and learn by trial and error whether I was triggering UB. I did my best to create my own safe interfaces that abstract away platform details so users of my code can write safe and portable Rust code. But I am a single engineer and I can only guarantee my code works on the systems I'm most familiar with. It would be much more efficient if there were a well-staffed team of OS/platform engineers for libc and the std::{fs, path, io} stdlib modules who could ensure Rust code aligns with modern best practices for safe and efficient i/o, instead of every single crate maintainer having to solve this problem from scratch. I can tell you that it would make my job contributing to the zip crate much much easier, which means more time I can spend on my area of expertise.

Ecosystem case study: third-party crates reimplementing Path and OsStr

The std::path module is in a particularly bleak state. For example, what would you expect the following code to do?

use std::path::Path;

fn main() {
    let p1 = Path::new("a/b/c");
    let p2 = Path::new("a/b/./c/");
    assert_eq!(p1, p2);
    dbg!(p1);
    dbg!(p2);
}

Here's the answer:

[src/main.rs:7:5] p1 = "a/b/c"
[src/main.rs:8:5] p2 = "a/b/./c/"

This implicit normalization process also occurs during hashing, which is never mentioned in the docs and introduces hidden quadratic behavior in many common scenarios. To its credit, the method Path::components() is exceptionally well-documented. However, it raises a new conundrum:

in particular, a/c and a/b/../c are distinct, to account for the possibility that b is a symbolic link (so its parent isn’t a).

Basically, is Path more like a string, or more like a handle to a filesystem object? Let's ask std::path:

This module provides two types, PathBuf and Path (akin to String and str), for working with paths abstractly. These types are thin wrappers around OsString and OsStr respectively, meaning that they work directly on strings according to the local platform’s path syntax.

So paths are strings? But strings with implicit semantics that differ by platform? And then there's e.g. Path::try_exists(), which is an alias for fs::exists(), so now we're mixing together strings and filesystem state, and we have to include repeated disclaimers like:

You should only use it in scenarios where those bugs are not an issue.

Our advice to avoid TOCTOU bugs says:

Keep file open for the duration of operations.

But then the stdlib doesn't provide any sort of directory handle to codify these external resource lifetimes at the type level!

So you have a weird kind of string that lets you get some information about the filesystem, at best. But if you want to achieve the kind of type safety provided by e.g. Arc vs Weak vs & vs &mut, where you have to explicitly call an unsafe method to risk breaking its guarantees, you have to implement all of that yourself.

type safety with path strings

The Rust stdlib Path very tightly couples its string processing logic with assumptions about the current OS. Splitting path strings into components is considered a platform-specific operation in Rust, even though it doesn't perform any syscalls or hit the filesystem. This approach entirely precludes a very critical component of portability: translating between supported platforms. All of these distinctions use the same Path type in Rust (which also performs implicit normalization for hashing and equality).

Here's one alternative: Python is the state of the art here currently (https://docs.python.org/3/library/pathlib.html):

Now consider bstr's description of platform conversions (https://docs.rs/bstr/):

The reason why using byte strings for this is potentially superior than the standard library’s approach is that a lot of Rust code is already lossily converting file paths to Rust’s Unicode strings, which are required to be valid UTF-8, and thus contain latent bugs on Unix where paths with invalid UTF-8 are not terribly uncommon. If you instead use byte strings, then you’re guaranteed to write correct code for Unix, at the cost of getting a corner case wrong on Windows.

bstr has more to say here, though: https://docs.rs/bstr/latest/bstr/#when-should-i-use-byte-strings. Paths are not just used to query the filesystem--users expect to and need be able to serialize them into bytes:

Why would you ever want to run a regex on a &[u8] though? Well, &[u8] is the fundamental way at which one reads data from all sorts of streams, via the standard library’s Read trait.

The stdlib essentially made an architectural misstep by failing to separate the string representation of a Path from the platform-specific logic in std::fs, and instead of iterating and fixing that initial mistake, we just make Paths impossible to serialize at all, which leads to the problem from this RFC: table-stakes functionality in third-party crates.

This library is past version 1 and is expected to remain at version 1 for the foreseeable future. Therefore, it is encouraged to put types from bstr (like BStr and BString) in your public API if that makes sense for your crate.

Andrew Gallant does incredible work and I am so grateful he has robust financial support that allows him to make these guarantees to the Rust community.

Why isn't this true for the Rust standard library?

[cosmicexplorer]

"Batteries included" standard libraries for languages are a well-established way to achieve this goal. It is a safe and reliable solution so long as there are resources to pursue it. There are great examples to look to for ideas regarding architecture, implementation, and policies.

I am assuming the Rust Foundation and ISRG are also in a position to remunerate infrastructural and maintenance work on the Rust stdlib that finally includes some batteries. Can I do a counter-RFC? Do those orgs have a grant application process?

[scottmcm]

4. Which hex or base64 crate? The one that compiles to small code (yay, good for embedded and wasm) or the one with SIMD support (good for my 128 core server load where only speed matters).

I think the core here is that we want a language+crates feature allowing people to declare more things like the allocator hook where multiple crates can provide an implementation without other crates needing to know about them all. That would be useful for a ton of things, even things already in core, like numeric formatting -- giving the end binary a choice between size, speed, or calling out to libc would be great.

And that would further reduce the need for ESL-like things, since you could always make your own base64 implementation with your own implementation of the algorithm if you wanted, without needing to patch everything in the tree that needs it.

[TimTheBig]

I agree with the std lib needing more support, however a mechanism to allow braking changes in standard libraries adds quite a of value for higher level utils were API really maters.

[fintelia]

The recently announced Rust Innovation Lab seems to have similar goals to this RFC and it appears there's also fair amount of overlap among individuals/organizations. If the proposal in this RFC is no longer under consideration, perhaps it would make sense to give an official update?

[Manishearth]

n.b. the Rust Foundation is not involved in the governance of the project, I don't think their actions merit an official announcement from the project for this plan here. The project can continue to consider this RFC on its merits, with the existence of the Innovation Lab as potentially additional evidence for alternative routes.