Establish a clear vision for Rust 2018

[aturon]

This WG hasn't managed to take off, largely because the leads have been heads-down trying to get futures 0.3 and async/await working. With an alpha nearly out the door, we want to try to get this broader group going again, and see what we can accomplish by the Rust 2018 release date (December 6th).

[yoshuawuyts]

see what we can accomplish by the Rust 2018 release date

If I'm correct that's 12 weeks or so from now right? That's probably not a lot of time to tackle large things, but perhaps there's some smaller things we could do?

If we draw some parallelisms with the CLI WG; from my experience what ended up working well for us was to get authors of widely-used libraries involved in the communication channels. But also figuring out which features might be important to have for a "perfect" CLI experience, and building out a bunch of libraries to move towards that.

The leads of the larger CLI frameworks have kept on pushing their work (yay for clap v3 being scheduled for this fall!), but in the process we've been able to release an odd 7 or 8 libraries we thought would be helpful.

I would see a similar approach perhaps working well for the Net WG! It seems there's a handful of people leading some important, large efforts that will require some time to complete. But I reckon we could probably get a smaller group of people involved to identify speed bumps and write small, focused libraries help solve those.

This is probably a rather modest approach to rust 2018; but I feel with the time available the biggest win we can get is probably by focusing on small projects we can see through from start to finish!

Hope this is somewhat useful! Thanks!

Resources

• https://github.com/pillarjs - a group of JS HTTP framework authors came together in 2014, and started collaborating on shared dependencies. There's a pretty rich history contained in building small network libraries in here!
• https://github.com/yoshuawuyts/exponential-backoff - this is about the scope of libraries I'm thinking of. Small, reusable, mostly focused on one thing. I'd be happy to move any network related libraries I wrote to a shared GH org if we think that would be a viable approach!

[aturon]

Thanks @yoshuawuyts, that seems like a useful framing for sure! We'll cover this in detail in the kickoff meeting, but one thing I wanted to mention is that the Core Team is now planning to do an extended beta of the Edition (to be announced tomorrow), which means that we have about 19 weeks -- definitely enough time to make some significant progress, especially given that we have some full-timers we can allocate here.

Anyway, the kickoff meeting is where we'll really start to hash this out, and I'll post notes here afterwards!

[aturon]

Kickoff meeting agenda is here!

[aturon]

In the kickoff meeting, I proposed the following as a minimum bar for Rust 2018:

• Documentation providing:

  • Quick start for building web apps
  • Guidance on when/where/how to use async
  • Guidance on what crates are available and how they work together
  • Tons of examples
  • Best practices for networking crates

• Some solid end-to-end stories

  • The idea is to take a few typical scenarios where one might want to do network programming in Rust, and work on all of the relevant pieces to making those stories smooth. There's a lot of work needed to define in more detail what this might look like -- ideas requested!

• Web site: a major part of Rust 2018 is a revamped web site that will highlight each of the four "domains" we've targeted this year. We need to have a polished pitch, with entry points into various parts of the networking ecosystem.

The above is fairly generic. I want to hear from you what seems important and feasible to tackle on the Rust 2018 timeframe!

[aajtodd]

@aturon That all sounds great. Here is where my disconnect is though. I feel like we are missing a recap of where we are at vs where we would like to be. Futures vs futures-preview, the entire tokio stack, tower, futures-await, etc are all part of this story but it would be great to hear what the overall plan is for all of these pieces and where the team feels like they will fit in (or if they are to be deprecated).

We could certainly go write lots of documentation and examples from the current state of things but my understanding is we are not close to done with the async story, rather we are just getting started as async/await becomes a core concept in the language.

With that being said, some thoughts, concerns, and questions in no particular order:

1. In general do you envision all of the libraries/examples/documentation being written around what exists today or should we target async/await notation on nightly?
2. Follow up, if we are targeting what will be vs what exists now I think we need a better understanding of what that looks like.
3. I think it's going to be difficult to cement higher layers when the core language features and supporting library (futures-preview, async/await, etc) are in flux. Cement may not be the goal, perhaps just gaining experience with these things is good enough to start. I spent 2 hours earlier today trying to play with async/await and futures-preview and I'll be honest it was slow progress trying to understand how to get the abstraction to do what I wanted.

In my mind we need a lot more documentation and examples around core futures/async/await before we can provide guidance on higher layers.

All that being said I generally agree with you assessment, I would like to see the async book/guide you started completed. It looked like it would be extremely valuable to the community.

[tynril]

I agree with @aajtodd and share some of the same trouble. I love the goal of setting up guidelines for how to do it "right" when it comes to implementing network services, in terms of standard Rust systems to use, as well as in terms of libraries and middleware. Providing a good documentation there, some solid examples, as well as, ideally, a flexible ecosystem for the common use-cases, all sound great.

But the state in which Rust will be when the 2018 Edition ships will still be very much in flux when it comes to core components of this vision (specifically around async/await). Because of this, it seems challenging to provide all of this in a way that is usable with stable Rust. I also can't advocate for us setting up all of these guidelines and recommendations against nightly, both because then they'd be subject to change (or worse, would make it harder to change things that would need changing, like what happened with tokio to some extend), and because I think that it sends the wrong message (being forced to use nightly to do a lot of things is something that makes it harder for me to propose Rust as a solution).

While I understand and agree with the value of pushing as much things as possible out of the door with the 2018 Edition marketing operation, either of these options (nightly-only guidelines, or soon-to-be-deprecated ones) are, in my opinion, more trouble than value.

Because Rust is there to stay, and because its value proposition around network services especially is so great, I think that we could, and should, let that deadline slip. If we're able to articulate a great set of tools, guidelines, libraries, middlewares, and books to document it all, all of on stable, production-ready Rust, I think that the value will speak for itself. It might also just be enough content to do a separate, more focused marketing push at that point (although I will admit that I don't know how much effort and budget are involved there).

Mayhap, in the meantime and to provide something for the 2018 Edition push, we could deliver some kind of preview of this work, mainly aimed at acquiring early adopters and contributors, who could help us achieve the loftier, longer-term goal.

Thoughts?

[aajtodd]

@tynril I'm actually ok with working on and pushing nightly if the expectation is it will be stabilized sometime in the 2018 Edition. One could argue we already have solutions that work on stable (e.g. existing tokio, futures, etc).

I would much rather be working towards a vision of what we want it to look like than "wasting" time on something that will be deprecated or OBE.

What I do want is clarification on which to focus on as immediate goals.

Just my 2 cents.

[davidbarsky]

A small, but valuable, point to cover would be deployments, especially deployment using containers/Docker. This documentation could cover:

• How do you build small images with Docker?
• How should OpenSSL/libpq/etc. be handled?
• How can the resulting binary remain debuggable once deployed?
• How can users optimize for Docker's cache to reduce build times?

I've been referring to Tõnis Tiigi' Advanced multi-stage build patterns recently. It has a lot of valuable points.

[tynril]

@aajtodd Agreed, I believe that a longer term solution, targeting nightly at the time the 2018 Edition ships (in December), is probably best.

There are downsides to this, though:

1. That a lot of useful things in the Rust ecosystem requires nightly is a fair and common complaint. This would add to this problem, at least temporarily.
2. It might diminishes the impact of the marketing push in December, by adding caveats to what the WG has to offer (i.e. it being nightly only and still in development, rather than ready for production use).

I think that this is worth it anyway, as it allows us to think slightly longer term (at a time when async/await is stable). I'm interested in the core team's perspective on this. @aturon?

[MajorBreakfast]

I personally think that we should focus on solutions that work on stable. Here's why:

async/await is unstable for a reason: It is not done yet. The syntax and the APIs are subject to change. Documenting it for everyone and presenting it as a "preview" is the same as stabilizing it prematurely. We don't want that. We need it to be unstable for a few more months because there are many areas that we haven't explored yet. There is for example a disagreement about the async function syntax itself. The only way to settle it, is to implement the not yet implemented parts and to actually try out which way works better. Also, I believe that when we stabilize, the essential parts of the ecosystem, in particular Tokio, should run on 0.3 without a compatibility layer. We need the experience from migrating these libraries to check that the futures 0.3 API isn't lacking in certain areas. Fully implementing async/await in the compiler and fully migrating some essential parts of the ecosystem over to 0.3 to make sure our APIs are good will take months, but I think it's the only way to ship async/await with confidence.

That said, I think we should devote a section on the website to explain which areas are being worked on, how to get involved and where to find updates to these efforts.

[MajorBreakfast]

Ah, I should add: It's a long time until December and it might very well be that we've finalized the syntax and API by then and it just hasn't made it into the stable build yet. That could be. We'll know whether it's realistic once we're closer to stabilization.