Implement "pipelined" rustc compilation

[alexcrichton]

One possible feature we've talked about for a long time but never got around to implementing is the concept of pipelined rustc compilation. Currently today let's say that we have a crate A and a crate B that depends on A. Let's also say we're both compiling rlibs. When compiling this project Cargo will compile A first and then wait for it to finish completely before starting to compile B.

In reality, though, the compiler doesn't need the full compilation results of A to start B. Instead the compiler only needs metadata from A to start compiling B. Ideally Cargo would start compiling B as soon as A's metadata is ready to go.

This idea of pipelining rustc and starting rustc sooner doesn't reduce the overal work being done on each build, but it does in theory greatly increase the parallelism of the build as we can spawn rustc faster and keep all of a machine's cores warm doing Rust compilation. This is expected to have even bigger wins in release mode where post-metadata work in the compiler often takes quite some time. Furthermore incremental release builds should see huge wins because during incremental rebuilds of a chain of crates you can keep all cores busy instead of just dealing with one crate at a time.

There's three main parts of this implementation that need to happen:

• First, the compiler needs to be fixed to actually accept metadata as input for dependencies when producing an rlib. I believe currently it requires all inputs to be rlibs, but it shouldn't be too hard to fix this. (Tracking issue for rustc changes is Implement "pipelined" rustc compilation (compiler side) rust#58465)
• Second, the compiler will need to send a signal to Cargo as soon as metadata is produced. Cargo will instruct rustc to produce both metadata and an rlib, and once Cargo gets a signal metadata is ready then rustc will go on continuing to produce an rlib and Cargo will start subsequent rustc invocations that it can. Note that when I say "signal" here I don't really mean unix signals, but rather something like a message over a TCP socket or something like that.
• Finally Cargo will need to be refactored to listen for these signals, fork off more work in parallel, and properly synchronize the results of everything. This is where I suspect the bulk of the work will be happening (hence the issue in this repository).

In the ideal world the compiler would also wait just before linking for Cargo to let it know that all dependencies are ready. That's somewhat difficult, however, so I think it's probably best to start out incrementally and simply say that Cargo doesn't start a compilation that requires linking until all dependencies are finished (as it does today).

I've talked with @ehuss about possibly implementing this as well as the compiler team about the idea, but I think this is a small enough chunk of work (although certainly not trivial) to be done in the near future!

[dwijnand]

Duplicate of #4831?

[alexcrichton]

Heh yep! I'm gonna close that one in favor of this though

[mshal]

What is the reason for using a signalling mechanism instead of just using 2 rustc processes? (The first for the metadata and the second for the rlib.) Then the 2nd and 3rd of the proposed steps largely disappear, as the "signal" for the metadata being ready is just that the metadata rustc process returns. I believe this would allow better integration with other build systems since the dependency information would be captured in the DAG rather than via a special interaction between cargo and rustc.

[luser]

From when we discussed the previous issue, my only quibble here is that for consumption of a build plan by external build tools the "rustc sends a signal to cargo" part of the design gives a fairly tight coupling. Using a single rustc process to emit both the metadata and the rlib would of course be more efficient than spawning rustc twice in series, but using separate processes would be very straightforward to represent in a build plan and also likely require fewer changes to cargo's existing job execution architecture.

I don't know enough about the internals of rustc to offer an informed opinion about whether the latter approach would be feasible from that side of things.

[alexcrichton]

@mshal despite incremental compilation being implemented and fairly quick it's not "zero time" and so to get maximal benefit we'd need to keep rustc running and avoid it having to re-parse all internal state and re-typecheck and all that (even if incremental). It's true it might be simpler to just emit the metadata and spawn another process, but I'd fear that it wouldn't be as big of a win as we could possibly get.

@luser currently build plans are already a very lossy view of what Cargo does (although we have plans to fix that!). Depending on how the implementation goes here we could always of course generate a build plan that spawns multiple processes.

[eternaleye]

@alexcrichton

I feel like while two processes wouldn't get the same benefits right away, it definitely offers a path to the same benefits, by allowing later addition of --emit metadata --pipeline-phase=1 or whatever to pass along the typechecking (and other) info that --emit link --pipeline-phase=2 would otherwise recompute. It also has the benefit of a simpler and easier-to-drive execution model, which seems like it'd be hard to retrofit onto the "signaling" model after the fact.

I'd favor a simpler and easy-to-drive MVP with a path towards optimal results over a more-complex and harder-to-drive MVP that tries to jump there in one go.

[eddyb]

I think that while we should still experiment with this, we should be wary of stabilizing something here, since MIR-only rlibs and/or multi-crate incremental sessions may end up being better solutions overall.

That is, this pipelining is something we should've done a long time ago, but now we're not so sure it's the best way.

Other than that though, I think experimenting with this and gathering statistics on it is good!

cc @michaelwoerister

[michaelwoerister]

The version that invokes rustc twice would have an overhead roughly of doing an additional incremental check build (see https://perf.rust-lang.org/?start=&end=&absolute=true&stat=wall-time). That is usually pretty fast (3-10 times faster than a normal build) but not free.

From a rustc perspective the changes needed would not be too big and they don't add a whole lot of complexity. If this stays an internal feature that only Cargo knows how to use and that can be removed again at any time, I don't have any objections here. I still hope that we can do something smarter (MIR-only-RLIB-like) in the future but it's unclear whether we'd make any progress on this front within 2019.

One thing that might be interesting: Once rustc supports metadata-only dependencies for producing regular RLIBs (which it has to do for signaling variant) then the two process version would basically just work, right?

[alexcrichton]

@eddyb I disagree that we should continue to block for MIR-only rlibs, we've collected data and research showing that it's not a silver bullet and has fundamental downsides that the current model of compilation solves. In that sense I wouldn't consider this an experiment, but this is how I'd personally like Cargo to permanently compile Rust code. If MIR-only rlibs ship one day then it'd presumably also benefit from a pipelining architecture, although perhaps not quite as much.

@michaelwoerister correct, the two-process version should work after rmetas are consume!

Also FWIW I'd personally like to push as much as possible for "this is an unstable interface between rustc". I find the current state somewhat draconian where if we want Cargo, the primary build tool for Rust, to do anything different today we have to go through a huge stabilization process. This hugely hinders experimentation and development of a much richer interface. Cargo has all sorts of knowledge that rustc has to either relearn and/or spend time recompution. This is just a tiny feature ("signal something via some method") which is quite easy technically to implement in rustc and would be a bummer if it is slowed down via process.

[eddyb]

@alexcrichton I don't disagree regarding the current situation, but I suppose if you're taking those downsides into account, the long term silver bullet might be multi-crate compilation sessions.

I also don't want to block anything here, as long as it stays between Cargo and rustc.
Especially your last paragraph, strong agree with everything there!

[luser]

One additional issue with the single process approach is that it would break sccache unless we also implemented support for making sccache act as an intermediary between cargo and rustc (this would be a bit of a pain but not impossible). However, sccache also gained support for icecream-style distributed compilation, including distributed Rust compilation, and I don't think the single-process model would be compatible with distributed compilation. This would be especially unfortunate because a distributed compilation configuration would stand to benefit the most from unlocking extra paralellism in the build.

[eddyb]

@luser That just sounds like sccache is misplaced in the compilation process (which had been my suspicion for a long time, given the differences in compilation models of file-based compilers and rustc, which multi-crate sessions only exacerbate).

Distributed rustc could work in a similar fashion to "parallel rustc", albeit we need a design based more on opportunistic deduplication between threads than locks everywhere.

At the very least, what you want to share is the new incremental persistence, not the old metadata.
(and we should make that as portable as possible, so that you can e.g. use an AArch64 cluster to precompute incremenal state for x86 builds, and vice-versa)

EDIT: my bad, @eternaleye pointed out on IRC that you were talking about the signalling approach and not necessarily anything regarding further developments in incremental/parallel rustc.

Regarding that, I agree running rustc twice is closer to a "pure computation dependency graph" which is the ideal case for distributed compilation.
Ideally, incremental would reach "sub-frame" (i.e. <16ms) times on an (almost) clean build, even on large projects, and be primarily limited by disk access (which IDEs/pipelining could bypass).