Distributed crate cache

[brson]

Large scale build farms (like Gecko's) really need distributed caching of build artifacts ala distcc/ccache. This will eventually be a blocker for continued adoption of Rust in Gecko.

We've talked about this several times but don't have even a vague design.

cc @larsberg

[larsberg]

I think you want @larsbergstrom?

[brson]

Yes, thanks larsberg!

[glandium]

For what it's worth: Gecko build slaves are using sccache, which is a tool kind of like ccache, except it works for gcc/clang and MSVC, and uses a network storage (S3). It's currently written in python, but for multiple reasons, I'm planning to rewrite it in rust. Which makes me think there's maybe a base to share here.

[eddyb]

Would having a machine-local build cache be covered by this, or should I open a new issue?

[larsbergstrom]

There are two issues I was curious about here:

1. Can we make a ccache/sccache-like tool work with Rust? Can third parties just do it themselves, or is there support needed in the compiler/cargo?

2. For situations like artifact builds in Firefox (https://groups.google.com/forum/#!topic/mozilla.dev.builds/jGg69m0x6Ck ) where the final binaries are retrieved from a Server.

[larsbergstrom]

From @ncalexan on the second issue:

@larsbergstrom this ticket is pretty broad, and the discussion of caching doesn't really say what you want to do. Artifact builds download the major compiled libraries from Mozilla's automation; the big one is libxul.so. I have only followed the rustc integration at a distance, but it's my understanding that y'all produce .o files that are linked into existing libraries (including libxul.so). If that's true, then any Rust integration into the Gecko tree should "just work" with artifact builds. (Of course, Rust developers won't be able to get the speed advantages of artifact builds, but that'll teach you to use a compiler :))

@rillian is the person I tap to keep abreast of Rust/Gecko integration progress -- perhaps he can add color here?

[rillian]

Right now the rust code gets linked into libxul, so @ncalexan is correct that Artifact builds should be unaffected.

Having cargo be able to query a crate-level build cache would be interesting, and an easier integration point for projects outside gecko.

@eddyb asked about machine-local caches. While that's not what we're talking about here, and it would be simple enough to teach cargo to link build results into a shared cache under .cargo like it does for sources, a ccache/distcc-oriented interface would work for both.

[alexcrichton]

I had some more thoughts about this today when discussing with @wycats. I think a lot of desires could be solved with something like this:

• I built a similar set of dependencies in two locations on my computer, I shouldn't rebuild everything.
• I have two separate Cargo projects which may have a shared crate graph, neither of which produced a final artifact. I then create a later project which links on the two. Each project builds independently, then the main project wants to link them together but not rebuild everything.
• Build farms where I want to distribute my build across many machines, or I simply want to reuse the previously cached results from a build on another machine.

My thinking of how we'd implement this is a relatively simple system:

• Each artifact produced by Cargo would have a unique key. This key includes all information needed to produce that artifact including, but not limited to, the rustc version, dependencies, crates.io version, maybe source code for path deps, compiler flags, and compiler target.
• This unique key is then hashed, e.g. with sha256.
• This smaller key is then used as a key in a key/value store to look up the artifact (or set of artifacts)
• When Cargo does a build, it has knowledge of a "global crate cache" (perhaps configured via .cargo/config). This may default to the filesystem.
• When building a key, it queries the cache if it has it, and if so the build is skip and the next step happens.
• If a key is not in the cache, cargo does the build and then pushes the artifacts into the cache.

The idea here is that a the value for a key never changes (kinda like a CAS system). That way if a cache gets concurrent pushes it can ignore all but the first. (and maybe assert they're all the same). Cargo could then support custom backends for the get/put functionality, for example to push to S3 or to push to an internal company caching server.

Some caveats:

• Build scripts and plugins may be very difficult to cache. They can perform arbitrary code and we don't actually know what the set of inputs are, nor do we always know the set of outputs.
• Some environments have extra information as input to the key. For example it may be a source code revision in the monorepo or something else like that, so there should be a way for a custom implementation to mix in data with the key that Cargo itself generates.

[larsbergstrom]

@alexcrichton Thanks! This sounds exciting and that looks good to me at first blush.

@luser @glandium Are there cases where this key into the compiled crate might break, based on your experiences with CCACHE/SCCACHE/artifact builds?

[luser]

sccache uses the output of the preprocessor + the compiler commandline as input to the hash key. I suspect the equivalent in the Rust world would be something like a hash of some IR + compiler options, and some hash of any external crates being used.

[luser]

@alexcrichton, @glandium and I met yesterday to discuss this, and we came up with what seems like a workable path to using sccache for caching crate compilations that are done via cargo. We'd have cargo invoke sccache as a wrapper around rustc, just like we do for C++ compilation in Gecko. (Cargo currently doesn't support rustc wrapper binaries, we probably want to add that as a feature, but for now we can just set RUSTC=wrapper_script.) sccache does compiler detection, so we'd teach it to detect rustc, and then generate the hash key from the following:

• The full rustc compiler commandline
• The contents of all files specified on the commandline
• The contents of all files listed in the dependency info output by rustc (this should be fast)
  ** we wanted to double-check that files generated from build scripts were listed in this output
• The contents of the compiler binary (we were unsure whether we would need to also hash the sharedlibs that the rustc binary links to, since the rustc driver contains almost no code)
• Any native libraries that are being linked to the final output. These are only provided as linker arguments, AIUI, so sccache would need to replicate the linker's search mechanism here.
• Some environment variables that impact the output of rustc (We didn't enumerate these, it would be good if we had a specific list!)

One big caveat that Alex mentioned is that plugins providing procedural macros can break our assumptions here. We expect that most well-behaved plugins will not be a problem, as they should be idempotent on the input, so as long as we hash the input and the plugin binary that should be sufficient. We could just declare that non-idempotent plugins will produce bad results, but we might want to do some special handling there if we intend to enable this more broadly in the future. Alex also mentioned that procedural macros that use data from external files will be an issue, those files don't currently make their way into the dependency graph. It looks like the built-in macros like include_bytes do the right thing here, but we might need to add a way for procedural macros to feed extra files they reference back to the compiler.

This would all fit very well into the existing sccache codebase, and doesn't require any changes to rustc (modulo the note above about making procedural macros work better with external files), and only one small change to cargo (allowing a rustc wrapper), all of which makes it a very appealing approach. Alex believed that given the strong hashing that rustc uses for crate dependencies, which wind up embedded in rlibs, hashing all the inputs on the compiler commandline should provide a strong hash key.

Alex, Mike, if there's anything important that I left out please let me know!

We're extremely interested in this from the Gecko side, as we start to add more Rust code to the Firefox build, and it sounds like this would have broad benefit to all large Rust projects.