Fuzzing incremental compilation against crates.io packages

[nikomatsakis]

To really have confidence in incremental compilation, we need to do large-scale testing. One specific way to do this is to fuzz against different versions of crates from crates.io. This corresponds to a very particular, and common, workflow:

• cargo update to get new packages
• cargo build to build with them

The idea here would be to write a tool which iterates over the last N published crates (that is, the last N versions that have been published to crates.io). For each version V that has been published, we would:

• Create a fresh directory
• Load version V-1 (that is, the previous version available on crates.io)
  • if there is no previous version, skip this crate
• Compile version V-1 using -Z incremental=some-temporary-directory
• Load version V in place (overwriting the old files)
• Compile version V using -Z incremental=some-temporary-directory
• Create another directory
• Compile version V from scratch
• Compare the results of the two compilations of version V:
  • they should produce the same errors/warnings
  • if successful, they should produce the same output file
    • or at least it should contain the same symbols etc
    • we'll have to work this out a bit =)

As of this writing, we're not yet at the point where incremental compilation actually works, but we are already tracking dependencies throughout the front-end; just tracking those dependencies sometimes causes ICEs, which means this tool would be of immediate use. Plus, we expect to be tracking dependencies soon.

I imagine that this tool would be written in Rust. Rather than placing it in src/test/tools, it might be better if it lived as its own crate (maybe in rust-lang?), so that it can draw on the full crates.io ecosystem for utility functions and so forth. We would eventually just run it on a regular and continous basis and open issues for any problems that we find.

Since it has relatively few prerequisites, this bug seems like a good "entry vector" for contributing to Rust. I would love to "mentor" someone to work on it.

[mrmiywj]

Hi, Matsakis. It looks fun, I want to work on this. Actually the workflow you described looks clear, I need more details.

1. Maybe just compile one crate with incremental way and common way, then compare the output and code is better? For your provided workflow, it seems that it should compile a whole project with some crates.
2. To compare the outputs, is there any API provided by rustc to generate inner data struct of compile info? If not, maybe comparing the output from stdout and stderr is worth considering.
3. To compare output file, is it possible that the files generated by the two ways is same if compared bit by bit? Sorry that I don't know how rustc incremental compilation works, could you explain it?

[alexcrichton]

FWIW I've got a script which @jonathandturner has also been working with which is essentially "use cargo to iterate over all crates on crates.io and build them", and that may end up being useful here!

[mrmiywj]

@alexcrichton Hi, could you offer the link of the script? It must be helpful!

[alexcrichton]

Certainly! I've pushed it up here -- https://github.com/alexcrichton/cargo-apply

It doesn't have much documentation, and it's kinda a big hack, but the gist is:

• First, modify src/main.rs (near the bottom) to configure how rustc is run over each crate.
• Next cargo run --release
• Wait for output to get populated with a bunch of files
• output/$crate/stdio is the result for each crate.

[nikomatsakis]

Great to see so much interest! I want to point out that there are more opportunities for fuzzing, so whatever we do, it might make sense to try and have some common code with multiple modes. Some other examples I've been thinking about:

• go through git histories and try to compare subsequent commits
• similarly, use git histories to guide random fuzzing; this could be particularly good for detecting that we still report all the errors we should

[cramertj]

It seems as though others have expressed interest already, but I'd enjoy helping out if there's space for one more.

[mrmiywj]

@nikomatsakis
Hi, Niko
These days I'm trying on this. But there are still some issues about this tool I don't understand.

1. Actually what should the tool work on with? Compile a project with Cargo.tom, trying to compile it with dependencies of various version. Or just compile a pkg ?
2. I tried to hack the code provided by @alexcrichton and found that the only difference between result compiled with incremental or not is just a .rlib file.
3. What does load in your workflow mean to? Actually with the tool, I don't need to load any file. Just figure out which crate and version, with ops::compile_pkg will help us compile it.

[mrmiywj]

@nikomatsakis
Hi, I just finish a stereotype of this project here. Could you offer some advices?

1. Does it work correctly? I mean, does it actually produce the incrementally compiled code?
2. What should I do next? How to compare the result of two ways?

Hope for ur advice!

[nikomatsakis]

@mrmiywj

What does load in your workflow mean to? Actually with the tool, I don't need to load any file. Just figure out which crate and version, with ops::compile_pkg will help us compile it.

What I meant by "load" was "download the sources and expand them into a directory". You're right that once we've done that once there isn't really a need to do anything more after that.

Does it work correctly? I mean, does it actually produce the incrementally compiled code?

Hmm. :) Good question! The code looks like it sort of roughly does the right thing, but for incremental mode, it will have to first run the compiler (roughly as you are doing now) with the source from v1 and then run the compiler again with the source from v2 (it seems like you don't currently have the notion of two revisions, just one).

[mrmiywj]

@nikomatsakis
Hi, I updated my repo.
Actually there was something wrong with dup2. Now I think it works. You can read the stdio in incremental and no incremental directory to find that it actually compile incrementally after you run it!

Now the issue is:
How to compile the file and output of the two ways' compilation. Since I read the outputs, I found that they can not be compare word by word. And, how to compare the lib and binary file generated, bit by bit?

[nikomatsakis]

@mrmiywj neat, I will try to take a look tomorrow. I think that the compiler output ought to be comparable word-by-word, perhaps with some filename normalization...is that not the case?

As for compiling the binary generating, that's a trickier problem. I do not believe they will necessarily be binary identical, because there is some freedom permitted to the linker to reorder symbols and the like. We might be able to dump some of the compiler's internal representation (or the LLVM IR) can compare that, though. I'll have to experiment a bit!