LTO: 10x regression

[luben]

I noticed a significant regression since nightly 2018-01-27 when enabling LTO. Here is a simplified example that has 10x difference.

fn fill<T: Clone + 'static>(init: &'static T) -> Box<Fn(&usize) -> Vec<T>> {
    Box::new(move |&i| {
        let mut vec = Vec::<T>::new();
        vec.resize(i, init.clone());
        vec
    })
}

fn main() {
    let zeroes = fill(&0); 
    for _ in 0..1_000_000 {
        zeroes(&1000);
    }
}

Without LTO enabled:

$ perf stat -B -e cycles,instructions ./target/release/columnar

 Performance counter stats for './target/release/regr':

       335,160,070      cycles                                                      
       539,078,439      instructions              #    1.61  insn per cycle         

       0.112408607 seconds time elapsed

with LTO enabled:

 Performance counter stats for './target/release/regr':

     1,121,648,955      cycles                                                      
     5,233,309,105      instructions              #    4.67  insn per cycle         

       0.361163946 seconds time elapsed

Tested variants
Bad: -C lto=fat
Good: -C lto=thin
Good: -C lto=fat -C codegen-units=1

[siddontang]

Refer http://www.brendangregg.com/perf.html#CPUstatistics, Gregg says:

This includes instructions per cycle (IPC), labled "insns per cycle", or in earlier versions, "IPC". This is a commonly examined metric, either IPC or its invert, CPI. Higher IPC values mean higher instruction throughput, and lower values indicate more stall cycles. I'd generally interpret high IPC values (eg, over 1.0) as good, indicating optimal processing of work.

But I am confused that why LTO uses so many instructions.

[ollie27]

What results do you get with -Ccodegen-units=1?

[luben]

@siddontang , higher IPC is good if we execute the same (or similar) number if instructions. Here with LTO it executes 10x number of instructions but without 10x increase of IPC that leads to 3x increased run-time.

@ollie27 , just tested, it solves it. Another correction, -C lto=thin also solves it, I didn't test it properly before.

[alexcrichton]

The change list for this regression is unfortunately pretty large (between rust-nightly-2018-01-26 and rust-nightly-2018-01-27). Like @luben I can confirm that -C lto=fat produces poor performance while -C lto=thin produces good performance.

The primary change in that change list though was switching the default back to "fat" LTO, it seems at some point in the past our fat lto regressed? I'm running out of time to help narrow this down but what I've found so far is that we had a surprising regression of this program between nightly-2017-12-15 and nightly-2017-12-16 of just the optimized version (not even the LTO version). The change list doesn't have anything suspicious though.

[nikomatsakis]

(Just to be clear, this is a regression in the perf of the compiled code, right?)

[nikomatsakis]

@alexcrichton do you think you will have time to chase this down? Should we try to find someone to assign?

[nikomatsakis]

triage: P-high

Need to figure out what's happening here.

[alexcrichton]

@nikomatsakis unfortunately I'm pretty tight on time, so it's probably best to find another soul to help out with this.

[alexcrichton]

Er ok had some time to debug this. @eddyb it looks like this regression may be caused by #46623 or at least something is caused by that.

The logs I have are:

before

+ rustc -vV
rustc 1.24.0-nightly (50f6c3ece 2017-12-15)
binary: rustc
commit-hash: 50f6c3ece0ec738da48f8e77e6379a14bd02d1f4
commit-date: 2017-12-15
host: x86_64-unknown-linux-gnu
release: 1.24.0-nightly
LLVM version: 4.0
+ rustc foo.rs -O -C codegen-units=1
+ perf stat -e cycles ./foo

 Performance counter stats for './foo':

       331,404,612      cycles

       0.085892079 seconds time elapsed

after

+ rustc -vV
rustc 1.24.0-nightly (77efd6800 2017-12-15)
binary: rustc
commit-hash: 77efd6800c57ba83923dddbbabf03c7afa6a34a4
commit-date: 2017-12-15
host: x86_64-unknown-linux-gnu
release: 1.24.0-nightly
LLVM version: 4.0
+ rustc foo.rs -O -C codegen-units=1
+ perf stat -e cycles ./foo

 Performance counter stats for './foo':

     3,129,021,965      cycles

       0.806494586 seconds time elapsed

That is, with #46623 the benchmark without LTO regressed by 10x

[eddyb]

@alexcrichton Maybe it's similar to #47062 (comment)? Is there any change around that nightly?

[alexcrichton]

False alarm, @eddyb is right in that the commit I bisected above was fixed by #47007