LLVM ERROR: Broken module found, compilation aborted! when building Firefox with full LTO

[glandium]

This didn't happen before 8840da2.

STR:

• Download https://drive.google.com/file/d/1eTOTaCkLfmShfSDF9OAF77sgxeJ-fgqT/view?usp=sharing
  (unfortunately large, this is Firefox)
• Extract
• Run: cd repr; ld.lld @response.txt

llvm.dbg.declare has larger fragment size than alloca size 
  call void @llvm.dbg.declare(metadata ptr %8, metadata !26454674, metadata !DIExpression()), !dbg !26454731
llvm.dbg.declare has larger fragment size than alloca size 
  call void @llvm.dbg.declare(metadata ptr %17, metadata !26462027, metadata !DIExpression()), !dbg !26462037
llvm.dbg.declare has larger fragment size than alloca size 
  call void @llvm.dbg.declare(metadata ptr %8, metadata !26468121, metadata !DIExpression()), !dbg !26468178
llvm.dbg.declare has larger fragment size than alloca size 
  call void @llvm.dbg.declare(metadata ptr %17, metadata !26475461, metadata !DIExpression()), !dbg !26475471
llvm.dbg.declare has larger fragment size than alloca size 
  call void @llvm.dbg.declare(metadata ptr %8, metadata !26481438, metadata !DIExpression()), !dbg !26481495
llvm.dbg.declare has larger fragment size than alloca size 
  call void @llvm.dbg.declare(metadata ptr %17, metadata !26488783, metadata !DIExpression()), !dbg !26488793
llvm.dbg.declare has larger fragment size than alloca size 
  call void @llvm.dbg.declare(metadata ptr %8, metadata !26494766, metadata !DIExpression()), !dbg !26494823
llvm.dbg.declare has larger fragment size than alloca size 
  call void @llvm.dbg.declare(metadata ptr %17, metadata !26502101, metadata !DIExpression()), !dbg !26502111
llvm.dbg.declare has larger fragment size than alloca size 
  call void @llvm.dbg.declare(metadata ptr %8, metadata !26508252, metadata !DIExpression()), !dbg !26508309
llvm.dbg.declare has larger fragment size than alloca size 
  call void @llvm.dbg.declare(metadata ptr %18, metadata !26516098, metadata !DIExpression()), !dbg !26516124
LLVM ERROR: Broken module found, compilation aborted!
PLEASE submit a bug report to https://github.com/llvm/llvm-project/issues/ and include the crash backtrace.
Stack dump:
0.	Program arguments: /tmp/gecko/clang/bin/ld.lld @response.txt
 #0 0x00007f765d87c8a7 llvm::sys::PrintStackTrace(llvm::raw_ostream&, int) (/tmp/gecko/clang/bin/../lib/libLLVM-18git.so+0x28398a7)
 #1 0x00007f765d87cd8b SignalHandler(int) Signals.cpp:0:0
 #2 0x00007f765ab7bfd0 (/lib/x86_64-linux-gnu/libc.so.6+0x3bfd0)
 #3 0x00007f765abcad3c __pthread_kill_implementation ./nptl/pthread_kill.c:44:76
 #4 0x00007f765ab7bf32 raise ./signal/../sysdeps/posix/raise.c:27:6
 #5 0x00007f765ab66472 abort ./stdlib/abort.c:81:7
 #6 0x00007f765d818698 llvm::report_fatal_error(llvm::Twine const&, bool) (/tmp/gecko/clang/bin/../lib/libLLVM-18git.so+0x27d5698)
 #7 0x00007f765cb90bd5 (/tmp/gecko/clang/bin/../lib/libLLVM-18git.so+0x1b4dbd5)
 #8 0x00007f765d96b097 (/tmp/gecko/clang/bin/../lib/libLLVM-18git.so+0x2928097)
 #9 0x00007f765e47c6ed llvm::detail::PassModel<llvm::Module, llvm::VerifierPass, llvm::PreservedAnalyses, llvm::AnalysisManager<llvm::Module>>::run(llvm::Module&, llvm::AnalysisManager<llvm::Module>&) LTOBackend.cpp:0:0
#10 0x00007f765d555bdc llvm::PassManager<llvm::Module, llvm::AnalysisManager<llvm::Module>>::run(llvm::Module&, llvm::AnalysisManager<llvm::Module>&) (/tmp/gecko/clang/bin/../lib/libLLVM-18git.so+0x2512bdc)
#11 0x00007f765e477af4 llvm::lto::opt(llvm::lto::Config const&, llvm::TargetMachine*, unsigned int, llvm::Module&, bool, llvm::ModuleSummaryIndex*, llvm::ModuleSummaryIndex const*, std::vector<unsigned char, std::allocator<unsigned char>> const&) (/tmp/gecko/clang/bin/../lib/libLLVM-18git.so+0x3434af4)
#12 0x00007f765e479594 llvm::lto::backend(llvm::lto::Config const&, std::function<llvm::Expected<std::unique_ptr<llvm::CachedFileStream, std::default_delete<llvm::CachedFileStream>>> (unsigned int, llvm::Twine const&)>, unsigned int, llvm::Module&, llvm::ModuleSummaryIndex&) (/tmp/gecko/clang/bin/../lib/libLLVM-18git.so+0x3436594)
#13 0x00007f765e46f30f llvm::lto::LTO::runRegularLTO(std::function<llvm::Expected<std::unique_ptr<llvm::CachedFileStream, std::default_delete<llvm::CachedFileStream>>> (unsigned int, llvm::Twine const&)>) (/tmp/gecko/clang/bin/../lib/libLLVM-18git.so+0x342c30f)
#14 0x00007f765e46eb73 llvm::lto::LTO::run(std::function<llvm::Expected<std::unique_ptr<llvm::CachedFileStream, std::default_delete<llvm::CachedFileStream>>> (unsigned int, llvm::Twine const&)>, std::function<llvm::Expected<std::function<llvm::Expected<std::unique_ptr<llvm::CachedFileStream, std::default_delete<llvm::CachedFileStream>>> (unsigned int, llvm::Twine const&)>> (unsigned int, llvm::StringRef, llvm::Twine const&)>) (/tmp/gecko/clang/bin/../lib/libLLVM-18git.so+0x342bb73)
#15 0x00005582ad0e6860 lld::elf::BitcodeCompiler::compile() (/tmp/gecko/clang/bin/ld.lld+0x25d860)
#16 0x00005582ad040549 lld::elf::LinkerDriver::link(llvm::opt::InputArgList&) (/tmp/gecko/clang/bin/ld.lld+0x1b7549)
#17 0x00005582ad02fa34 lld::elf::LinkerDriver::linkerMain(llvm::ArrayRef<char const*>) (/tmp/gecko/clang/bin/ld.lld+0x1a6a34)
#18 0x00005582ad02dfe0 lld::elf::link(llvm::ArrayRef<char const*>, llvm::raw_ostream&, llvm::raw_ostream&, bool, bool) (/tmp/gecko/clang/bin/ld.lld+0x1a4fe0)
#19 0x00005582acf72845 lld::unsafeLldMain(llvm::ArrayRef<char const*>, llvm::raw_ostream&, llvm::raw_ostream&, llvm::ArrayRef<lld::DriverDef>, bool) (/tmp/gecko/clang/bin/ld.lld+0xe9845)
#20 0x00005582acf70182 lld_main(int, char**, llvm::ToolContext const&) (/tmp/gecko/clang/bin/ld.lld+0xe7182)
#21 0x00005582acf7004e main (/tmp/gecko/clang/bin/ld.lld+0xe704e)
#22 0x00007f765ab671ca __libc_start_call_main ./csu/../sysdeps/nptl/libc_start_call_main.h:74:3
#23 0x00007f765ab67285 call_init ./csu/../csu/libc-start.c:128:20
#24 0x00007f765ab67285 __libc_start_main ./csu/../csu/libc-start.c:347:5
#25 0x00005582acf71b72 _start (/tmp/gecko/clang/bin/ld.lld+0xe8b72)

Cc: @nikic

(It's also unfortunate that it doesn't tell what specific module is broken)

[nikic]

The verifier error already occurs on libxul.so.0.0.preopt.bc, so I think that means that one of the inputs is already broken or it happens during module merging.

[nikic]

Running something like find . -name '*.o' -print0 | xargs -0 -L1 -P16 ~/repos/llvm-project/build/bin/opt -disable-output 2>&1 | grep "larger fragment doesn't give any results, so I think that means it must happen merging modules?

The relevant allocas look like this:

  %8 = alloca %struct.AVPacket.153109, align 8

I'm not sure how to debug this further.

[teresajohnson]

This is also breaking some of our ThinLTO builds (8840da2#commitcomment-129914039)

Could it be reverted while the fix is investigated?

It looks like somehow the debug declare and alloca get out of sync when the IRMover/IRLinker is invoked?

For regular LTO like in the case described here, the IRMover is invoked from LTO each time a module is linked:

llvm-project/llvm/lib/LTO/LTO.cpp

Line 995 in 844c731

return RegularLTO.Mover->move(std::move(Mod.M), Keep, nullptr,

. ThinLTO does something similar during importing, but I think it will probably be more straightforward to track down with the full LTO case since the IR merging happens immediately upon the linker adding modules to LTO.

[nikic]

@teresajohnson Feel free to revert, I can't do it myself right now.

[llvmbot]

@llvm/issue-subscribers-debuginfo

Author: Mike Hommey (glandium)

This didn't happen before 8840da2.

STR:

• Download https://drive.google.com/file/d/1eTOTaCkLfmShfSDF9OAF77sgxeJ-fgqT/view?usp=sharing
  (unfortunately large, this is Firefox)
• Extract
• Run: cd repr; ld.lld @<!-- -->response.txt

llvm.dbg.declare has larger fragment size than alloca size 
  call void @<!-- -->llvm.dbg.declare(metadata ptr %8, metadata !26454674, metadata !DIExpression()), !dbg !26454731
llvm.dbg.declare has larger fragment size than alloca size 
  call void @<!-- -->llvm.dbg.declare(metadata ptr %17, metadata !26462027, metadata !DIExpression()), !dbg !26462037
llvm.dbg.declare has larger fragment size than alloca size 
  call void @<!-- -->llvm.dbg.declare(metadata ptr %8, metadata !26468121, metadata !DIExpression()), !dbg !26468178
llvm.dbg.declare has larger fragment size than alloca size 
  call void @<!-- -->llvm.dbg.declare(metadata ptr %17, metadata !26475461, metadata !DIExpression()), !dbg !26475471
llvm.dbg.declare has larger fragment size than alloca size 
  call void @<!-- -->llvm.dbg.declare(metadata ptr %8, metadata !26481438, metadata !DIExpression()), !dbg !26481495
llvm.dbg.declare has larger fragment size than alloca size 
  call void @<!-- -->llvm.dbg.declare(metadata ptr %17, metadata !26488783, metadata !DIExpression()), !dbg !26488793
llvm.dbg.declare has larger fragment size than alloca size 
  call void @<!-- -->llvm.dbg.declare(metadata ptr %8, metadata !26494766, metadata !DIExpression()), !dbg !26494823
llvm.dbg.declare has larger fragment size than alloca size 
  call void @<!-- -->llvm.dbg.declare(metadata ptr %17, metadata !26502101, metadata !DIExpression()), !dbg !26502111
llvm.dbg.declare has larger fragment size than alloca size 
  call void @<!-- -->llvm.dbg.declare(metadata ptr %8, metadata !26508252, metadata !DIExpression()), !dbg !26508309
llvm.dbg.declare has larger fragment size than alloca size 
  call void @<!-- -->llvm.dbg.declare(metadata ptr %18, metadata !26516098, metadata !DIExpression()), !dbg !26516124
LLVM ERROR: Broken module found, compilation aborted!
PLEASE submit a bug report to https://github.com/llvm/llvm-project/issues/ and include the crash backtrace.
Stack dump:
0.	Program arguments: /tmp/gecko/clang/bin/ld.lld @<!-- -->response.txt
 #<!-- -->0 0x00007f765d87c8a7 llvm::sys::PrintStackTrace(llvm::raw_ostream&amp;, int) (/tmp/gecko/clang/bin/../lib/libLLVM-18git.so+0x28398a7)
 #<!-- -->1 0x00007f765d87cd8b SignalHandler(int) Signals.cpp:0:0
 #<!-- -->2 0x00007f765ab7bfd0 (/lib/x86_64-linux-gnu/libc.so.6+0x3bfd0)
 #<!-- -->3 0x00007f765abcad3c __pthread_kill_implementation ./nptl/pthread_kill.c:44:76
 #<!-- -->4 0x00007f765ab7bf32 raise ./signal/../sysdeps/posix/raise.c:27:6
 #<!-- -->5 0x00007f765ab66472 abort ./stdlib/abort.c:81:7
 #<!-- -->6 0x00007f765d818698 llvm::report_fatal_error(llvm::Twine const&amp;, bool) (/tmp/gecko/clang/bin/../lib/libLLVM-18git.so+0x27d5698)
 #<!-- -->7 0x00007f765cb90bd5 (/tmp/gecko/clang/bin/../lib/libLLVM-18git.so+0x1b4dbd5)
 #<!-- -->8 0x00007f765d96b097 (/tmp/gecko/clang/bin/../lib/libLLVM-18git.so+0x2928097)
 #<!-- -->9 0x00007f765e47c6ed llvm::detail::PassModel&lt;llvm::Module, llvm::VerifierPass, llvm::PreservedAnalyses, llvm::AnalysisManager&lt;llvm::Module&gt;&gt;::run(llvm::Module&amp;, llvm::AnalysisManager&lt;llvm::Module&gt;&amp;) LTOBackend.cpp:0:0
#<!-- -->10 0x00007f765d555bdc llvm::PassManager&lt;llvm::Module, llvm::AnalysisManager&lt;llvm::Module&gt;&gt;::run(llvm::Module&amp;, llvm::AnalysisManager&lt;llvm::Module&gt;&amp;) (/tmp/gecko/clang/bin/../lib/libLLVM-18git.so+0x2512bdc)
#<!-- -->11 0x00007f765e477af4 llvm::lto::opt(llvm::lto::Config const&amp;, llvm::TargetMachine*, unsigned int, llvm::Module&amp;, bool, llvm::ModuleSummaryIndex*, llvm::ModuleSummaryIndex const*, std::vector&lt;unsigned char, std::allocator&lt;unsigned char&gt;&gt; const&amp;) (/tmp/gecko/clang/bin/../lib/libLLVM-18git.so+0x3434af4)
#<!-- -->12 0x00007f765e479594 llvm::lto::backend(llvm::lto::Config const&amp;, std::function&lt;llvm::Expected&lt;std::unique_ptr&lt;llvm::CachedFileStream, std::default_delete&lt;llvm::CachedFileStream&gt;&gt;&gt; (unsigned int, llvm::Twine const&amp;)&gt;, unsigned int, llvm::Module&amp;, llvm::ModuleSummaryIndex&amp;) (/tmp/gecko/clang/bin/../lib/libLLVM-18git.so+0x3436594)
#<!-- -->13 0x00007f765e46f30f llvm::lto::LTO::runRegularLTO(std::function&lt;llvm::Expected&lt;std::unique_ptr&lt;llvm::CachedFileStream, std::default_delete&lt;llvm::CachedFileStream&gt;&gt;&gt; (unsigned int, llvm::Twine const&amp;)&gt;) (/tmp/gecko/clang/bin/../lib/libLLVM-18git.so+0x342c30f)
#<!-- -->14 0x00007f765e46eb73 llvm::lto::LTO::run(std::function&lt;llvm::Expected&lt;std::unique_ptr&lt;llvm::CachedFileStream, std::default_delete&lt;llvm::CachedFileStream&gt;&gt;&gt; (unsigned int, llvm::Twine const&amp;)&gt;, std::function&lt;llvm::Expected&lt;std::function&lt;llvm::Expected&lt;std::unique_ptr&lt;llvm::CachedFileStream, std::default_delete&lt;llvm::CachedFileStream&gt;&gt;&gt; (unsigned int, llvm::Twine const&amp;)&gt;&gt; (unsigned int, llvm::StringRef, llvm::Twine const&amp;)&gt;) (/tmp/gecko/clang/bin/../lib/libLLVM-18git.so+0x342bb73)
#<!-- -->15 0x00005582ad0e6860 lld::elf::BitcodeCompiler::compile() (/tmp/gecko/clang/bin/ld.lld+0x25d860)
#<!-- -->16 0x00005582ad040549 lld::elf::LinkerDriver::link(llvm::opt::InputArgList&amp;) (/tmp/gecko/clang/bin/ld.lld+0x1b7549)
#<!-- -->17 0x00005582ad02fa34 lld::elf::LinkerDriver::linkerMain(llvm::ArrayRef&lt;char const*&gt;) (/tmp/gecko/clang/bin/ld.lld+0x1a6a34)
#<!-- -->18 0x00005582ad02dfe0 lld::elf::link(llvm::ArrayRef&lt;char const*&gt;, llvm::raw_ostream&amp;, llvm::raw_ostream&amp;, bool, bool) (/tmp/gecko/clang/bin/ld.lld+0x1a4fe0)
#<!-- -->19 0x00005582acf72845 lld::unsafeLldMain(llvm::ArrayRef&lt;char const*&gt;, llvm::raw_ostream&amp;, llvm::raw_ostream&amp;, llvm::ArrayRef&lt;lld::DriverDef&gt;, bool) (/tmp/gecko/clang/bin/ld.lld+0xe9845)
#<!-- -->20 0x00005582acf70182 lld_main(int, char**, llvm::ToolContext const&amp;) (/tmp/gecko/clang/bin/ld.lld+0xe7182)
#<!-- -->21 0x00005582acf7004e main (/tmp/gecko/clang/bin/ld.lld+0xe704e)
#<!-- -->22 0x00007f765ab671ca __libc_start_call_main ./csu/../sysdeps/nptl/libc_start_call_main.h:74:3
#<!-- -->23 0x00007f765ab67285 call_init ./csu/../csu/libc-start.c:128:20
#<!-- -->24 0x00007f765ab67285 __libc_start_main ./csu/../csu/libc-start.c:347:5
#<!-- -->25 0x00005582acf71b72 _start (/tmp/gecko/clang/bin/ld.lld+0xe8b72)

Cc: @nikic

(It's also unfortunate that it doesn't tell what specific module is broken)

[nikic]

I've reverted the patch in a72d88f.

I've also reduced the issue to these two files:

llvm-link tmp/gecko/obj-x86_64-pc-linux-gnu/dom/media/platforms/ffmpeg/ffvpx/Unified_cpp_ffmpeg_ffvpx0.o tmp/gecko/obj-x86_64-pc-linux-gnu/dom/media/platforms/ffmpeg/libav53/Unified_cpp_ffmpeg_libav530.o

And then to the following reasonably minimal test case with the files at https://gist.github.com/nikic/ed40c6ddfaac1a3f32b9e8e4f862e9da:

llvm-link test1.ll test2.ll -S > test3.ll

The cause is pretty obvious now: We have a AVPacket types defined in both modules, but with incompatible definitions with different sizes -- presumably there is UB due to ODR violations involved here.

Linking then picks one of those, which is then incompatible with the alloca and verification complains.

Question to the debuginfo folks: What is the expected behavior in this case? Does that just mean verification is just fundamentally doomed due to possible ODR violations in debuginfo? Or should incompatible type definitions not be merged -- somehow, I have no idea if this is even possible?

[dwblaikie]

Question to the debuginfo folks: What is the expected behavior in this case? Does that just mean verification is just fundamentally doomed due to possible ODR violations in debuginfo? Or should incompatible type definitions not be merged -- somehow, I have no idea if this is even possible?

@adrian-prantl @JDevlieghere @pogo59 - I /think/ historically we've let some of these things cause silent problems (like bad/weird debug info, but not a build/link break)), but hard breaks on ODR violations have been avoided?

Though I think we could go the other way, if the errors were more meaningful - like perhaps we could verify that merged types have the same size, if that's important (which it is)? But that has to happen during the merging - not sure if we have an option for "extra verification" during merging or not - maybe it's cheap enough to do all the time.

Such a failure should mention which input files the inconsistent definitions come from, their name, and probably/ideally, their source location (header file+line number). It's not perfect/not enough in some cases (eg: the same type, from the same header, but due to contextual preprocessor definitions or other compiler flags, the definitions come out with different sizes) but hopefully enough to make it actionable.

(total aside: Could be nice to have a "real" ODR checking of some kind - at least Google dropped ODR checking when we switched from GCC to Clang due to the two compilers having a different definition of where a class started (& the ODR checking in gold just checked the definitions came from the same file/line anyway, I think - or maybe it only did functions, now that I think about it... (since we used -gmlt for it))

[adrian-prantl]

I /think/ historically we've let some of these things cause silent problems (like bad/weird debug info, but not a build/link break)), but hard breaks on ODR violations have been avoided?

The compiler is allowed to assume that ODR violations don't happen and I don't think we ever went out of our way to provide any guarantees about the behavior if they do.

Could be nice to have a "real" ODR checking of some kind

You would have to, e.g., build a global DenseSet of all "name-addressed" types that is shared between all MetadataReaders and perform some form of checking (e.g., size, number of members, declcontext) there. I dont' know if we currently deserialize duplicate types from Bitcode, but if we do so already, it may not even be very expensive.