Error during the `asan` CI job: "LLVM ERROR: inconsistency in registered CommandLine options"

[DilumAluthge]

This issue is to track the progress on fixing the "LLVM ERROR: inconsistency in registered CommandLine options" error seen in the asan CI job.

    LINK tmp/test-asan/asan/usr/lib/libjulia-codegen-debug.so.1
    LINK tmp/test-asan/asan/usr/lib/libjulia-codegen-debug.so
make[1]: Leaving directory '/cache/build/amdci8-0/julialang/julia-master/tmp/test-asan/asan/src'
make[1]: Entering directory '/cache/build/amdci8-0/julialang/julia-master/tmp/test-asan/asan'
    JULIA tmp/test-asan/asan/usr/lib/julia/corecompiler.ji
: CommandLine Error: Option 'help-list' registered more than once!
LLVM ERROR: inconsistency in registered CommandLine options
Aborted
make[1]: *** [/cache/build/amdci8-0/julialang/julia-master/sysimage.mk:61: /cache/build/amdci8-0/julialang/julia-master/tmp/test-asan/asan/usr/lib/julia/corecompiler.ji] Error 134
make[1]: Leaving directory '/cache/build/amdci8-0/julialang/julia-master/tmp/test-asan/asan'
make: *** [/cache/build/amdci8-0/julialang/julia-master/Makefile:82: julia-sysimg-ji] Error 2
make: Leaving directory '/cache/build/amdci8-0/julialang/julia-master/tmp/test-asan/asan'
🚨 Error: The command exited with status 2

Example log: https://buildkite.com/julialang/julia-master/builds/4455#3d8f2204-c0ae-43b8-a232-cfeb2a5be0c6

Introduced in #41936

[vchuravy]

So what this feel like is like we load two different LLVM libraries with the symbol versions being equal. When someone looks into it, I would be curious to see a backtraces on that error.

[tkf]

I tried this with LLVM_SANITIZE=1 and LLVM_DEBUG=1. I get an ODR violation instead:

    JULIA usr/lib/julia/corecompiler.ji
=================================================================
==16275==ERROR: AddressSanitizer: odr-violation (0x7f53ceff4620):
  [1] size=4 'llvm::DisableABIBreakingChecks' /home/arakaki/repos/watch/_wt/julia/llvm-asan-debug/deps/srccache/llvm-julia-12.0.1-4/llvm/lib/Support/ABIBreak.cpp:20:5
  [2] size=4 'llvm::DisableABIBreakingChecks' /home/arakaki/repos/watch/_wt/julia/llvm-asan-debug/deps/srccache/llvm-julia-12.0.1-4/llvm/lib/Support/ABIBreak.cpp:20:5
These globals were registered at these points:
  [1]:
    #0 0x42bf87 in __asan_register_globals /home/arakaki/repos/watch/_wt/julia/toolchain-asan-debug/deps/srccache/llvm-julia-12.0.1-4/compiler-rt/lib/asan/asan_globals.cpp:360:3
    #1 0x7f53c03f3f5b in asan.module_ctor (/home/arakaki/repos/watch/_wt/julia/llvm-asan-debug/usr/bin/../lib/libLLVM-12jl.so+0x3324f5b)

  [2]:
    #0 0x42bf87 in __asan_register_globals /home/arakaki/repos/watch/_wt/julia/toolchain-asan-debug/deps/srccache/llvm-julia-12.0.1-4/compiler-rt/lib/asan/asan_globals.cpp:360:3
    #1 0x7f53ce65a30b in asan.module_ctor (/home/arakaki/repos/watch/_wt/julia/llvm-asan-debug/usr/bin/../lib/libjulia-internal-debug.so.1+0x42a30b)

If I disable the ODR violation detection with make -j32 ASAN_OPTIONS=detect_leaks=0:fast_unwind_on_malloc=0:allow_user_segv_handler=1:malloc_context_size=2:detect_odr_violation=0, I get

julia-debug: /home/arakaki/repos/watch/_wt/julia/llvm-asan-debug/deps/srccache/llvm-julia-12.0.1-4/llvm/lib/Support/CommandLine.cpp:361: void (anonymous namespace)::CommandLineParser::registerCategory(llvm::cl::OptionCategory *): Assertion `count_if(RegisteredOptionCategories, [cat](const OptionCategory *Category) { return cat->getName() == Category->getName(); }) == 0 && "Duplicate option categories"' failed.

It's not quite the same as the error in the OP but it looks like it's coming from the same place?

(Edit: hmm... or is it just telling me that there's something wrong in my build configuration?)

[vtjnash]

@JeffBezanson I think we either need to fix this or revert the codegen-split PR. We've already gone more than a month with CI disabled due to this.

[DilumAluthge]

Just to clarify, it's only the asan CI job that has been marked as "allow failures". All of the other Buildkite CI jobs are running normally (with failures disallowed).

That being said, I do think it would be nice to fix this failure so that we can switch the asan CI job to run normally.

[tkf]

this feel like is like we load two different LLVM libraries with the symbol versions being equal

I tried LD_DEBUG=all ../usr/bin/julia-debug -C "native" --output-ji ../usr/lib/julia/corecompiler.ji.tmp --startup-file=no --warn-overwrite=yes -g0 -O0 compiler/compiler.jl 2>&1 | grep -v 'trying file=' | grep -Eo '/[^ ]*/libLLVM[^ ]*.so' | sort -u to see if this is true. But it prints a single path /home/arakaki/repos/watch/_wt/julia/llvm-asan-debug/usr/bin/../lib/libLLVM-12jl.so which indicates that we only load one libLLVM?

Also, @vtjnash mentioned that LLVM 13 may fix it #42602

So maybe we are hitting an edge case in LLVM 12? I'll be happy if 13 fixes it anyway though :)