LLVM MemorySSA assertion failure when running with ASAN: "Not the same accesses in the same order"

[topolarity]

On my system (Ubuntu 22.04 WSL2) the following changes are needed for ASAN support to build locally (due partially to #47698):

diff --git a/src/dlload.c b/src/dlload.c
index 4e9e9c9ce4..a01f4f5ea6 100644
--- a/src/dlload.c
+++ b/src/dlload.c
@@ -188,7 +188,7 @@ JL_DLLEXPORT JL_NO_SANITIZE void *jl_dlopen(const char *filename, unsigned flags
         dlopen = (dlopen_prototype*)dlsym(RTLD_NEXT, "dlopen");
         if (!dlopen)
             return NULL;
-        void *libdl_handle = dlopen("libdl.so", RTLD_NOW | RTLD_NOLOAD);
+        void *libdl_handle = dlopen("libdl.so.2", RTLD_NOW | RTLD_NOLOAD);
         assert(libdl_handle);
         dlopen = (dlopen_prototype*)dlsym(libdl_handle, "dlopen");
         dlclose(libdl_handle);
diff --git a/src/julia.h b/src/julia.h
index 2140b0ad0a..f9ed65a729 100644
--- a/src/julia.h
+++ b/src/julia.h
@@ -2118,13 +2118,13 @@ void (ijl_longjmp)(jmp_buf _Buf, int _Value);
 #define jl_setjmp_name "sigsetjmp"
 #endif
 #define jl_setjmp(a,b) sigsetjmp(a,b)
-#if defined(_COMPILER_ASAN_ENABLED_) && __GLIBC__
-// Bypass the ASAN longjmp wrapper - we're unpoisoning the stack ourselves.
-JL_DLLIMPORT int __attribute__ ((nothrow)) (__libc_siglongjmp)(jl_jmp_buf buf, int val);
-#define jl_longjmp(a,b) __libc_siglongjmp(a,b)
-#else
+//#if defined(_COMPILER_ASAN_ENABLED_) && __GLIBC__
+//// Bypass the ASAN longjmp wrapper - we're unpoisoning the stack ourselves.
+//JL_DLLIMPORT int __attribute__ ((nothrow)) (__libc_siglongjmp)(jl_jmp_buf buf, int val);
+//#define jl_longjmp(a,b) __libc_siglongjmp(a,b)
+//#else
 #define jl_longjmp(a,b) siglongjmp(a,b)
-#endif
+//#endif
 #endif

If I then build with contrib/asan/build.sh ./tmp/test-asan -j4 debug the terminal is filled with JIT session error: Duplicate section.

The build completes, but the REPL crashes if you start Julia and hit Enter:

$ ./tmp/test-asan/asan/julia
JIT session error: Duplicate section
JIT session error: Duplicate section
JIT session error: Duplicate section
JIT session error: Duplicate section
JIT session error: Duplicate section
JIT session error: Duplicate section
JIT session error: Duplicate section
JIT session error: Duplicate section
              _
  _       _ _(_)_     |  Documentation: https://docs.julialang.org
 (_)     | (_) (_)    |
  _ _   _| |_  __ _   |  Type "?" for help, "]?" for Pkg help.
 | | | | | | |/ _` |  |
 | | |_| | | | (_| |  |  Version 1.10.0-DEV.1488 (2023-06-14)
_/ |\__'_|_|_|\__'_|  |  Commit 0fb86dd5c5* (0 days old master)
|__/                   |

JIT session error: Duplicate section
JIT session error: Duplicate section
JIT session error: Duplicate section
JIT session error: Duplicate section
JIT session error: Duplicate section
JIT session error: Duplicate section
JIT session error: Duplicate section
JIT session error: Duplicate section
JIT session error: Duplicate section
JIT session error: Duplicate section
JIT session error: Duplicate section
JIT session error: Duplicate section
JIT session error: Duplicate section
JIT session error: Duplicate section
JIT session error: Duplicate section
JIT session error: Duplicate section
JIT session error: Duplicate section
JIT session error: Duplicate section
JIT session error: Duplicate section
JIT session error: Duplicate section
JIT session error: Duplicate section
JIT session error: Duplicate section
JIT session error: Duplicate section
JIT session error: Duplicate section
JIT session error: Duplicate section
JIT session error: Duplicate section
JIT session error: Duplicate section
JIT session error: Duplicate section
JIT session error: Duplicate section
JIT session error: Duplicate section
JIT session error: Duplicate section
JIT session error: Duplicate section
JIT session error: Duplicate section
JIT session error: Duplicate section
JIT session error: Duplicate section
JIT session error: Duplicate section
JIT session error: Duplicate section
JIT session error: Duplicate section
JIT session error: Duplicate section
JIT session error: Duplicate section
JIT session error: Duplicate section
JIT session error: Duplicate section
JIT session error: Duplicate section
JIT session error: Duplicate section
JIT session error: Duplicate section
JIT session error: Duplicate section
JIT session error: Duplicate section
JIT session error: Duplicate section
JIT session error: Duplicate section
JIT session error: Duplicate section
JIT session error: Duplicate section
JIT session error: Duplicate section
JIT session error: Duplicate section
JIT session error: Duplicate section
JIT session error: Duplicate section
JIT session error: Duplicate section
JIT session error: Duplicate section
JIT session error: Duplicate section
JIT session error: Duplicate section
JIT session error: Duplicate section
julia> JIT session error: Duplicate section
JIT session error: Duplicate section
JIT session error: Duplicate section
JIT session error: Duplicate section
JIT session error: Duplicate section
JIT session error: Duplicate section
JIT session error: Duplicate section
JIT session error: Duplicate section
JIT session error: Duplicate section
JIT session error: Duplicate section
JIT session error: Duplicate section
JIT session error: Duplicate section
JIT session error: Duplicate section
JIT session error: Duplicate section
JIT session error: Duplicate section
JIT session error: Duplicate section
JIT session error: Duplicate section
JIT session error: Duplicate section
JIT session error: Duplicate section
JIT session error: Duplicate section
JIT session error: Duplicate section
JIT session error: Duplicate section
JIT session error: Duplicate section
julia>
JIT session error: Duplicate section
JIT session error: Duplicate section
JIT session error: Duplicate section
JIT session error: Duplicate section
JIT session error: Duplicate section
JIT session error: Duplicate section
julia: /workspace/srcdir/llvm-project/llvm/lib/Analysis/MemorySSA.cpp:2079: void llvm::MemorySSA::verifyOrderingDominationAndDefUses(llvm::Function&, llvm::MemorySSA::VerificationLevel) const: Assertion `&*ALI == *AAI && "Not the same accesses in the same order"' failed.

[22631] signal (6.-6): Aborted
in expression starting at none:0
pthread_kill at /lib/x86_64-linux-gnu/libc.so.6 (unknown line)
raise at /lib/x86_64-linux-gnu/libc.so.6 (unknown line)
abort at /lib/x86_64-linux-gnu/libc.so.6 (unknown line)
unknown function (ip: 0x7f8d572d771a)
__assert_fail at /lib/x86_64-linux-gnu/libc.so.6 (unknown line)
_ZNK4llvm9MemorySSA34verifyOrderingDominationAndDefUsesERNS_8FunctionENS0_17VerificationLevelE at /home/topolarity/repos/julia/tmp/test-asan/asan/usr/bin/../lib/libLLVM-15jl.so (unknown line)
_ZNK4llvm9MemorySSA15verifyMemorySSAENS0_17VerificationLevelE at /home/topolarity/repos/julia/tmp/test-asan/asan/usr/bin/../lib/libLLVM-15jl.so (unknown line)
run at /home/topolarity/repos/julia/src/llvm-julia-licm.cpp:412
run at /home/topolarity/repos/julia/tmp/test-asan/asan/usr/include/llvm/IR/PassManagerInternal.h:88
_ZN4llvm11PassManagerINS_4LoopENS_15AnalysisManagerIS1_JRNS_27LoopStandardAnalysisResultsEEEEJS4_RNS_10LPMUpdaterEEE13runSinglePassIS1_St10unique_ptrINS_6detail11PassConceptIS1_S5_JS4_S7_EEESt14default_deleteISD_EEEENS_8OptionalINS_17PreservedAnalysesEEERT_RT0_RS5_S4_S7_RNS_19PassInstrumentationE at /home/topolarity/repos/julia/tmp/test-asan/asan/usr/bin/../lib/libLLVM-15jl.so (unknown line)
_ZN4llvm11PassManagerINS_4LoopENS_15AnalysisManagerIS1_JRNS_27LoopStandardAnalysisResultsEEEEJS4_RNS_10LPMUpdaterEEE24runWithoutLoopNestPassesERS1_RS5_S4_S7_ at /home/topolarity/repos/julia/tmp/test-asan/asan/usr/bin/../lib/libLLVM-15jl.so (unknown line)
_ZN4llvm11PassManagerINS_4LoopENS_15AnalysisManagerIS1_JRNS_27LoopStandardAnalysisResultsEEEEJS4_RNS_10LPMUpdaterEEE3runERS1_RS5_S4_S7_ at /home/topolarity/repos/julia/tmp/test-asan/asan/usr/bin/../lib/libLLVM-15jl.so (unknown line)
run at /home/topolarity/repos/julia/tmp/test-asan/asan/usr/include/llvm/IR/PassManagerInternal.h:88
_ZN4llvm25FunctionToLoopPassAdaptor3runERNS_8FunctionERNS_15AnalysisManagerIS1_JEEE at /home/topolarity/repos/julia/tmp/test-asan/asan/usr/bin/../lib/libLLVM-15jl.so (unknown line)
run at /home/topolarity/repos/julia/tmp/test-asan/asan/usr/include/llvm/IR/PassManagerInternal.h:88
_ZN4llvm11PassManagerINS_8FunctionENS_15AnalysisManagerIS1_JEEEJEE3runERS1_RS3_ at /home/topolarity/repos/julia/tmp/test-asan/asan/usr/bin/../lib/libLLVM-15jl.so (unknown line)
run at /home/topolarity/repos/julia/tmp/test-asan/asan/usr/include/llvm/IR/PassManagerInternal.h:88
_ZN4llvm27ModuleToFunctionPassAdaptor3runERNS_6ModuleERNS_15AnalysisManagerIS1_JEEE at /home/topolarity/repos/julia/tmp/test-asan/asan/usr/bin/../lib/libLLVM-15jl.so (unknown line)
run at /home/topolarity/repos/julia/tmp/test-asan/asan/usr/include/llvm/IR/PassManagerInternal.h:88
_ZN4llvm11PassManagerINS_6ModuleENS_15AnalysisManagerIS1_JEEEJEE3runERS1_RS3_ at /home/topolarity/repos/julia/tmp/test-asan/asan/usr/bin/../lib/libLLVM-15jl.so (unknown line)
run at /home/topolarity/repos/julia/src/pipeline.cpp:665
operator() at /home/topolarity/repos/julia/src/jitlayers.cpp:1218
withModuleDo<(lambda at /home/topolarity/repos/julia/src/jitlayers.cpp:1183:30)> at /home/topolarity/repos/julia/tmp/test-asan/asan/usr/include/llvm/ExecutionEngine/Orc/ThreadSafeModule.h:136
operator() at /home/topolarity/repos/julia/src/jitlayers.cpp:1183
CallImpl<(anonymous namespace)::OptimizerT> at /home/topolarity/repos/julia/tmp/test-asan/asan/usr/include/llvm/ADT/FunctionExtras.h:222
_ZN4llvm3orc16IRTransformLayer4emitESt10unique_ptrINS0_29MaterializationResponsibilityESt14default_deleteIS3_EENS0_16ThreadSafeModuleE at /home/topolarity/repos/julia/tmp/test-asan/asan/usr/bin/../lib/libLLVM-15jl.so (unknown line)
emit at /home/topolarity/repos/julia/src/jitlayers.cpp:690
_ZN4llvm3orc31BasicIRLayerMaterializationUnit11materializeESt10unique_ptrINS0_29MaterializationResponsibilityESt14default_deleteIS3_EE at /home/topolarity/repos/julia/tmp/test-asan/asan/usr/bin/../lib/libLLVM-15jl.so (unknown line)
_ZN4llvm3orc19MaterializationTask3runEv at /home/topolarity/repos/julia/tmp/test-asan/asan/usr/bin/../lib/libLLVM-15jl.so (unknown line)
_ZN4llvm6detail18UniqueFunctionBaseIvJSt10unique_ptrINS_3orc4TaskESt14default_deleteIS4_EEEE8CallImplIPFvS7_EEEvPvRS7_ at /home/topolarity/repos/julia/tmp/test-asan/asan/usr/bin/../lib/libLLVM-15jl.so (unknown line)
_ZN4llvm3orc16ExecutionSession12dispatchTaskESt10unique_ptrINS0_4TaskESt14default_deleteIS3_EE at /home/topolarity/repos/julia/tmp/test-asan/asan/usr/bin/../lib/libLLVM-15jl.so (unknown line)
_ZN4llvm3orc16ExecutionSession22dispatchOutstandingMUsEv at /home/topolarity/repos/julia/tmp/test-asan/asan/usr/bin/../lib/libLLVM-15jl.so (unknown line)
_ZN4llvm3orc16ExecutionSession17OL_completeLookupESt10unique_ptrINS0_21InProgressLookupStateESt14default_deleteIS3_EESt10shared_ptrINS0_23AsynchronousSymbolQueryEESt8functionIFvRKNS_8DenseMapIPNS0_8JITDylibENS_8DenseSetINS0_15SymbolStringPtrENS_12DenseMapInfoISF_vEEEENSG_ISD_vEENS_6detail12DenseMapPairISD_SI_EEEEEE at /home/topolarity/repos/julia/tmp/test-asan/asan/usr/bin/../lib/libLLVM-15jl.so (unknown line)
_ZN4llvm3orc25InProgressFullLookupState8completeESt10unique_ptrINS0_21InProgressLookupStateESt14default_deleteIS3_EE at /home/topolarity/repos/julia/tmp/test-asan/asan/usr/bin/../lib/libLLVM-15jl.so (unknown line)
_ZN4llvm3orc16ExecutionSession19OL_applyQueryPhase1ESt10unique_ptrINS0_21InProgressLookupStateESt14default_deleteIS3_EENS_5ErrorE at /home/topolarity/repos/julia/tmp/test-asan/asan/usr/bin/../lib/libLLVM-15jl.so (unknown line)
_ZN4llvm3orc16ExecutionSession6lookupENS0_10LookupKindERKSt6vectorISt4pairIPNS0_8JITDylibENS0_19JITDylibLookupFlagsEESaIS8_EENS0_15SymbolLookupSetENS0_11SymbolStateENS_15unique_functionIFvNS_8ExpectedINS_8DenseMapINS0_15SymbolStringPtrENS_18JITEvaluatedSymbolENS_12DenseMapInfoISI_vEENS_6detail12DenseMapPairISI_SJ_EEEEEEEEESt8functionIFvRKNSH_IS6_NS_8DenseSetISI_SL_EENSK_IS6_vEENSN_IS6_SV_EEEEEE at /home/topolarity/repos/julia/tmp/test-asan/asan/usr/bin/../lib/libLLVM-15jl.so (unknown line)
_ZN4llvm3orc16ExecutionSession6lookupERKSt6vectorISt4pairIPNS0_8JITDylibENS0_19JITDylibLookupFlagsEESaIS7_EENS0_15SymbolLookupSetENS0_10LookupKindENS0_11SymbolStateESt8functionIFvRKNS_8DenseMapIS5_NS_8DenseSetINS0_15SymbolStringPtrENS_12DenseMapInfoISI_vEEEENSJ_IS5_vEENS_6detail12DenseMapPairIS5_SL_EEEEEE at /home/topolarity/repos/julia/tmp/test-asan/asan/usr/bin/../lib/libLLVM-15jl.so (unknown line)
addModule at /home/topolarity/repos/julia/src/jitlayers.cpp:1497
jl_add_to_ee at /home/topolarity/repos/julia/src/jitlayers.cpp:1934
jl_add_to_ee at /home/topolarity/repos/julia/src/jitlayers.cpp:1913
_jl_compile_codeinst at /home/topolarity/repos/julia/src/jitlayers.cpp:243
jl_generate_fptr_impl at /home/topolarity/repos/julia/src/jitlayers.cpp:493
jl_compile_method_internal at /home/topolarity/repos/julia/src/gf.c:2475
_jl_invoke at /home/topolarity/repos/julia/src/gf.c:2881
ijl_apply_generic at /home/topolarity/repos/julia/src/gf.c:3071
unknown function (ip: 0x7f8d3ac0b356)
unknown function (ip: 0x7f8d3abea45f)
unknown function (ip: 0x7f8d3abe8500)
unknown function (ip: 0x7f8d3abe62b4)
jl_fptr_args at /home/topolarity/repos/julia/src/gf.c:2531
_jl_invoke at /home/topolarity/repos/julia/src/gf.c:2889
ijl_apply_generic at /home/topolarity/repos/julia/src/gf.c:3071
jl_apply at /home/topolarity/repos/julia/src/julia.h:1963
jl_f__call_latest at /home/topolarity/repos/julia/src/builtins.c:812
unknown function (ip: 0x7f8d3ac40266)
unknown function (ip: 0x7f8d3ac40357)
_jl_invoke at /home/topolarity/repos/julia/src/gf.c:2889
ijl_apply_generic at /home/topolarity/repos/julia/src/gf.c:3071
unknown function (ip: 0x7f8d3d4af0db)
unknown function (ip: 0x7f8d3d8c3448)
unknown function (ip: 0x7f8d3d8c42c7)
_jl_invoke at /home/topolarity/repos/julia/src/gf.c:2889
ijl_apply_generic at /home/topolarity/repos/julia/src/gf.c:3071
unknown function (ip: 0x7f8d3d8f2d57)
unknown function (ip: 0x7f8d3d8f00d7)
unknown function (ip: 0x7f8d3d8f00ef)
_jl_invoke at /home/topolarity/repos/julia/src/gf.c:2889
ijl_apply_generic at /home/topolarity/repos/julia/src/gf.c:3071
jl_apply at /home/topolarity/repos/julia/src/julia.h:1963
_start_task at /home/topolarity/repos/julia/src/task.c:1238
Allocations: 2897347 (Pool: 0; Other: 2897347); GC: 10
Allocations: 2897347 (Pool: 0; Other: 2897347); GC: 10
[1]    22631 IOT instruction  ./tmp/test-asan/asan/julia

[topolarity]

Running with JL_FORCE_JITLINK without ASAN does not reproduce this problem - it appears to be related to some other difference when running with ASAN enabled.

[vchuravy]

The duplicated section seems similar to #50044

@lhames have you tried JITLink and the sanitizers?

[lhames]

@vchuravy They didn't work last time I tried (a couple of years back). They went on my unwritten to-do list, but I haven't gotten around to them yet.

I think this is worth opening an LLVM issue for -- at least one other person has asked about it recently.