invalid instruction on 1.9.3 when `using LLVM`

[bjarthur]

i have an on-premise cluster with a shared filesystem. on julia 1.8, using CUDA works just fine when i switch back and forth between a machine with a Nvidia V100 and a separate machine with an A100. but on 1.9-rc3, while it properly recompiles if i first do it on the A100 machine, and then switch to the V100, if i do it the other way around (first V100 where it compiles, and then switch to A100) i get the error below instead of a nice message that it is precompiling. seemingly julia doesn't understand that it needs to recompile.

julia> using CUDA
Invalid instruction at 0x14bd6b8e32f8: 0x62, 0xf1, 0x7d, 0x08, 0x76, 0xc1, 0xc5, 0xf8, 0x44, 0xc0, 0xc5, 0xfb, 0x93, 0xc0, 0xa8

[2060742] signal (4.2): Illegal instruction
in expression starting at REPL[1]:1
__init__ at /groups/scicompsoft/home/arthurb/.julia/packages/LLVM/TLGyi/src/LLVM.jl:103
jfptr___init___863 at /groups/scicompsoft/home/arthurb/.julia/compiled/v1.9/LLVM/e8NBy_sTSrY.so (unknown line)
_jl_invoke at /cache/build/default-amdci4-4/julialang/julia-release-1-dot-9/src/gf.c:2758 [inlined]
ijl_apply_generic at /cache/build/default-amdci4-4/julialang/julia-release-1-dot-9/src/gf.c:2940
jl_apply at /cache/build/default-amdci4-4/julialang/julia-release-1-dot-9/src/julia.h:1879 [inlined]
jl_module_run_initializer at /cache/build/default-amdci4-4/julialang/julia-release-1-dot-9/src/toplevel.c:75
ijl_init_restored_modules at /cache/build/default-amdci4-4/julialang/julia-release-1-dot-9/src/module.c:982
register_restored_modules at ./loading.jl:1074
_include_from_serialized at ./loading.jl:1020
_tryrequire_from_serialized at ./loading.jl:1356
_require_search_from_serialized at ./loading.jl:1459
_require at ./loading.jl:1748
_require_prelocked at ./loading.jl:1625
macro expansion at ./loading.jl:1613 [inlined]
macro expansion at ./lock.jl:267 [inlined]
require at ./loading.jl:1576
jfptr_require_51256.clone_1 at /groups/scicompsoft/home/arthurb/.julia/juliaup/julia-1.9.0-rc3+0.x64.linux.gnu/lib/julia/sys.so (unknown line)
_jl_invoke at /cache/build/default-amdci4-4/julialang/julia-release-1-dot-9/src/gf.c:2758 [inlined]
ijl_apply_generic at /cache/build/default-amdci4-4/julialang/julia-release-1-dot-9/src/gf.c:2940
jl_apply at /cache/build/default-amdci4-4/julialang/julia-release-1-dot-9/src/julia.h:1879 [inlined]
call_require at /cache/build/default-amdci4-4/julialang/julia-release-1-dot-9/src/toplevel.c:466 [inlined]
eval_import_path at /cache/build/default-amdci4-4/julialang/julia-release-1-dot-9/src/toplevel.c:503
jl_toplevel_eval_flex at /cache/build/default-amdci4-4/julialang/julia-release-1-dot-9/src/toplevel.c:731
jl_toplevel_eval_flex at /cache/build/default-amdci4-4/julialang/julia-release-1-dot-9/src/toplevel.c:856
ijl_toplevel_eval_in at /cache/build/default-amdci4-4/julialang/julia-release-1-dot-9/src/toplevel.c:971
<SNIP>

[maleadt]

Can you reproduce this when just loading LLVM.jl, which seems like what's failing here?
Also, since I can't reproduce, maybe you could record an rr trace using julia --bug-report=rr (again, preferably when only loading LLVM.jl, not all of CUDA.jl).

[Keno]

between a machine with a Nvidia V100 and a separate machine with an A100

What are the CPU models of these two machines?

[bjarthur]

the V100 is Intel(R) Xeon(R) Platinum 8160 CPU @ 2.10GHz, the A100 is AMD EPYC 7413 24-Core Processor. i can provide the full output of cat /proc/cpuinfo if you need more details.

[bjarthur]

am having trouble recording an rr trace. after adding BugReporting (v0.3.2), i envoke julia with JULIA_RR_RECORD_ARGS=-n julia --bug-report=rr. it reports "Saving execution to trace directory /tmp/jl_csVrsA/julia-0". i ctrl-Z out and see a bunch of stuff there. then i using LLVM (instead of CUDA) and get the same error as above. immediately after that error is this one:

### IMPORTANT =============================================================
You are about to upload a trace directory to a publicly accessible location.
Such traces contain any information that was accessed by the traced
executable during its execution. This includes any code loaded, any
secrets entered, the contents of any configuration files, etc.

DO NOT proceed, if you do not wish to make this information publicly available.
By proceeding you explicitly agree to waive any privacy interest in the
uploaded information.
### =======================================================================

WARNING: both Exceptions and ConcurrentUtilities export "try_with_timeout"; uses of it in module Connections must be qualified
ERROR: TaskFailedException
Stacktrace:
  [1] try_yieldto(undo::typeof(Base.ensure_rescheduled))
    @ Base ./task.jl:920
  [2] wait()
    @ Base ./task.jl:984
  [3] wait(c::Base.GenericCondition{ReentrantLock}; first::Bool)
    @ Base ./condition.jl:130
  [4] wait
    @ ./condition.jl:125 [inlined]
  [5] take_unbuffered(c::Channel{Any})
    @ Base ./channels.jl:473
  [6] take!
    @ ./channels.jl:450 [inlined]
  [7] get_upload_params()
    @ BugReporting ~/.julia/packages/BugReporting/Qj6TJ/src/BugReporting.jl:618
  [8] (::BugReporting.var"#33#35"{Vector{String}})(trace_dir::String)
    @ BugReporting ~/.julia/packages/BugReporting/Qj6TJ/src/BugReporting.jl:549
  [9] mktempdir(fn::BugReporting.var"#33#35"{Vector{String}}, parent::String; prefix::String)
    @ Base.Filesystem ./file.jl:760
 [10] mktempdir (repeats 2 times)
    @ ./file.jl:756 [inlined]
 [11] make_interactive_report(report_arg::String, ARGS::Vector{String})
    @ BugReporting ~/.julia/packages/BugReporting/Qj6TJ/src/BugReporting.jl:545
<SNIP>

julia then exits, and i try to look in the "/tmp/jl_csVrsA/julia-0" again and it does not exist.

[giordano]

Feels like the problem is that you're attempting to load native code compiled for an ISA (skylake) on a CPU with a smaller ISA (zen3), and the fact that this happens only on v1.9 is because only on this version packages cache native code.

@vchuravy is the compiled cache indexed by the CPU target?

[Keno]

@vchuravy is the compiled cache indexed by the CPU target?

It's supposed to be, yes, but it's possible something is off with the microarchitecture feature tracking for that.

[giordano]

It's supposed to be, yes

Where exactly? Skimming through the code I could only find

julia/base/loading.jl

Lines 2175 to 2178 in b9b8b38

	cpu_target = get(ENV, "JULIA_CPU_TARGET", nothing)
	if cpu_target === nothing
	cpu_target = unsafe_string(JLOptions().cpu_target)
	end

but if I remember correctly the default unsafe_string(JLOptions().cpu_target) is just native.

[vchuravy]

What is JULIA_DEBUG=loading. We should be writing out the target features we used... And discard anything that can't be loaded.

https://github.com/JuliaLang/julia/blob/b9b8b38ec09b4e91b46fa833c609e0cf2b9eba45/base/loading.jl#LL2788C1-L2788C20

[bjarthur]

maybe a 1.9 milestone tag for this? would be nice to see it fixed in a point release. thx.

[KristofferC]

How would a 1.9 milestone tag help with anything here? 1.9 is already released.

[bjarthur]

@mkitti

[ViralBShah]

Please reopen if still an issue with the latest 1.9.x