A number of packages failed to precompile on Julia master (macOS Apple Silicon)

[truedichotomy]

The following packages failed to precompile in julia master on a macOS Apple silicon machine. However, I don't see any obvious pattern that connects their failures.

julia> Pkg.precompile()
Precompiling project...
  ✗ JuliaInterpreter
  ✗ GRIB
  ✗ LoweredCodeUtils
  ✗ Revise
  ✗ Genie
  0 dependencies successfully precompiled in 9 seconds. 490 already precompiled.

ERROR: The following 2 direct dependencies failed to precompile:

GRIB [b16dfd50-4035-11e9-28d4-9dfe17e6779b]

Failed to precompile GRIB [b16dfd50-4035-11e9-28d4-9dfe17e6779b] to /Users/gong/.julia/compiled/v1.9/GRIB/jl_CSGzR8.
ERROR: LoadError: InitError: could not load library "/Users/gong/.julia/artifacts/9ebc5e497534b4f1fb3aed45fce27c53bdb8938f/lib/libeccodes_f90.dylib"
dlopen(/Users/gong/.julia/artifacts/9ebc5e497534b4f1fb3aed45fce27c53bdb8938f/lib/libeccodes_f90.dylib, 0x0001): Library not loaded: @rpath/libquadmath.0.dylib
  Referenced from: /Users/gong/.julia/artifacts/9ebc5e497534b4f1fb3aed45fce27c53bdb8938f/lib/libeccodes_f90.dylib
  Reason: tried: '/Users/gong/.julia/artifacts/9ebc5e497534b4f1fb3aed45fce27c53bdb8938f/lib/../lib/libquadmath.0.dylib' (no such file), '/Users/gong/.julia/artifacts/9ebc5e497534b4f1fb3aed45fce27c53bdb8938f/lib/../lib/libquadmath.0.dylib' (no such file), '/Users/gong/GitHub/julia/usr/lib/libquadmath.0.dylib' (no such file), '/Users/gong/GitHub/julia/usr/bin/../lib/libquadmath.0.dylib' (no such file), '/usr/local/lib/libquadmath.0.dylib' (no such file), '/usr/lib/libquadmath.0.dylib' (no such file)
Stacktrace:
  [1] dlopen(s::String, flags::UInt32; throw_error::Bool)
    @ Base.Libc.Libdl ./libdl.jl:117
  [2] dlopen(s::String, flags::UInt32)
    @ Base.Libc.Libdl ./libdl.jl:117
  [3] macro expansion
    @ ~/.julia/packages/JLLWrappers/QpMQW/src/products/library_generators.jl:54 [inlined]
  [4] __init__()
    @ eccodes_jll ~/.julia/packages/eccodes_jll/nIoAN/src/wrappers/aarch64-apple-darwin-libgfortran5.jl:18
  [5] _include_from_serialized(path::String, depmods::Vector{Any})
    @ Base ./loading.jl:811
  [6] _require_search_from_serialized(pkg::Base.PkgId, sourcepath::String, depth::Int64)
    @ Base ./loading.jl:919
  [7] _require_search_from_serialized
    @ ./loading.jl:892 [inlined]
  [8] _require(pkg::Base.PkgId)
    @ Base ./loading.jl:1194
  [9] _require_prelocked(uuidkey::Base.PkgId)
    @ Base ./loading.jl:1087
 [10] macro expansion
    @ ./loading.jl:1067 [inlined]
 [11] macro expansion
    @ ./lock.jl:223 [inlined]
 [12] require(into::Module, mod::Symbol)
    @ Base ./loading.jl:1031
 [13] include
    @ ./Base.jl:426 [inlined]
 [14] include_package_for_output(pkg::Base.PkgId, input::String, depot_path::Vector{String}, dl_load_path::Vector{String}, load_path::Vector{String}, concrete_deps::Vector{Pair{Base.PkgId, UInt64}}, source::Nothing)
    @ Base ./loading.jl:1418
 [15] top-level scope
    @ stdin:1
during initialization of module eccodes_jll
in expression starting at /Users/gong/.julia/packages/GRIB/6rlik/src/GRIB.jl:1
in expression starting at stdin:1

Genie [c43c736e-a2d1-11e8-161f-af95117fbd1e]

Failed to precompile Genie [c43c736e-a2d1-11e8-161f-af95117fbd1e] to /Users/gong/.julia/compiled/v1.9/Genie/jl_C8DidE.

signal (11): Segmentation fault: 11
in expression starting at /Users/gong/.julia/packages/JuliaInterpreter/wCcib/src/packagedef.jl:150
ml_matches at /Users/gong/GitHub/julia/src/gf.c:2833
_gf_invoke_lookup at /Users/gong/GitHub/julia/src/gf.c:2561 [inlined]
ijl_gf_invoke_lookup at /Users/gong/GitHub/julia/src/gf.c:2573
whichtt at /Users/gong/.julia/packages/JuliaInterpreter/wCcib/src/utils.jl:35
#prepare_call#41 at /Users/gong/.julia/packages/JuliaInterpreter/wCcib/src/construct.jl:240
prepare_call##kw at /Users/gong/.julia/packages/JuliaInterpreter/wCcib/src/construct.jl:233 [inlined]
#determine_method_for_expr#45 at /Users/gong/.julia/packages/JuliaInterpreter/wCcib/src/construct.jl:571
determine_method_for_expr##kw at /Users/gong/.julia/packages/JuliaInterpreter/wCcib/src/construct.jl:563 [inlined]
#enter_call_expr#46 at /Users/gong/.julia/packages/JuliaInterpreter/wCcib/src/construct.jl:612
enter_call_expr at /Users/gong/.julia/packages/JuliaInterpreter/wCcib/src/construct.jl:611 [inlined]
macro expansion at /Users/gong/.julia/packages/JuliaInterpreter/wCcib/src/construct.jl:745 [inlined]
_precompile_ at /Users/gong/.julia/packages/JuliaInterpreter/wCcib/src/precompile.jl:7
unknown function (ip: 0x12a288eb7)
_jl_invoke at /Users/gong/GitHub/julia/src/gf.c:0 [inlined]
ijl_apply_generic at /Users/gong/GitHub/julia/src/gf.c:2549
jl_apply at /Users/gong/GitHub/julia/src/./julia.h:1830 [inlined]
do_call at /Users/gong/GitHub/julia/src/interpreter.c:126
eval_body at /Users/gong/GitHub/julia/src/interpreter.c:0
jl_interpret_toplevel_thunk at /Users/gong/GitHub/julia/src/interpreter.c:750
jl_toplevel_eval_flex at /Users/gong/GitHub/julia/src/toplevel.c:906
jl_toplevel_eval_flex at /Users/gong/GitHub/julia/src/toplevel.c:850
ijl_toplevel_eval at /Users/gong/GitHub/julia/src/toplevel.c:915 [inlined]
ijl_toplevel_eval_in at /Users/gong/GitHub/julia/src/toplevel.c:965
eval at ./boot.jl:368 [inlined]
include_string at ./loading.jl:1295
_jl_invoke at /Users/gong/GitHub/julia/src/gf.c:0 [inlined]
ijl_apply_generic at /Users/gong/GitHub/julia/src/gf.c:2549
_include at ./loading.jl:1352
include at ./Base.jl:426
jfptr_include_43257 at /Users/gong/GitHub/julia/usr/lib/julia/sys.dylib (unknown line)
_jl_invoke at /Users/gong/GitHub/julia/src/gf.c:0 [inlined]
ijl_apply_generic at /Users/gong/GitHub/julia/src/gf.c:2549
jl_apply at /Users/gong/GitHub/julia/src/./julia.h:1830 [inlined]
jl_f__call_latest at /Users/gong/GitHub/julia/src/builtins.c:769
include at /Users/gong/.julia/packages/JuliaInterpreter/wCcib/src/JuliaInterpreter.jl:1
unknown function (ip: 0x104ce40d3)
_jl_invoke at /Users/gong/GitHub/julia/src/gf.c:0 [inlined]
ijl_apply_generic at /Users/gong/GitHub/julia/src/gf.c:2549
jl_apply at /Users/gong/GitHub/julia/src/./julia.h:1830 [inlined]
do_call at /Users/gong/GitHub/julia/src/interpreter.c:126
eval_body at /Users/gong/GitHub/julia/src/interpreter.c:0
jl_interpret_toplevel_thunk at /Users/gong/GitHub/julia/src/interpreter.c:750
jl_toplevel_eval_flex at /Users/gong/GitHub/julia/src/toplevel.c:906
jl_eval_module_expr at /Users/gong/GitHub/julia/src/toplevel.c:203 [inlined]
jl_toplevel_eval_flex at /Users/gong/GitHub/julia/src/toplevel.c:709
jl_toplevel_eval_flex at /Users/gong/GitHub/julia/src/toplevel.c:850
ijl_toplevel_eval at /Users/gong/GitHub/julia/src/toplevel.c:915 [inlined]
ijl_toplevel_eval_in at /Users/gong/GitHub/julia/src/toplevel.c:965
eval at ./boot.jl:368 [inlined]
include_string at ./loading.jl:1295
_jl_invoke at /Users/gong/GitHub/julia/src/gf.c:0 [inlined]
ijl_apply_generic at /Users/gong/GitHub/julia/src/gf.c:2549
_include at ./loading.jl:1352
include at ./Base.jl:426 [inlined]
include_package_for_output at ./loading.jl:1418
jfptr_include_package_for_output_50016 at /Users/gong/GitHub/julia/usr/lib/julia/sys.dylib (unknown line)
_jl_invoke at /Users/gong/GitHub/julia/src/gf.c:0 [inlined]
ijl_apply_generic at /Users/gong/GitHub/julia/src/gf.c:2549
jl_apply at /Users/gong/GitHub/julia/src/./julia.h:1830 [inlined]
do_call at /Users/gong/GitHub/julia/src/interpreter.c:126
eval_body at /Users/gong/GitHub/julia/src/interpreter.c:0
jl_interpret_toplevel_thunk at /Users/gong/GitHub/julia/src/interpreter.c:750
jl_toplevel_eval_flex at /Users/gong/GitHub/julia/src/toplevel.c:906
jl_toplevel_eval_flex at /Users/gong/GitHub/julia/src/toplevel.c:850
ijl_toplevel_eval at /Users/gong/GitHub/julia/src/toplevel.c:915 [inlined]
ijl_toplevel_eval_in at /Users/gong/GitHub/julia/src/toplevel.c:965
jlplt_ijl_toplevel_eval_in_11959 at /Users/gong/GitHub/julia/usr/lib/julia/sys.dylib (unknown line)
eval at ./boot.jl:368 [inlined]
include_string at ./loading.jl:1295
include_string at ./loading.jl:1305
_jl_invoke at /Users/gong/GitHub/julia/src/gf.c:0 [inlined]
ijl_apply_generic at /Users/gong/GitHub/julia/src/gf.c:2549
exec_options at ./client.jl:301
_start at ./client.jl:522
jfptr__start_26160 at /Users/gong/GitHub/julia/usr/lib/julia/sys.dylib (unknown line)
_jl_invoke at /Users/gong/GitHub/julia/src/gf.c:0 [inlined]
ijl_apply_generic at /Users/gong/GitHub/julia/src/gf.c:2549
jl_apply at /Users/gong/GitHub/julia/src/./julia.h:1830 [inlined]
true_main at /Users/gong/GitHub/julia/src/jlapi.c:567
jl_repl_entrypoint at /Users/gong/GitHub/julia/src/jlapi.c:711
Allocations: 5516904 (Pool: 5507851; Big: 9053); GC: 6
ERROR: LoadError: Failed to precompile JuliaInterpreter [aa1ae85d-cabe-5617-a682-6adf51b2e16a] to /Users/gong/.julia/compiled/v1.9/JuliaInterpreter/jl_79tRRA.
Stacktrace:
  [1] error(s::String)
    @ Base ./error.jl:35
  [2] compilecache(pkg::Base.PkgId, path::String, internal_stderr::IO, internal_stdout::IO, ignore_loaded_modules::Bool)
    @ Base ./loading.jl:1569
  [3] compilecache
    @ ./loading.jl:1513 [inlined]
  [4] _require(pkg::Base.PkgId)
    @ Base ./loading.jl:1217
  [5] _require_prelocked(uuidkey::Base.PkgId)
    @ Base ./loading.jl:1087
  [6] macro expansion
    @ ./loading.jl:1067 [inlined]
  [7] macro expansion
    @ ./lock.jl:223 [inlined]
  [8] require(into::Module, mod::Symbol)
    @ Base ./loading.jl:1031
  [9] include
    @ ./Base.jl:426 [inlined]
 [10] include_package_for_output(pkg::Base.PkgId, input::String, depot_path::Vector{String}, dl_load_path::Vector{String}, load_path::Vector{String}, concrete_deps::Vector{Pair{Base.PkgId, UInt64}}, source::String)
    @ Base ./loading.jl:1418
 [11] top-level scope
    @ stdin:1
in expression starting at /Users/gong/.julia/packages/Revise/wjNpr/src/Revise.jl:1
in expression starting at stdin:1
ERROR: LoadError: Failed to precompile Revise [295af30f-e4ad-537b-8983-00126c2a3abe] to /Users/gong/.julia/compiled/v1.9/Revise/jl_8nLkWe.
Stacktrace:
  [1] error(s::String)
    @ Base ./error.jl:35
  [2] compilecache(pkg::Base.PkgId, path::String, internal_stderr::IO, internal_stdout::IO, ignore_loaded_modules::Bool)
    @ Base ./loading.jl:1569
  [3] compilecache
    @ ./loading.jl:1513 [inlined]
  [4] _require(pkg::Base.PkgId)
    @ Base ./loading.jl:1217
  [5] _require_prelocked(uuidkey::Base.PkgId)
    @ Base ./loading.jl:1087
  [6] macro expansion
    @ ./loading.jl:1067 [inlined]
  [7] macro expansion
    @ ./lock.jl:223 [inlined]
  [8] require(into::Module, mod::Symbol)
    @ Base ./loading.jl:1031
  [9] include
    @ ./Base.jl:426 [inlined]
 [10] include_package_for_output(pkg::Base.PkgId, input::String, depot_path::Vector{String}, dl_load_path::Vector{String}, load_path::Vector{String}, concrete_deps::Vector{Pair{Base.PkgId, UInt64}}, source::Nothing)
    @ Base ./loading.jl:1418
 [11] top-level scope
    @ stdin:1
in expression starting at /Users/gong/.julia/packages/Genie/drXWm/src/Genie.jl:1
in expression starting at stdin:1

Stacktrace:
 [1] pkgerror(msg::String)
   @ Pkg.Types ~/GitHub/julia/usr/share/julia/stdlib/v1.9/Pkg/src/Types.jl:67
 [2] precompile(ctx::Pkg.Types.Context, pkgs::Vector{String}; internal_call::Bool, strict::Bool, warn_loaded::Bool, already_instantiated::Bool, kwargs::Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
   @ Pkg.API ~/GitHub/julia/usr/share/julia/stdlib/v1.9/Pkg/src/API.jl:1427
 [3] precompile
   @ ~/GitHub/julia/usr/share/julia/stdlib/v1.9/Pkg/src/API.jl:1060 [inlined]
 [4] #precompile#225
   @ ~/GitHub/julia/usr/share/julia/stdlib/v1.9/Pkg/src/API.jl:1057 [inlined]
 [5] precompile (repeats 2 times)
   @ ~/GitHub/julia/usr/share/julia/stdlib/v1.9/Pkg/src/API.jl:1057 [inlined]
 [6] top-level scope
   @ REPL[4]:1

versioninfo():

julia> versioninfo()
Julia Version 1.9.0-DEV.149
Commit 9a48dc1df4 (2022-03-07 04:16 UTC)
Platform Info:
  OS: macOS (arm64-apple-darwin21.3.0)
  CPU: 8 × Apple M1
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-13.0.1 (ORCJIT, apple-m1)
  Threads: 1 on 4 virtual cores

[Keno]

I'm seeing this also.

[maleadt]

Bisected to #44448, because JuliaInterpreter calls jl_gf_invoke_lookup which changed in that PR https://github.com/JuliaDebug/JuliaInterpreter.jl/blob/1a92d9a083e193d0be09bd7c9cb8bbb3a386c30b/src/utils.jl#L35-L41
So basically this is fixed already and just needs a JuliaInterpreter tag.

[Keno]

Ok, so JuliaInterpreter just needs a new release. @aviatesk Can you change JuliaInterpreter to call into Core.Compiler instead? That way it'll pick up future changes on the C side automatically or at least throw a nicer error.

[aviatesk]

JuliaInterpreter@0.9.8 is on its way.