Assertion failure in `lookup_root()`

[kpamnany]

Running with FORCE_ASSERTIONS=1, we're seeing either:

julia: /tmp/nix-build-julia-1.9.0-beta4-patched.drv-0/source/src/method.c:1233: lookup_root: Assertion `key == 0' failed.

Or:

julia: /tmp/nix-build-julia-1.9.0-beta4-patched.drv-0/source/src/julia.h:1022: jl_array_ptr_ref: Assertion `i < jl_array_len(a)' failed.

Quite regularly. A full stack trace for one of these is below. Still trying to get an rr trace, but hopefully this will be useful anyway.

Full stack trace.

[9681] signal (6.-6): Aborted
in expression starting at /tmp/nix-build-raicode-tests.sorting.drv-0/rai-server-source/.julia/packages/HiGHS/QRk9O/src/MOI_wrapper.jl:246
gsignal at /nix/store/0xxjx37fcy2nl3yz6igmv4mag2a7giq6-glibc-2.33-123/lib/libc.so.6 (unknown line)
abort at /nix/store/0xxjx37fcy2nl3yz6igmv4mag2a7giq6-glibc-2.33-123/lib/libc.so.6 (unknown line)
__assert_fail_base.cold.0 at /nix/store/0xxjx37fcy2nl3yz6igmv4mag2a7giq6-glibc-2.33-123/lib/libc.so.6 (unknown line)
__assert_fail at /nix/store/0xxjx37fcy2nl3yz6igmv4mag2a7giq6-glibc-2.33-123/lib/libc.so.6 (unknown line)
jl_array_ptr_ref at /tmp/nix-build-julia-1.9.0-beta4-patched.drv-0/source/src/julia.h:1022
jl_array_ptr_ref at /tmp/nix-build-julia-1.9.0-beta4-patched.drv-0/source/src/method.c:1239 [inlined]
lookup_root at /tmp/nix-build-julia-1.9.0-beta4-patched.drv-0/source/src/method.c:1234
jl_decode_value at /tmp/nix-build-julia-1.9.0-beta4-patched.drv-0/source/src/ircode.c:737
jl_decode_value_expr at /tmp/nix-build-julia-1.9.0-beta4-patched.drv-0/source/src/ircode.c:555 [inlined]
jl_decode_value at /tmp/nix-build-julia-1.9.0-beta4-patched.drv-0/source/src/ircode.c:679
jl_decode_value_array at /tmp/nix-build-julia-1.9.0-beta4-patched.drv-0/source/src/ircode.c:498
ijl_uncompress_ir at /tmp/nix-build-julia-1.9.0-beta4-patched.drv-0/source/src/ircode.c:892
retrieve_ir_for_inlining at ./compiler/ssair/inlining.jl:986 [inlined]
#resolve_todo#463 at ./compiler/ssair/inlining.jl:890
resolve_todo at ./compiler/ssair/inlining.jl:840 [inlined]
#analyze_method!#464 at ./compiler/ssair/inlining.jl:982
analyze_method! at ./compiler/ssair/inlining.jl:943 [inlined]
#handle_match!#468 at ./compiler/ssair/inlining.jl:1460
handle_match! at ./compiler/ssair/inlining.jl:1450 [inlined]
#handle_any_const_result!#465 at ./compiler/ssair/inlining.jl:1322
handle_any_const_result! at ./compiler/ssair/inlining.jl:1304 [inlined]
compute_inlining_cases at ./compiler/ssair/inlining.jl:1397
handle_call! at ./compiler/ssair/inlining.jl:1443 [inlined]
assemble_inline_todo! at ./compiler/ssair/inlining.jl:1669
ssa_inlining_pass! at ./compiler/ssair/inlining.jl:79 [inlined]
run_passes at ./compiler/optimize.jl:539
run_passes at ./compiler/optimize.jl:554 [inlined]
optimize at ./compiler/optimize.jl:503 [inlined]
_typeinf at ./compiler/typeinfer.jl:271
typeinf at ./compiler/typeinfer.jl:215
typeinf_edge at ./compiler/typeinfer.jl:947
abstract_call_method at ./compiler/abstractinterpretation.jl:609
abstract_call_gf_by_type at ./compiler/abstractinterpretation.jl:153
abstract_call_known at ./compiler/abstractinterpretation.jl:1916
jfptr_abstract_call_known_16518.clone_1 at /nix/store/prmmzflxy02wajsrvw3k0jf1s8rixigy-julia-1.9.0-beta4-patched/lib/julia/sys.so (unknown line)
_jl_invoke at /tmp/nix-build-julia-1.9.0-beta4-patched.drv-0/source/src/gf.c:2670 [inlined]
ijl_invoke at /tmp/nix-build-julia-1.9.0-beta4-patched.drv-0/source/src/gf.c:2696
tojlinvoke21222.clone_1 at /nix/store/prmmzflxy02wajsrvw3k0jf1s8rixigy-julia-1.9.0-beta4-patched/lib/julia/sys.so (unknown line)
j_abstract_call_known_13421.clone_1 at /nix/store/prmmzflxy02wajsrvw3k0jf1s8rixigy-julia-1.9.0-beta4-patched/lib/julia/sys.so (unknown line)
abstract_call at ./compiler/abstractinterpretation.jl:1987
abstract_call at ./compiler/abstractinterpretation.jl:1966
abstract_eval_statement_expr at ./compiler/abstractinterpretation.jl:2132
abstract_eval_statement at ./compiler/abstractinterpretation.jl:2346
abstract_eval_basic_statement at ./compiler/abstractinterpretation.jl:2558
typeinf_local at ./compiler/abstractinterpretation.jl:2733
typeinf_nocycle at ./compiler/abstractinterpretation.jl:2839
_typeinf at ./compiler/typeinfer.jl:244
typeinf at ./compiler/typeinfer.jl:215
typeinf_edge at ./compiler/typeinfer.jl:947
abstract_call_method at ./compiler/abstractinterpretation.jl:609
abstract_call_gf_by_type at ./compiler/abstractinterpretation.jl:153
abstract_call_known at ./compiler/abstractinterpretation.jl:1916
jfptr_abstract_call_known_16518.clone_1 at /nix/store/prmmzflxy02wajsrvw3k0jf1s8rixigy-julia-1.9.0-beta4-patched/lib/julia/sys.so (unknown line)
_jl_invoke at /tmp/nix-build-julia-1.9.0-beta4-patched.drv-0/source/src/gf.c:2670 [inlined]
ijl_invoke at /tmp/nix-build-julia-1.9.0-beta4-patched.drv-0/source/src/gf.c:2696
tojlinvoke21222.clone_1 at /nix/store/prmmzflxy02wajsrvw3k0jf1s8rixigy-julia-1.9.0-beta4-patched/lib/julia/sys.so (unknown line)
j_abstract_call_known_13421.clone_1 at /nix/store/prmmzflxy02wajsrvw3k0jf1s8rixigy-julia-1.9.0-beta4-patched/lib/julia/sys.so (unknown line)
abstract_call at ./compiler/abstractinterpretation.jl:1987
abstract_call at ./compiler/abstractinterpretation.jl:1966
abstract_eval_statement_expr at ./compiler/abstractinterpretation.jl:2132
abstract_eval_statement at ./compiler/abstractinterpretation.jl:2346
abstract_eval_basic_statement at ./compiler/abstractinterpretation.jl:2558
typeinf_local at ./compiler/abstractinterpretation.jl:2733
typeinf_nocycle at ./compiler/abstractinterpretation.jl:2839
_typeinf at ./compiler/typeinfer.jl:244
typeinf at ./compiler/typeinfer.jl:215
typeinf_edge at ./compiler/typeinfer.jl:947
abstract_call_method at ./compiler/abstractinterpretation.jl:609
abstract_call_gf_by_type at ./compiler/abstractinterpretation.jl:153
abstract_call_known at ./compiler/abstractinterpretation.jl:1916
jfptr_abstract_call_known_16518.clone_1 at /nix/store/prmmzflxy02wajsrvw3k0jf1s8rixigy-julia-1.9.0-beta4-patched/lib/julia/sys.so (unknown line)
_jl_invoke at /tmp/nix-build-julia-1.9.0-beta4-patched.drv-0/source/src/gf.c:2670 [inlined]
ijl_invoke at /tmp/nix-build-julia-1.9.0-beta4-patched.drv-0/source/src/gf.c:2696
tojlinvoke21222.clone_1 at /nix/store/prmmzflxy02wajsrvw3k0jf1s8rixigy-julia-1.9.0-beta4-patched/lib/julia/sys.so (unknown line)
j_abstract_call_known_13421.clone_1 at /nix/store/prmmzflxy02wajsrvw3k0jf1s8rixigy-julia-1.9.0-beta4-patched/lib/julia/sys.so (unknown line)
abstract_call at ./compiler/abstractinterpretation.jl:1987
abstract_call at ./compiler/abstractinterpretation.jl:1966
abstract_eval_statement_expr at ./compiler/abstractinterpretation.jl:2132
abstract_eval_statement at ./compiler/abstractinterpretation.jl:2346
abstract_eval_basic_statement at ./compiler/abstractinterpretation.jl:2558
typeinf_local at ./compiler/abstractinterpretation.jl:2733
typeinf_nocycle at ./compiler/abstractinterpretation.jl:2839
_typeinf at ./compiler/typeinfer.jl:244
typeinf at ./compiler/typeinfer.jl:215
typeinf_edge at ./compiler/typeinfer.jl:947
abstract_call_method at ./compiler/abstractinterpretation.jl:609
abstract_call_gf_by_type at ./compiler/abstractinterpretation.jl:153
abstract_call_known at ./compiler/abstractinterpretation.jl:1916
jfptr_abstract_call_known_16518.clone_1 at /nix/store/prmmzflxy02wajsrvw3k0jf1s8rixigy-julia-1.9.0-beta4-patched/lib/julia/sys.so (unknown line)
_jl_invoke at /tmp/nix-build-julia-1.9.0-beta4-patched.drv-0/source/src/gf.c:2670 [inlined]
ijl_invoke at /tmp/nix-build-julia-1.9.0-beta4-patched.drv-0/source/src/gf.c:2696
tojlinvoke21222.clone_1 at /nix/store/prmmzflxy02wajsrvw3k0jf1s8rixigy-julia-1.9.0-beta4-patched/lib/julia/sys.so (unknown line)
j_abstract_call_known_13421.clone_1 at /nix/store/prmmzflxy02wajsrvw3k0jf1s8rixigy-julia-1.9.0-beta4-patched/lib/julia/sys.so (unknown line)
abstract_call at ./compiler/abstractinterpretation.jl:1987
abstract_call at ./compiler/abstractinterpretation.jl:1966
abstract_eval_statement_expr at ./compiler/abstractinterpretation.jl:2132
abstract_eval_statement at ./compiler/abstractinterpretation.jl:2346
abstract_eval_basic_statement at ./compiler/abstractinterpretation.jl:2558
typeinf_local at ./compiler/abstractinterpretation.jl:2733
typeinf_nocycle at ./compiler/abstractinterpretation.jl:2839
_typeinf at ./compiler/typeinfer.jl:244
typeinf at ./compiler/typeinfer.jl:215
typeinf_edge at ./compiler/typeinfer.jl:947
abstract_call_method at ./compiler/abstractinterpretation.jl:609
abstract_call_gf_by_type at ./compiler/abstractinterpretation.jl:153
abstract_call_known at ./compiler/abstractinterpretation.jl:1916
jfptr_abstract_call_known_16518.clone_1 at /nix/store/prmmzflxy02wajsrvw3k0jf1s8rixigy-julia-1.9.0-beta4-patched/lib/julia/sys.so (unknown line)
_jl_invoke at /tmp/nix-build-julia-1.9.0-beta4-patched.drv-0/source/src/gf.c:2670 [inlined]
ijl_invoke at /tmp/nix-build-julia-1.9.0-beta4-patched.drv-0/source/src/gf.c:2696
tojlinvoke21222.clone_1 at /nix/store/prmmzflxy02wajsrvw3k0jf1s8rixigy-julia-1.9.0-beta4-patched/lib/julia/sys.so (unknown line)
j_abstract_call_known_13421.clone_1 at /nix/store/prmmzflxy02wajsrvw3k0jf1s8rixigy-julia-1.9.0-beta4-patched/lib/julia/sys.so (unknown line)
abstract_call at ./compiler/abstractinterpretation.jl:1987
abstract_call at ./compiler/abstractinterpretation.jl:1966
abstract_eval_statement_expr at ./compiler/abstractinterpretation.jl:2132
abstract_eval_statement at ./compiler/abstractinterpretation.jl:2346
abstract_eval_basic_statement at ./compiler/abstractinterpretation.jl:2558
typeinf_local at ./compiler/abstractinterpretation.jl:2733
typeinf_nocycle at ./compiler/abstractinterpretation.jl:2839
_typeinf at ./compiler/typeinfer.jl:244
typeinf at ./compiler/typeinfer.jl:215
typeinf_edge at ./compiler/typeinfer.jl:947
abstract_call_method at ./compiler/abstractinterpretation.jl:609
abstract_call_gf_by_type at ./compiler/abstractinterpretation.jl:153
abstract_call_known at ./compiler/abstractinterpretation.jl:1916
jfptr_abstract_call_known_16518.clone_1 at /nix/store/prmmzflxy02wajsrvw3k0jf1s8rixigy-julia-1.9.0-beta4-patched/lib/julia/sys.so (unknown line)
_jl_invoke at /tmp/nix-build-julia-1.9.0-beta4-patched.drv-0/source/src/gf.c:2670 [inlined]
ijl_invoke at /tmp/nix-build-julia-1.9.0-beta4-patched.drv-0/source/src/gf.c:2696
tojlinvoke21222.clone_1 at /nix/store/prmmzflxy02wajsrvw3k0jf1s8rixigy-julia-1.9.0-beta4-patched/lib/julia/sys.so (unknown line)
j_abstract_call_known_13421.clone_1 at /nix/store/prmmzflxy02wajsrvw3k0jf1s8rixigy-julia-1.9.0-beta4-patched/lib/julia/sys.so (unknown line)
abstract_call at ./compiler/abstractinterpretation.jl:1987
abstract_call at ./compiler/abstractinterpretation.jl:1966
abstract_eval_statement_expr at ./compiler/abstractinterpretation.jl:2132
abstract_eval_statement at ./compiler/abstractinterpretation.jl:2346
abstract_eval_basic_statement at ./compiler/abstractinterpretation.jl:2558
typeinf_local at ./compiler/abstractinterpretation.jl:2733
typeinf_nocycle at ./compiler/abstractinterpretation.jl:2839
_typeinf at ./compiler/typeinfer.jl:244
typeinf at ./compiler/typeinfer.jl:215
typeinf_edge at ./compiler/typeinfer.jl:947
abstract_call_method at ./compiler/abstractinterpretation.jl:609
abstract_call_gf_by_type at ./compiler/abstractinterpretation.jl:153
abstract_call_known at ./compiler/abstractinterpretation.jl:1916
jfptr_abstract_call_known_16518.clone_1 at /nix/store/prmmzflxy02wajsrvw3k0jf1s8rixigy-julia-1.9.0-beta4-patched/lib/julia/sys.so (unknown line)
_jl_invoke at /tmp/nix-build-julia-1.9.0-beta4-patched.drv-0/source/src/gf.c:2670 [inlined]
ijl_invoke at /tmp/nix-build-julia-1.9.0-beta4-patched.drv-0/source/src/gf.c:2696
tojlinvoke21222.clone_1 at /nix/store/prmmzflxy02wajsrvw3k0jf1s8rixigy-julia-1.9.0-beta4-patched/lib/julia/sys.so (unknown line)
j_abstract_call_known_13421.clone_1 at /nix/store/prmmzflxy02wajsrvw3k0jf1s8rixigy-julia-1.9.0-beta4-patched/lib/julia/sys.so (unknown line)
abstract_call at ./compiler/abstractinterpretation.jl:1987
abstract_call at ./compiler/abstractinterpretation.jl:1966
abstract_eval_statement_expr at ./compiler/abstractinterpretation.jl:2132
abstract_eval_statement at ./compiler/abstractinterpretation.jl:2346
abstract_eval_basic_statement at ./compiler/abstractinterpretation.jl:2558
typeinf_local at ./compiler/abstractinterpretation.jl:2733
typeinf_nocycle at ./compiler/abstractinterpretation.jl:2839
_typeinf at ./compiler/typeinfer.jl:244
typeinf at ./compiler/typeinfer.jl:215
typeinf_edge at ./compiler/typeinfer.jl:947
abstract_call_method at ./compiler/abstractinterpretation.jl:609
abstract_call_gf_by_type at ./compiler/abstractinterpretation.jl:153
abstract_call_known at ./compiler/abstractinterpretation.jl:1916
jfptr_abstract_call_known_16518.clone_1 at /nix/store/prmmzflxy02wajsrvw3k0jf1s8rixigy-julia-1.9.0-beta4-patched/lib/julia/sys.so (unknown line)
_jl_invoke at /tmp/nix-build-julia-1.9.0-beta4-patched.drv-0/source/src/gf.c:2670 [inlined]
ijl_invoke at /tmp/nix-build-julia-1.9.0-beta4-patched.drv-0/source/src/gf.c:2696
tojlinvoke21222.clone_1 at /nix/store/prmmzflxy02wajsrvw3k0jf1s8rixigy-julia-1.9.0-beta4-patched/lib/julia/sys.so (unknown line)
j_abstract_call_known_13421.clone_1 at /nix/store/prmmzflxy02wajsrvw3k0jf1s8rixigy-julia-1.9.0-beta4-patched/lib/julia/sys.so (unknown line)
abstract_call at ./compiler/abstractinterpretation.jl:1987
abstract_call at ./compiler/abstractinterpretation.jl:1966
abstract_eval_statement_expr at ./compiler/abstractinterpretation.jl:2132
abstract_eval_statement at ./compiler/abstractinterpretation.jl:2346
abstract_eval_basic_statement at ./compiler/abstractinterpretation.jl:2558
typeinf_local at ./compiler/abstractinterpretation.jl:2733
typeinf_nocycle at ./compiler/abstractinterpretation.jl:2839
_typeinf at ./compiler/typeinfer.jl:244
typeinf at ./compiler/typeinfer.jl:215
typeinf_edge at ./compiler/typeinfer.jl:947
abstract_call_method at ./compiler/abstractinterpretation.jl:609
abstract_call_gf_by_type at ./compiler/abstractinterpretation.jl:153
abstract_call_known at ./compiler/abstractinterpretation.jl:1916
jfptr_abstract_call_known_16518.clone_1 at /nix/store/prmmzflxy02wajsrvw3k0jf1s8rixigy-julia-1.9.0-beta4-patched/lib/julia/sys.so (unknown line)
_jl_invoke at /tmp/nix-build-julia-1.9.0-beta4-patched.drv-0/source/src/gf.c:2670 [inlined]
ijl_invoke at /tmp/nix-build-julia-1.9.0-beta4-patched.drv-0/source/src/gf.c:2696
tojlinvoke21222.clone_1 at /nix/store/prmmzflxy02wajsrvw3k0jf1s8rixigy-julia-1.9.0-beta4-patched/lib/julia/sys.so (unknown line)
j_abstract_call_known_13421.clone_1 at /nix/store/prmmzflxy02wajsrvw3k0jf1s8rixigy-julia-1.9.0-beta4-patched/lib/julia/sys.so (unknown line)
abstract_call at ./compiler/abstractinterpretation.jl:1987
abstract_call at ./compiler/abstractinterpretation.jl:1966
abstract_eval_statement_expr at ./compiler/abstractinterpretation.jl:2132
abstract_eval_statement at ./compiler/abstractinterpretation.jl:2346
abstract_eval_basic_statement at ./compiler/abstractinterpretation.jl:2558
typeinf_local at ./compiler/abstractinterpretation.jl:2733
typeinf_nocycle at ./compiler/abstractinterpretation.jl:2839
_typeinf at ./compiler/typeinfer.jl:244
typeinf at ./compiler/typeinfer.jl:215
typeinf_ext at ./compiler/typeinfer.jl:1072
typeinf_ext_toplevel at ./compiler/typeinfer.jl:1105
typeinf_ext_toplevel at ./compiler/typeinfer.jl:1101
jfptr_typeinf_ext_toplevel_18838.clone_1 at /nix/store/prmmzflxy02wajsrvw3k0jf1s8rixigy-julia-1.9.0-beta4-patched/lib/julia/sys.so (unknown line)
_jl_invoke at /tmp/nix-build-julia-1.9.0-beta4-patched.drv-0/source/src/gf.c:2670 [inlined]
ijl_apply_generic at /tmp/nix-build-julia-1.9.0-beta4-patched.drv-0/source/src/gf.c:2871
jl_apply at /tmp/nix-build-julia-1.9.0-beta4-patched.drv-0/source/src/julia.h:1874 [inlined]
jl_type_infer at /tmp/nix-build-julia-1.9.0-beta4-patched.drv-0/source/src/gf.c:317
jl_generate_fptr_impl at /tmp/nix-build-julia-1.9.0-beta4-patched.drv-0/source/src/jitlayers.cpp:431
jl_compile_method_internal at /tmp/nix-build-julia-1.9.0-beta4-patched.drv-0/source/src/gf.c:2286 [inlined]
jl_compile_method_internal at /tmp/nix-build-julia-1.9.0-beta4-patched.drv-0/source/src/gf.c:2204
_jl_invoke at /tmp/nix-build-julia-1.9.0-beta4-patched.drv-0/source/src/gf.c:2681 [inlined]
ijl_apply_generic at /tmp/nix-build-julia-1.9.0-beta4-patched.drv-0/source/src/gf.c:2871
jl_apply at /tmp/nix-build-julia-1.9.0-beta4-patched.drv-0/source/src/julia.h:1874 [inlined]
do_call at /tmp/nix-build-julia-1.9.0-beta4-patched.drv-0/source/src/interpreter.c:126
eval_value at /tmp/nix-build-julia-1.9.0-beta4-patched.drv-0/source/src/interpreter.c:226
eval_stmt_value at /tmp/nix-build-julia-1.9.0-beta4-patched.drv-0/source/src/interpreter.c:177 [inlined]
eval_body at /tmp/nix-build-julia-1.9.0-beta4-patched.drv-0/source/src/interpreter.c:624
jl_interpret_toplevel_thunk at /tmp/nix-build-julia-1.9.0-beta4-patched.drv-0/source/src/interpreter.c:762
jl_toplevel_eval_flex at /tmp/nix-build-julia-1.9.0-beta4-patched.drv-0/source/src/toplevel.c:912
jl_toplevel_eval_flex at /tmp/nix-build-julia-1.9.0-beta4-patched.drv-0/source/src/toplevel.c:856
ijl_toplevel_eval_in at /tmp/nix-build-julia-1.9.0-beta4-patched.drv-0/source/src/toplevel.c:971
eval at ./boot.jl:370 [inlined]
include_string at ./loading.jl:1849
_jl_invoke at /tmp/nix-build-julia-1.9.0-beta4-patched.drv-0/source/src/gf.c:2670 [inlined]
ijl_apply_generic at /tmp/nix-build-julia-1.9.0-beta4-patched.drv-0/source/src/gf.c:2871
_include at ./loading.jl:1909
include at ./Base.jl:457
jfptr_include_45116.clone_1 at /nix/store/prmmzflxy02wajsrvw3k0jf1s8rixigy-julia-1.9.0-beta4-patched/lib/julia/sys.so (unknown line)
_jl_invoke at /tmp/nix-build-julia-1.9.0-beta4-patched.drv-0/source/src/gf.c:2670 [inlined]
ijl_apply_generic at /tmp/nix-build-julia-1.9.0-beta4-patched.drv-0/source/src/gf.c:2871
jl_apply at /tmp/nix-build-julia-1.9.0-beta4-patched.drv-0/source/src/julia.h:1874 [inlined]
jl_f__call_latest at /tmp/nix-build-julia-1.9.0-beta4-patched.drv-0/source/src/builtins.c:774
include at /tmp/nix-build-raicode-tests.sorting.drv-0/rai-server-source/.julia/packages/HiGHS/QRk9O/src/HiGHS.jl:6
unknown function (ip: 0x7fffd00c1372)
_jl_invoke at /tmp/nix-build-julia-1.9.0-beta4-patched.drv-0/source/src/gf.c:2670 [inlined]
ijl_apply_generic at /tmp/nix-build-julia-1.9.0-beta4-patched.drv-0/source/src/gf.c:2871
jl_apply at /tmp/nix-build-julia-1.9.0-beta4-patched.drv-0/source/src/julia.h:1874 [inlined]
do_call at /tmp/nix-build-julia-1.9.0-beta4-patched.drv-0/source/src/interpreter.c:126
eval_value at /tmp/nix-build-julia-1.9.0-beta4-patched.drv-0/source/src/interpreter.c:226
eval_stmt_value at /tmp/nix-build-julia-1.9.0-beta4-patched.drv-0/source/src/interpreter.c:177 [inlined]
eval_body at /tmp/nix-build-julia-1.9.0-beta4-patched.drv-0/source/src/interpreter.c:624
jl_interpret_toplevel_thunk at /tmp/nix-build-julia-1.9.0-beta4-patched.drv-0/source/src/interpreter.c:762
jl_toplevel_eval_flex at /tmp/nix-build-julia-1.9.0-beta4-patched.drv-0/source/src/toplevel.c:912
jl_eval_module_expr at /tmp/nix-build-julia-1.9.0-beta4-patched.drv-0/source/src/toplevel.c:203 [inlined]
jl_toplevel_eval_flex at /tmp/nix-build-julia-1.9.0-beta4-patched.drv-0/source/src/toplevel.c:715
jl_toplevel_eval_flex at /tmp/nix-build-julia-1.9.0-beta4-patched.drv-0/source/src/toplevel.c:856
ijl_toplevel_eval_in at /tmp/nix-build-julia-1.9.0-beta4-patched.drv-0/source/src/toplevel.c:971
eval at ./boot.jl:370 [inlined]
include_string at ./loading.jl:1849
_jl_invoke at /tmp/nix-build-julia-1.9.0-beta4-patched.drv-0/source/src/gf.c:2670 [inlined]
ijl_apply_generic at /tmp/nix-build-julia-1.9.0-beta4-patched.drv-0/source/src/gf.c:2871
_include at ./loading.jl:1909
include at ./Base.jl:457 [inlined]
_require at ./loading.jl:1786
_require_prelocked at ./loading.jl:1610
macro expansion at ./loading.jl:1598 [inlined]
macro expansion at ./lock.jl:267 [inlined]
require at ./loading.jl:1561
jfptr_require_45249.clone_1 at /nix/store/prmmzflxy02wajsrvw3k0jf1s8rixigy-julia-1.9.0-beta4-patched/lib/julia/sys.so (unknown line)
_jl_invoke at /tmp/nix-build-julia-1.9.0-beta4-patched.drv-0/source/src/gf.c:2670 [inlined]
ijl_apply_generic at /tmp/nix-build-julia-1.9.0-beta4-patched.drv-0/source/src/gf.c:2871
jl_apply at /tmp/nix-build-julia-1.9.0-beta4-patched.drv-0/source/src/julia.h:1874 [inlined]
call_require at /tmp/nix-build-julia-1.9.0-beta4-patched.drv-0/source/src/toplevel.c:466 [inlined]
eval_import_path at /tmp/nix-build-julia-1.9.0-beta4-patched.drv-0/source/src/toplevel.c:503
jl_toplevel_eval_flex at /tmp/nix-build-julia-1.9.0-beta4-patched.drv-0/source/src/toplevel.c:731
jl_eval_module_expr at /tmp/nix-build-julia-1.9.0-beta4-patched.drv-0/source/src/toplevel.c:203 [inlined]
jl_toplevel_eval_flex at /tmp/nix-build-julia-1.9.0-beta4-patched.drv-0/source/src/toplevel.c:715
jl_toplevel_eval_flex at /tmp/nix-build-julia-1.9.0-beta4-patched.drv-0/source/src/toplevel.c:856
ijl_toplevel_eval_in at /tmp/nix-build-julia-1.9.0-beta4-patched.drv-0/source/src/toplevel.c:971
eval at ./boot.jl:370 [inlined]
include_string at ./loading.jl:1849
_jl_invoke at /tmp/nix-build-julia-1.9.0-beta4-patched.drv-0/source/src/gf.c:2670 [inlined]
ijl_apply_generic at /tmp/nix-build-julia-1.9.0-beta4-patched.drv-0/source/src/gf.c:2871
_include at ./loading.jl:1909
include at ./Base.jl:457 [inlined]
include_package_for_output at ./loading.jl:1995
jfptr_include_package_for_output_28651.clone_1 at /nix/store/prmmzflxy02wajsrvw3k0jf1s8rixigy-julia-1.9.0-beta4-patched/lib/julia/sys.so (unknown line)
_jl_invoke at /tmp/nix-build-julia-1.9.0-beta4-patched.drv-0/source/src/gf.c:2670 [inlined]
ijl_apply_generic at /tmp/nix-build-julia-1.9.0-beta4-patched.drv-0/source/src/gf.c:2871
jl_apply at /tmp/nix-build-julia-1.9.0-beta4-patched.drv-0/source/src/julia.h:1874 [inlined]
do_call at /tmp/nix-build-julia-1.9.0-beta4-patched.drv-0/source/src/interpreter.c:126
eval_value at /tmp/nix-build-julia-1.9.0-beta4-patched.drv-0/source/src/interpreter.c:226
eval_stmt_value at /tmp/nix-build-julia-1.9.0-beta4-patched.drv-0/source/src/interpreter.c:177 [inlined]
eval_body at /tmp/nix-build-julia-1.9.0-beta4-patched.drv-0/source/src/interpreter.c:624
jl_interpret_toplevel_thunk at /tmp/nix-build-julia-1.9.0-beta4-patched.drv-0/source/src/interpreter.c:762
jl_toplevel_eval_flex at /tmp/nix-build-julia-1.9.0-beta4-patched.drv-0/source/src/toplevel.c:912
jl_toplevel_eval_flex at /tmp/nix-build-julia-1.9.0-beta4-patched.drv-0/source/src/toplevel.c:856
ijl_toplevel_eval_in at /tmp/nix-build-julia-1.9.0-beta4-patched.drv-0/source/src/toplevel.c:971
eval at ./boot.jl:370 [inlined]
include_string at ./loading.jl:1849
include_string at ./loading.jl:1859 [inlined]
exec_options at ./client.jl:305
_start at ./client.jl:522
jfptr__start_37954.clone_1 at /nix/store/prmmzflxy02wajsrvw3k0jf1s8rixigy-julia-1.9.0-beta4-patched/lib/julia/sys.so (unknown line)
_jl_invoke at /tmp/nix-build-julia-1.9.0-beta4-patched.drv-0/source/src/gf.c:2670 [inlined]
ijl_apply_generic at /tmp/nix-build-julia-1.9.0-beta4-patched.drv-0/source/src/gf.c:2871
jl_apply at /tmp/nix-build-julia-1.9.0-beta4-patched.drv-0/source/src/julia.h:1874 [inlined]
true_main at /tmp/nix-build-julia-1.9.0-beta4-patched.drv-0/source/src/jlapi.c:573
jl_repl_entrypoint at /tmp/nix-build-julia-1.9.0-beta4-patched.drv-0/source/src/jlapi.c:717
main at /tmp/nix-build-julia-1.9.0-beta4-patched.drv-0/source/cli/loader_exe.c:59
__libc_start_main at /nix/store/0xxjx37fcy2nl3yz6igmv4mag2a7giq6-glibc-2.33-123/lib/libc.so.6 (unknown line)
unknown function (ip: 0x401089)
Allocations: 14006610 (Pool: 14005184; Big: 1426); GC: 19

[timholy]

Thanks for the report. Can you provide a reproducer?

Due to teaching deadlines it could be ~1 week before I can get to this.

[timholy]

If you can't provide a reproducer...maybe a bit of explanation may help. The code here is part of the solution for the problem described most completely in the top set of bullets of the OP in #42016.

The system image is key = 0; for efficiency, during system image building we just keep track of how many roots got added before writing out the sysimage (see #43881). If you're looking up a root by "naked index," it had better be for the sysimage. Any other root gets numbered by (buildid, offset) to keep such references independent of the order in which we load modules.

One way this could break is if you're inserting some weird custom build steps where the standard rules of "either we're building a sysimg or we're precompiling a pkgimg" for some reason aren't true. For example, in the REPL you can force addition of new roots with no module-buildid provenance. But the clean environment of standard package precompilation should not make that possible (I think).

[kpamnany]

Thanks @timholy, for the explanation. We're seeing these assertion failures (segfaults if assertions are turned off) after a sequence of warnings like:

┌ Warning: The call to compilecache failed to create a usable precompiled cache file for StatsBase [2913bbd2-ae8a-5f71-8c99-4fb6c76f3a91]
│   exception = Required dependency StatsAPI [82ae8749-77ed-4fe6-ae5f-f523153014b0] failed to load from a cache file.
└ @ Base loading.jl:1768

Followed by:

┌ Warning: Module StatsBase with build ID ffffffff-ffff-ffff-0000-07c6b38e87bc is missing from the cache.
│ This may mean StatsBase [2913bbd2-ae8a-5f71-8c99-4fb6c76f3a91] does not support precompilation but is imported by a module that does.
└ @ Base loading.jl:1743

Our test suite first runs Pkg.build, which precompiles the project. We then use Distributed to run tests on multiple workers concurrently (via XUnit.jl). On 1.9.0-beta4, the workers all start precompiling packages again, failing with the messages above (presumably due to races?) and then throwing these assertion failures.

@vchuravy pointed out to us that the workers must be started with the same arguments as the Julia process that did the precompilation (in particular, bounds checking, code coverage, and optimization level), so we're trying to tweak things so that they don't try to precompile again. This will hopefully either fix these assertion failures or eliminate one possible reason for them. Will keep you posted.

Cc: @NHDaly

[timholy]

Thanks for the extra details. FWIW, I built a debug build of release-1.9 and am not seeing those warnings when I precompile StatsBase.

[NHDaly]

🤔 i'm not sure why the test's worker processes should even have to be doing any package compilation in parallel, since from what I understand, Pkg runs Pkg.precompile() before starting the test job...

here, i think?:
https://github.com/JuliaLang/Pkg.jl/blob/30cc570a0608b2e010d11b483a06e2bf5d3aaa98/src/Operations.jl#L1607
https://github.com/JuliaLang/Pkg.jl/blob/30cc570a0608b2e010d11b483a06e2bf5d3aaa98/src/Operations.jl#L1615-L1630

So it should be that everything is already precompiled, right?

[vchuravy]

Seen on PkgEval

https://s3.amazonaws.com/julialang-reports/nanosoldier/pkgeval/by_hash/dabb822_vs_aacfcf0/MPIReco.primary.log

[timholy]

This is looking more like a package or Pkg bug and less like a problem in the method roots code. I'm at least temporarily unassigning myself since that's less obviously an area of expertise for me. But I'll be happy to keep poking at it as more info comes in. FWIW I did a pkg> add MPIReco and pkg> test MPIReco on my debug build of Julia and did not reproduce the observed error. I'm guessing there's some kind of race condition?

[NHDaly]

our tests are running with multiple distributed worker processes, so a race seems likely, indeed.

How can we help with debugging the possible pkg bug?

[KristofferC]

You can run with JULIA_DEBUG=loading and see why the existing precompile files are not used.

[NHDaly]

Thanks, great suggestion. We're seeing messages like these, which imply to us that somehow the Pkg.precompile() step run by Pkg.test() isn't doing precompilation with the expected correct flags for --code-coverage:

┌ Debug: Rejecting cache file /tmp/nix-build-raicode-tests.unit.drv-0/rai-server-source/.julia/compiled/v1.9/HTTP/zXWya_qRjh6.ji for HTTP [cd3eb016-35fb-5094-929b-558a96fad6f3] since it is for OpenSSL [4d8831e6-92b7-49fb-bdf8-b643e874388c] => 0x000045521956c329 instead
└ @ Base loading.jl:2696
┌ Debug: Rejecting cache file /tmp/nix-build-raicode-tests.unit.drv-0/rai-server-source/.julia/compiled/v1.9/HTTP/zXWya_j0bKM.ji for HTTP [cd3eb016-35fb-5094-929b-558a96fad6f3] since the flags are mismatched
│   current session: use_pkgimages = true, debug_level = 1, check_bounds = 1, inline = true, opt_level = 0
│   cache file:      use_pkgimages = true, debug_level = 1, check_bounds = 0, inline = true, opt_level = 0
└ @ Base loading.jl:2668
┌ Debug: Rejecting cache file /tmp/nix-build-raicode-tests.unit.drv-0/rai-server-source/.julia/compiled/v1.9/HTTP/zXWya_ylQ0R.ji for HTTP [cd3eb016-35fb-5094-929b-558a96fad6f3] since the flags are mismatched
│   current session: use_pkgimages = true, debug_level = 1, check_bounds = 1, inline = true, opt_level = 0
│   cache file:      use_pkgimages = true, debug_level = 1, check_bounds = 0, inline = true, opt_level = 2
└ @ Base loading.jl:2668
┌ Debug: Precompiling HTTP [cd3eb016-35fb-5094-929b-558a96fad6f3]
└ @ Base loading.jl:2124

then later:

┌ Warning: Module HTTP with build ID ffffffff-ffff-ffff-0000-45536692a1c7 is missing from the cache.
│ This may mean HTTP [cd3eb016-35fb-5094-929b-558a96fad6f3] does not support precompilation but is imported by a module that does.
└ @ Base loading.jl:1743

(We think the second, opt_level = 2 version is coming from our own Pkg.instantiate() step that we do earlier in our CI job's build process.)

Do you know why the cached .ji file should be built with check_bounds = 0?

[NHDaly]

This is how we're invoking the tests:

        
        @time Pkg.test("RAICode"; julia_args=[Base.julia_cmd().exec[2:end]..., "-p", "$(nworkers)", "--eval", code])

from a julia process started as:

julia -O0 ./test.jl