Function call getting stuck without errors or warnings in versions >=1.11.4

[anchal-physics]

In our package, we have defined a function that takes in arguments that could be from a union of large set of custom data types. This function used to execute flawlessly within 11..3 seconds including the first time compilation time. But from julia versions 1.11.4 onwards, the function call gets stuck endlessly without showing any errors or warning. On doing Ctrl-C, I can see that it was stuck in some compilation step but it is hard to comprehend for me. Can someone take a look here and help me understand what changed from version 1.11.3 to 1.11.4 that broke our code?

This issue persists in version 1.11.5 as well.

Github workflow example:

https://github.com/ProjectTorreyPines/IMASggd.jl/actions/runs/14480505875

Here, the jobs that used julia 1.11.3 worked but those that used 1.11.4 and 1.11.5 were stuck in the projection test step. The 1.11.4 case got cancelled after 6 hours. And then I re-ran it for 1.11.5 which still

Works on Julia 1.11.3:

(base) gupta@F-CJXNMY7L7 IMASggd % julia +1.11.3 --project test/runtests.jl --projection
json2imas() time:   5.009234 seconds (33.22 M allocations: 1.330 GiB, 12.02% gc time, 56.81% compilation time)
project_prop_on_subset!(prop, from_subset, to_subset, space) time:  11.246959 seconds (278.42 M allocations: 12.242 GiB, 17.38% gc time, 95.45% compilation time)
project_prop_on_subset!(prop, from_subset, subset_core) time:   0.036126 seconds (46.90 k allocations: 2.377 MiB, 94.79% compilation time)
Test Summary:           | Pass  Total   Time
project_prop_on_subset! |    1      1  11.4s

Does not work on Julia 1.11.4 or higher:

(base) gupta@F-CJXNMY7L7 IMASggd % julia +1.11.4 --project test/runtests.jl --projection
json2imas() time:   4.988777 seconds (29.37 M allocations: 1.156 GiB, 11.32% gc time, 34.06% compilation time)
project_prop_on_subset!(prop, from_subset, to_subset, space) time: ^C
[43861] signal 2: Interrupt: 2
in expression starting at /Users/gupta/Git/ProjectTorreyPines/IMASggd/test/runtests.jl:159
statestack_set at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-XG3Q6T6R70.0/build/default-honeycrisp-XG3Q6T6R70-0/julialang/julia-release-1-dot-11/src/subtype.c:164 [inlined]
pick_union_decision at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-XG3Q6T6R70.0/build/default-honeycrisp-XG3Q6T6R70-0/julialang/julia-release-1-dot-11/src/subtype.c:189 [inlined]
pick_union_element at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-XG3Q6T6R70.0/build/default-honeycrisp-XG3Q6T6R70-0/julialang/julia-release-1-dot-11/src/subtype.c:202 [inlined]
subtype at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-XG3Q6T6R70.0/build/default-honeycrisp-XG3Q6T6R70-0/julialang/julia-release-1-dot-11/src/subtype.c:1400
subtype_unionall at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-XG3Q6T6R70.0/build/default-honeycrisp-XG3Q6T6R70-0/julialang/julia-release-1-dot-11/src/subtype.c:972
subtype_unionall at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-XG3Q6T6R70.0/build/default-honeycrisp-XG3Q6T6R70-0/julialang/julia-release-1-dot-11/src/subtype.c:965
local_forall_exists_subtype at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-XG3Q6T6R70.0/build/default-honeycrisp-XG3Q6T6R70-0/julialang/julia-release-1-dot-11/src/subtype.c:1622
subtype_ccheck at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-XG3Q6T6R70.0/build/default-honeycrisp-XG3Q6T6R70-0/julialang/julia-release-1-dot-11/src/subtype.c:698
var_gt at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-XG3Q6T6R70.0/build/default-honeycrisp-XG3Q6T6R70-0/julialang/julia-release-1-dot-11/src/subtype.c:812
local_forall_exists_subtype at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-XG3Q6T6R70.0/build/default-honeycrisp-XG3Q6T6R70-0/julialang/julia-release-1-dot-11/src/subtype.c:1600
forall_exists_equal at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-XG3Q6T6R70.0/build/default-honeycrisp-XG3Q6T6R70-0/julialang/julia-release-1-dot-11/src/subtype.c:1726
subtype at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-XG3Q6T6R70.0/build/default-honeycrisp-XG3Q6T6R70-0/julialang/julia-release-1-dot-11/src/subtype.c:1520
subtype_unionall at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-XG3Q6T6R70.0/build/default-honeycrisp-XG3Q6T6R70-0/julialang/julia-release-1-dot-11/src/subtype.c:965
subtype_unionall at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-XG3Q6T6R70.0/build/default-honeycrisp-XG3Q6T6R70-0/julialang/julia-release-1-dot-11/src/subtype.c:972
exists_subtype at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-XG3Q6T6R70.0/build/default-honeycrisp-XG3Q6T6R70-0/julialang/julia-release-1-dot-11/src/subtype.c:1744 [inlined]
forall_exists_subtype at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-XG3Q6T6R70.0/build/default-honeycrisp-XG3Q6T6R70-0/julialang/julia-release-1-dot-11/src/subtype.c:1773 [inlined]
ijl_subtype_env at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-XG3Q6T6R70.0/build/default-honeycrisp-XG3Q6T6R70-0/julialang/julia-release-1-dot-11/src/subtype.c:2232
ijl_subtype at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-XG3Q6T6R70.0/build/default-honeycrisp-XG3Q6T6R70-0/julialang/julia-release-1-dot-11/src/subtype.c:2269 [inlined]
subtype_tuple_tail at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-XG3Q6T6R70.0/build/default-honeycrisp-XG3Q6T6R70-0/julialang/julia-release-1-dot-11/src/subtype.c:1272 [inlined]
subtype_tuple at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-XG3Q6T6R70.0/build/default-honeycrisp-XG3Q6T6R70-0/julialang/julia-release-1-dot-11/src/subtype.c:1354
exists_subtype at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-XG3Q6T6R70.0/build/default-honeycrisp-XG3Q6T6R70-0/julialang/julia-release-1-dot-11/src/subtype.c:1744 [inlined]
forall_exists_subtype at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-XG3Q6T6R70.0/build/default-honeycrisp-XG3Q6T6R70-0/julialang/julia-release-1-dot-11/src/subtype.c:1773 [inlined]
ijl_subtype_env at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-XG3Q6T6R70.0/build/default-honeycrisp-XG3Q6T6R70-0/julialang/julia-release-1-dot-11/src/subtype.c:2232
jl_f_issubtype at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-XG3Q6T6R70.0/build/default-honeycrisp-XG3Q6T6R70-0/julialang/julia-release-1-dot-11/src/builtins.c:557
compute_inlining_cases at ./compiler/ssair/inlining.jl:1388
handle_call! at ./compiler/ssair/inlining.jl:1421 [inlined]
assemble_inline_todo! at ./compiler/ssair/inlining.jl:1675
ssa_inlining_pass! at ./compiler/ssair/inlining.jl:78 [inlined]
run_passes_ipo_safe at ./compiler/optimize.jl:994
run_passes_ipo_safe at ./compiler/optimize.jl:1009 [inlined]
optimize at ./compiler/optimize.jl:983
jfptr_optimize_42837.1 at /Users/gupta/.julia/juliaup/julia-1.11.4+0.aarch64.apple.darwin14/lib/julia/sys.dylib (unknown line)
finish_nocycle at ./compiler/typeinfer.jl:265
_typeinf at ./compiler/typeinfer.jl:249
typeinf at ./compiler/typeinfer.jl:215
typeinf_edge at ./compiler/typeinfer.jl:923
abstract_call_method at ./compiler/abstractinterpretation.jl:660
abstract_call_gf_by_type at ./compiler/abstractinterpretation.jl:102
abstract_call_known at ./compiler/abstractinterpretation.jl:2200
abstract_call at ./compiler/abstractinterpretation.jl:2282
abstract_call at ./compiler/abstractinterpretation.jl:2275
abstract_call at ./compiler/abstractinterpretation.jl:2423
abstract_eval_call at ./compiler/abstractinterpretation.jl:2438
abstract_eval_statement_expr at ./compiler/abstractinterpretation.jl:2454
abstract_eval_statement at ./compiler/abstractinterpretation.jl:2752
abstract_eval_basic_statement at ./compiler/abstractinterpretation.jl:3068
typeinf_local at ./compiler/abstractinterpretation.jl:3331
typeinf_nocycle at ./compiler/abstractinterpretation.jl:3413
_typeinf at ./compiler/typeinfer.jl:244
typeinf at ./compiler/typeinfer.jl:215
typeinf_edge at ./compiler/typeinfer.jl:923
abstract_call_method at ./compiler/abstractinterpretation.jl:660
abstract_call_gf_by_type at ./compiler/abstractinterpretation.jl:102
abstract_call_known at ./compiler/abstractinterpretation.jl:2200
abstract_call at ./compiler/abstractinterpretation.jl:2282
abstract_call at ./compiler/abstractinterpretation.jl:2275
abstract_call at ./compiler/abstractinterpretation.jl:2423
abstract_eval_call at ./compiler/abstractinterpretation.jl:2438
abstract_eval_statement_expr at ./compiler/abstractinterpretation.jl:2454
abstract_eval_statement at ./compiler/abstractinterpretation.jl:2752
abstract_eval_basic_statement at ./compiler/abstractinterpretation.jl:3044
typeinf_local at ./compiler/abstractinterpretation.jl:3331
typeinf_nocycle at ./compiler/abstractinterpretation.jl:3413
_typeinf at ./compiler/typeinfer.jl:244
typeinf at ./compiler/typeinfer.jl:215
typeinf_edge at ./compiler/typeinfer.jl:923
abstract_call_method at ./compiler/abstractinterpretation.jl:660
abstract_call_gf_by_type at ./compiler/abstractinterpretation.jl:102
abstract_call_known at ./compiler/abstractinterpretation.jl:2200
abstract_call at ./compiler/abstractinterpretation.jl:2282
abstract_call at ./compiler/abstractinterpretation.jl:2275
abstract_call at ./compiler/abstractinterpretation.jl:2423
abstract_eval_call at ./compiler/abstractinterpretation.jl:2438
abstract_eval_statement_expr at ./compiler/abstractinterpretation.jl:2454
abstract_eval_statement at ./compiler/abstractinterpretation.jl:2752
abstract_eval_basic_statement at ./compiler/abstractinterpretation.jl:3068
typeinf_local at ./compiler/abstractinterpretation.jl:3331
typeinf_nocycle at ./compiler/abstractinterpretation.jl:3413
_typeinf at ./compiler/typeinfer.jl:244
typeinf at ./compiler/typeinfer.jl:215
typeinf_ext at ./compiler/typeinfer.jl:1101
typeinf_ext_toplevel at ./compiler/typeinfer.jl:1139
typeinf_ext_toplevel at ./compiler/typeinfer.jl:1135
jfptr_typeinf_ext_toplevel_40033.1 at /Users/gupta/.julia/juliaup/julia-1.11.4+0.aarch64.apple.darwin14/lib/julia/sys.dylib (unknown line)
jl_apply at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-XG3Q6T6R70.0/build/default-honeycrisp-XG3Q6T6R70-0/julialang/julia-release-1-dot-11/src/./julia.h:2157 [inlined]
jl_type_infer at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-XG3Q6T6R70.0/build/default-honeycrisp-XG3Q6T6R70-0/julialang/julia-release-1-dot-11/src/gf.c:390
jl_generate_fptr_impl at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-XG3Q6T6R70.0/build/default-honeycrisp-XG3Q6T6R70-0/julialang/julia-release-1-dot-11/src/jitlayers.cpp:519
jl_compile_method_internal at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-XG3Q6T6R70.0/build/default-honeycrisp-XG3Q6T6R70-0/julialang/julia-release-1-dot-11/src/gf.c:2536
_jl_invoke at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-XG3Q6T6R70.0/build/default-honeycrisp-XG3Q6T6R70-0/julialang/julia-release-1-dot-11/src/gf.c:2940 [inlined]
ijl_apply_generic at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-XG3Q6T6R70.0/build/default-honeycrisp-XG3Q6T6R70-0/julialang/julia-release-1-dot-11/src/gf.c:3125
macro expansion at ./timing.jl:581 [inlined]
macro expansion at /Users/gupta/Git/ProjectTorreyPines/IMASggd/test/runtests.jl:168 [inlined]
macro expansion at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-XG3Q6T6R70.0/build/default-honeycrisp-XG3Q6T6R70-0/julialang/julia-release-1-dot-11/usr/share/julia/stdlib/v1.11/Test/src/Test.jl:1704 [inlined]
top-level scope at /Users/gupta/Git/ProjectTorreyPines/IMASggd/test/runtests.jl:161
jl_toplevel_eval_flex at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-XG3Q6T6R70.0/build/default-honeycrisp-XG3Q6T6R70-0/julialang/julia-release-1-dot-11/src/toplevel.c:934
jl_toplevel_eval_flex at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-XG3Q6T6R70.0/build/default-honeycrisp-XG3Q6T6R70-0/julialang/julia-release-1-dot-11/src/toplevel.c:886
ijl_toplevel_eval at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-XG3Q6T6R70.0/build/default-honeycrisp-XG3Q6T6R70-0/julialang/julia-release-1-dot-11/src/toplevel.c:952 [inlined]
ijl_toplevel_eval_in at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-XG3Q6T6R70.0/build/default-honeycrisp-XG3Q6T6R70-0/julialang/julia-release-1-dot-11/src/toplevel.c:994
eval at ./boot.jl:430 [inlined]
include_string at ./loading.jl:2734
_include at ./loading.jl:2794
include at ./Base.jl:557
jfptr_include_47006.1 at /Users/gupta/.julia/juliaup/julia-1.11.4+0.aarch64.apple.darwin14/lib/julia/sys.dylib (unknown line)
exec_options at ./client.jl:323
_start at ./client.jl:531
jfptr__start_73863.1 at /Users/gupta/.julia/juliaup/julia-1.11.4+0.aarch64.apple.darwin14/lib/julia/sys.dylib (unknown line)
jl_apply at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-XG3Q6T6R70.0/build/default-honeycrisp-XG3Q6T6R70-0/julialang/julia-release-1-dot-11/src/./julia.h:2157 [inlined]
true_main at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-XG3Q6T6R70.0/build/default-honeycrisp-XG3Q6T6R70-0/julialang/julia-release-1-dot-11/src/jlapi.c:900
jl_repl_entrypoint at /Users/julia/.julia/scratchspaces/a66863c6-20e8-4ff4-8a62-49f30b1f605e/agent-cache/default-honeycrisp-XG3Q6T6R70.0/build/default-honeycrisp-XG3Q6T6R70-0/julialang/julia-release-1-dot-11/src/jlapi.c:1059
unknown function (ip: 0x0)
Allocations: 202263693 (Pool: 202262685; Big: 1008); GC: 103

Reproduction:

• Clone the repo

git clone git@github.com:ProjectTorreyPines/IMASggd.jl.git

• Download the attached sample file in the samples directory in the clone.
• Checkout to branch:

git checkout compilation_issue

• Run with julia 1.11.3
  • Install 1.11.3 and instantiate project IMASggd

cd IMASggd.jl
juliaup add 1.11.3
julia +1.11.3 --project
julia> ]
IMASggs> instantiate
julia> exit()

• Run test with --projection option (to ignore other tests for now)

julia +1.11.3 --project test/runtests.jl --projection

Output:

Precompiling IMASggd...
  1 dependency successfully precompiled in 4 seconds. 119 already precompiled.
json2imas() time:   5.024165 seconds (29.37 M allocations: 1.156 GiB, 9.82% gc time, 34.81% compilation time)
project_prop_on_subset!(prop, from_subset, to_subset, space) time:  11.260369 seconds (278.13 M allocations: 12.227 GiB, 17.17% gc time, 95.55% compilation time)
project_prop_on_subset!(prop, from_subset, subset_core) time:   0.036846 seconds (46.90 k allocations: 2.371 MiB, 92.36% compilation time)
Test Summary:           | Pass  Total   Time
project_prop_on_subset! |    1      1  11.3s

• Run with julia 1.11.4 or 1.11.5
  • Install 1.11.4 and instantiate project IMASggd

cd IMASggd.jl
juliaup add 1.11.4
julia +1.11.4 --project
julia> ]
IMASggs> instantiate
julia> exit()

• Run test with --projection option (to ignore other tests for now)

julia +1.11.4 --project test/runtests.jl --projection

Output (gets stuck):

json2imas() time:   5.201588 seconds (29.37 M allocations: 1.156 GiB, 11.24% gc time, 37.43% compilation time)
project_prop_on_subset!(prop, from_subset, to_subset, space) time: 

Do Ctrl-Cto get out and generate the stack trace.

Possible culprit:

a8eace9

[N5N3]

I tried to reproduce this, but failed in step 2 (downloading the attached sample file).

$ git clone git@github.com:ProjectTorreyPines/SOLPSTestSamples.git
Cloning into 'SOLPSTestSamples'...
ERROR: Repository not found.
fatal: Could not read from remote repository.

CC @anchal-physics

[anchal-physics]

I'm sorry, I tried to attach the file here in the issue itself but somehow that didn't go through. Please find that file here:

time_dep_edge_profiles_last_step_only.json

[N5N3]

Reduced to the following

N = 370
for k in 1:N
    @eval struct $(Symbol(:T, k)){T} end
end
Tvar = TypeVar(:T)
V = UnionAll(Tvar, Union{[@eval($(Symbol(:T, k)){Tvar}) for k in 1:N]...})
Set{<:V} <: AbstractSet{<:V} #hit union bit stack overflow assertion.

So looks like a duplication of #56350.

[anchal-physics]

Thanks. There could be something more to this. When I changed the function to take arguments of any type, it still was getting stuck, probably because inside there is a function that still was defined for the union of the large number of types. Maybe it is the same issue.