High (default) gcthreads count causing slowdown on Julia 1.10.0-beta2

[jaakkor2]

Just showing versioninfo() takes 55 s (varies from run to run)

$ ./julia/julia-1.10.0-beta2/bin/julia --threads 55 --gcthreads=16 --startup-file=no
               _
   _       _ _(_)_     |  Documentation: https://docs.julialang.org
  (_)     | (_) (_)    |
   _ _   _| |_  __ _   |  Type "?" for help, "]?" for Pkg help.
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 1.10.0-beta2 (2023-08-17)
 _/ |\__'_|_|_|\__'_|  |  Official https://julialang.org/ release
|__/                   |

julia> @time display(versioninfo())
Julia Version 1.10.0-beta2
Commit a468aa198d0 (2023-08-17 06:27 UTC)
Build Info:
  Official https://julialang.org/ release
Platform Info:
  OS: Linux (x86_64-linux-gnu)
  CPU: 64 × AMD EPYC 7371 16-Core Processor
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-15.0.7 (ORCJIT, znver1)
  Threads: 70 on 64 virtual cores
nothing
 55.698993 seconds (439.98 k allocations: 29.438 MiB, 99.01% gc time, 99.99% compilation time)

but then setting --threads auto takes more than I could wait

$ ./julia/julia-1.10.0-beta2/bin/julia --threads auto --startup-file=no
               _
   _       _ _(_)_     |  Documentation: https://docs.julialang.org
  (_)     | (_) (_)    |
   _ _   _| |_  __ _   |  Type "?" for help, "]?" for Pkg help.
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 1.10.0-beta2 (2023-08-17)
 _/ |\__'_|_|_|\__'_|  |  Official https://julialang.org/ release
|__/                   |

julia> Threads.ngcthreads()
32

julia> @time display(versioninfo())

Setting --threads auto --gcthreads 1 seems to be fine.

[d-netto]

Can't reproduce:

./julia --threads=64 --gcthreads=1
               _
   _       _ _(_)_     |  Documentation: https://docs.julialang.org
  (_)     | (_) (_)    |
   _ _   _| |_  __ _   |  Type "?" for help, "]?" for Pkg help.
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 1.10.0-beta2 (2023-08-17)
 _/ |\__'_|_|_|\__'_|  |  
|__/                   |

julia> @time display(versioninfo())
Julia Version 1.10.0-beta2
Commit a468aa198d (2023-08-17 06:27 UTC)
Build Info:

    Note: This is an unofficial build, please report bugs to the project
    responsible for this build and not to the Julia project unless you can
    reproduce the issue using official builds available at https://julialang.org/downloads

Platform Info:
  OS: Linux (x86_64-linux-gnu)
  CPU: 128 × AMD EPYC 7502 32-Core Processor
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-15.0.7 (ORCJIT, znver2)
  Threads: 64 on 128 virtual cores
nothing
  0.497072 seconds (440.23 k allocations: 29.451 MiB, 1.12% gc time, 95.90% compilation time)

./julia --threads=64 --gcthreads=64
               _
   _       _ _(_)_     |  Documentation: https://docs.julialang.org
  (_)     | (_) (_)    |
   _ _   _| |_  __ _   |  Type "?" for help, "]?" for Pkg help.
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 1.10.0-beta2 (2023-08-17)
 _/ |\__'_|_|_|\__'_|  |  
|__/                   |

julia> @time display(versioninfo())
Julia Version 1.10.0-beta2
Commit a468aa198d (2023-08-17 06:27 UTC)
Build Info:

    Note: This is an unofficial build, please report bugs to the project
    responsible for this build and not to the Julia project unless you can
    reproduce the issue using official builds available at https://julialang.org/downloads

Platform Info:
  OS: Linux (x86_64-linux-gnu)
  CPU: 128 × AMD EPYC 7502 32-Core Processor
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-15.0.7 (ORCJIT, znver2)
  Threads: 127 on 128 virtual cores
nothing
  0.506744 seconds (440.23 k allocations: 29.451 MiB, 1.81% gc time, 95.35% compilation time)

[d-netto]

The results in the post above were measured using a smaller number of threads than the number of cores reported in versioninfo.

Setting the number of GC threads to something higher than the number of virtual cores seems to cause a regression:

./julia --threads=128 --gcthreads=1
               _
   _       _ _(_)_     |  Documentation: https://docs.julialang.org
  (_)     | (_) (_)    |
   _ _   _| |_  __ _   |  Type "?" for help, "]?" for Pkg help.
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 1.10.0-beta2 (2023-08-17)
 _/ |\__'_|_|_|\__'_|  |  
|__/                   |

julia> @time display(versioninfo())
Julia Version 1.10.0-beta2
Commit a468aa198d (2023-08-17 06:27 UTC)
Build Info:

    Note: This is an unofficial build, please report bugs to the project
    responsible for this build and not to the Julia project unless you can
    reproduce the issue using official builds available at https://julialang.org/downloads

Platform Info:
  OS: Linux (x86_64-linux-gnu)
  CPU: 128 × AMD EPYC 7502 32-Core Processor
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-15.0.7 (ORCJIT, znver2)
  Threads: 128 on 128 virtual cores
nothing
  0.498495 seconds (440.23 k allocations: 29.451 MiB, 1.73% gc time, 95.73% compilation time)

./julia --threads=128 --gcthreads=132
               _
   _       _ _(_)_     |  Documentation: https://docs.julialang.org
  (_)     | (_) (_)    |
   _ _   _| |_  __ _   |  Type "?" for help, "]?" for Pkg help.
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 1.10.0-beta2 (2023-08-17)
 _/ |\__'_|_|_|\__'_|  |  
|__/                   |

julia> @time display(versioninfo())
Julia Version 1.10.0-beta2
Commit a468aa198d (2023-08-17 06:27 UTC)
Build Info:

    Note: This is an unofficial build, please report bugs to the project
    responsible for this build and not to the Julia project unless you can
    reproduce the issue using official builds available at https://julialang.org/downloads

Platform Info:
  OS: Linux (x86_64-linux-gnu)
  CPU: 128 × AMD EPYC 7502 32-Core Processor
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-15.0.7 (ORCJIT, znver2)
  Threads: 259 on 128 virtual cores
nothing
  2.171682 seconds (440.23 k allocations: 29.451 MiB, 76.90% gc time, 98.86% compilation time)

[jaakkor2]

This problem persists on 1.10.0-rc1. This is a Linux box with 2 x AMD EPYC 7371.

$ ./julia-1.10.0-rc1/bin/julia --startup-file=no --threads=auto
               _
   _       _ _(_)_     |  Documentation: https://docs.julialang.org
  (_)     | (_) (_)    |
   _ _   _| |_  __ _   |  Type "?" for help, "]?" for Pkg help.
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 1.10.0-rc1 (2023-11-03)
 _/ |\__'_|_|_|\__'_|  |  Official https://julialang.org/ release
|__/                   |

julia> Threads.nthreads()
64

julia> Threads.ngcthreads()
32

julia> Threads.nthreadpools()
2

julia> versioninfo()

does not print anything as the the Julia process spins at 3200 %CPU.

Without -t auto I get

$ ./julia-1.10.0-rc1/bin/julia --startup-file=no
               _
   _       _ _(_)_     |  Documentation: https://docs.julialang.org
  (_)     | (_) (_)    |
   _ _   _| |_  __ _   |  Type "?" for help, "]?" for Pkg help.
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 1.10.0-rc1 (2023-11-03)
 _/ |\__'_|_|_|\__'_|  |  Official https://julialang.org/ release
|__/                   |

julia> versioninfo()
Julia Version 1.10.0-rc1
Commit 5aaa9485436 (2023-11-03 07:44 UTC)
Build Info:
  Official https://julialang.org/ release
Platform Info:
  OS: Linux (x86_64-linux-gnu)
  CPU: 64 × AMD EPYC 7371 16-Core Processor
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-15.0.7 (ORCJIT, znver1)
  Threads: 1 on 64 virtual cores

julia> (Threads.nthreads(), Threads.ngcthreads(), Threads.nthreadpools())
(1, 1, 2)

[jaakkor2]

@KristofferC I think this was removed from milestone 1.10 prematurely.

[LilithHafner]

Re-adding to the milestone because this regressing is visible with -t auto on systems with high core count (summarizing above results, I have not personally reproduced this)

Perhaps adjusting the default number of gc threads would be a reasonable hotfix for the 1.10 regression?