- Sponsor
-
Notifications
You must be signed in to change notification settings - Fork 5.6k
Closed
Labels
performanceMust go fasterMust go fasterregressionRegression in behavior compared to a previous versionRegression in behavior compared to a previous version
Description
This NFC PR to Chairmarks apparently made CI on nightly substantially (>2x) slower. I suspect the regression is due to a change in Julia.
Fast (https://github.com/LilithHafner/Chairmarks.jl/actions/runs/8147371039/job/22267929268)
Julia Version 1.12.0-DEV.124
Commit 0311aa4caf (2024-03-04 16:34 UTC)
Slow (https://github.com/LilithHafner/Chairmarks.jl/actions/runs/8174911720/job/22350814940)
Julia Version 1.12.0-DEV.138
Commit 90d84d4ab00 (2024-03-06 00:32 UTC)
Diff
https://github.com/JuliaLang/julia/compare/0311aa4caf..90d84d4ab00
See also: #53439
Metadata
Metadata
Assignees
Labels
performanceMust go fasterMust go fasterregressionRegression in behavior compared to a previous versionRegression in behavior compared to a previous version
Type
Projects
Milestone
Relationships
Development
Select code repository
Activity
Zentrik commentedon Mar 9, 2024
Bisected to fb71a5d
Bisect Script
LilithHafner commentedon Mar 9, 2024
Thanks for the bisection! I don't suppose I could tempt you into figuring out which of the PRs to Pkg is responsible? There are only 4 candidates (listed here) and I suspect that JuliaLang/Pkg.jl#3820 is the culprit but it would be nice to know for sure.
IanButterworth commentedon Mar 9, 2024
What is the latest master version tested here? There's been some instability
LilithHafner commentedon Mar 9, 2024
As of today (LilithHafner/Chairmarks.jl#77)
Chairmarks still takes about 2x longer to test on nightly than on 1.10.
IanButterworth commentedon Mar 9, 2024
Would you mind looking into the log line times to see where the main regressions are? Currently I'm looking into Base precompile sysimage/pkgimage coverage regression
KristofferC commentedon Mar 9, 2024
Pkg is not loaded into the process doing the testing and
shows that it is the testing itself that is slower?
So the only impact I can see from Pkg directly is the precompile files it generates. And the commits in that list touch that. So maybe it has something to do with that. Or something completely different.
LilithHafner commentedon Mar 9, 2024
In the OP
It looks like a combination of most parts of the pipeline getting slower.
Zentrik commentedon Mar 9, 2024
I only bisected the testing itself, so it's possible that the other slowdowns are from some other commit.
I think to test the different commits I'm supposed to just do
add https://github.com/JuliaLang/Pkg.jl/commit/e7d740ac82fa4c289bb58f9acb6e60eebb6d8a8e
.If that's true, the timings I was getting were far too noisy to work out what was going on.
LilithHafner commentedon Mar 9, 2024
To be honest, I don't know what the best way to bisect this within Pkg.jl is. Rebuilding Julia from source using a different Pkg version would probably do it, but I'm guessing that's a bit involved. It might be best to hand if over to @KristofferC who authored all 4 of those commits.
giordano commentedon Mar 9, 2024
Did you try to follow: https://github.com/JuliaLang/Pkg.jl/blob/e0821116e9f4364a2b37a77183e93d055a111f4d/README.md#using-the-development-version-of-pkgjl?
IanButterworth commentedon Mar 9, 2024
You don't need to change the UUID anymore, just set
--project
to the Pkg repo on startupKristofferC commentedon Mar 11, 2024
The only thing that seemed to have gotten slower here is the "Persistent task" test in Aqua.jl https://github.com/JuliaTesting/Aqua.jl/blob/b51f80035166f716b12a896587a84c89601c86dc/src/persistent_tasks.jl#L2 where it takes 40s on fb71a5d vs 3.3s on 427da5c.
Specifically, the
Pkg.precompile
in https://github.com/JuliaTesting/Aqua.jl/blob/b51f80035166f716b12a896587a84c89601c86dc/src/persistent_tasks.jl#L103 went fromto
So from what I can see, the only difference is:
which says to me we have to precompile the
Base.precompilepkgs
function (which I think @IanButterworth started doing in #53682)..KristofferC commentedon Mar 11, 2024
I don't think that is the best attitude towards someone wanting to learn how to do something. My analysis above required no advanced techniques, it was just basic stuff like adding
@time
and going down deeper into the functions.KristofferC commentedon Mar 11, 2024
The Aqua slowdown is already fixed on master by JuliaLang/Pkg.jl#3835.
Removing the Aqua test I get on master
and on 427da5c
so the change to Pkg did not affect this so I am "handing this back" now.
IanButterworth commentedon Mar 11, 2024
#53657 (comment) stated the issue remained on 53048b2 which is after JuliaLang/Pkg.jl#3835 was included in master (in a998082)
Maybe there's a more general 1.12 slowdown though, as that comment compared to 1.10, not earlier 1.12 runs
KristofferC commentedon Mar 11, 2024
Locally, for
@time Pkg.test()
I get a total time of 46s on master vs 44s on 427da5c and 39s on 1.10.LilithHafner commentedon Mar 11, 2024
I'm guessing this regression has something to do with precomilation or caching as it is present in CI but not locally.
IanButterworth commentedon Mar 11, 2024
How about local builds vs. juliaup's nightly build? (wondering if it's JULIA_CPU_TARGET not being set properly)
KristofferC commentedon Mar 11, 2024
Probably not since older cache files are invalid for new julia versions.
KristofferC commentedon Apr 16, 2024
I think the cause of the major issue was the missing Pkg precompilation, the rest is likely a dup of "general" slowdowns as reported in #53511 and #53570 so I am not sure another issue on that is worth having.