2x regression in testing time of small package

[KristofferC]

Looking at the time to take to test a simple package (https://github.com/KristofferC/LazilyInitializedFields.jl/actions/runs/8072420307/job/22079073318?pr=30) I see an over 2x regression end to end.

Some numbers:

# nightly
 18 dependencies successfully precompiled in 54 seconds. 31 already precompiled.
# 1.10
 18 dependencies successfully precompiled in 21 seconds. 3 already precompiled.

#nightly
Doctests: LazilyInitializedFields |    1      1  25.3s
# 1.10
Doctests: LazilyInitializedFields |    1      1  10.2s

It is possible this is mostly the same as #52592 which was closed, first without a comment and then with

There seemed to be nothing here to solve after looking into it

but a 2x regression end to end for a very standard simple package seems like something that should be solved

[KristofferC]

Adding a @time to the doctest shows that there is quite a lot of invalidations going on so maybe something with moving out stdlibs.

[KristofferC]

I think a big reason for the slowdown in the end to end testing is that it seems that precompiled code is turned off for stdlibs when running with code coverage now?

❯ julia +1.11 --code-coverage=user --project -q
julia> @time Pkg.status()
...
  6.017179 seconds (2.55 M allocations: 156.633 MiB, 0.42% gc time, 95.98% compilation time)


❯ julia +1.11 --project -q
julia> @time Pkg.status()
...
  0.524139 seconds (1.85 M allocations: 127.566 MiB, 11.85% gc time, 47.12% compilation time)

[IanButterworth]

Yeah. user is generally too crude as that includes stdlibs. Better to do it by path.

But even with the path approach, as soon as a package is loaded within that path, any further loads don't use pkgimage native code either. The fix is to load your test package last in the test suite.

[IanButterworth]

Perhaps Aqua should check that the test package is loaded last

[KristofferC]

I added some data in KristofferC/LazilyInitializedFields.jl#38. The last commit (loads the testing package last but it doesn't seem to have much effect.

[IanButterworth]

Regarding the coverage-on regression the likely relevant PRs are #53439 #53457 @vchuravy @vtjnash

[KristofferC]

I'm going to fold this one into #53570