GC: Large amounts of GC time spent scanning permanent pkgimage allocations

[topolarity]

On current Julia master, 40% of our time in using DeepPumas is spent in GC:

114.447107 seconds (144.80 M allocations: 11.006 GiB, 43.24% gc time, 24.67% compilation time: 61% of which was recompilation)

OmniPackage similarly spends ~30% of its time in the GC, and the GC mark phase seems to be the primary culprit.

@vtjnash suspects that this GC time is largely spent scanning over the pkgimage objects, and that we might be able to reduce it.

[vchuravy]

Do you have a profile to share? PProf or Vtunes?

[ViralBShah]

cc @chriselrod

[gbaraldi]

Loading https://github.com/JuliaComputing/OmniPackage.jl has very similar results just for reference.

[chriselrod]

Yes, OmniPackage is almost as big as DeepPumas, and should therefore should be able to serve as an open source reproducer.

[JeffBezanson]

Iterating over the list of loaded package images for ~every object mark is totally unacceptable. We cannot just casually tank GC performance like this.

[KristofferC]

Can someone explain why we need to iterate over the loaded package images in 1.9 but we didn't need to do the equivalent on 1.8? Naively, it doesn't feel like the precompile format should change the work the GC has to do.

[vchuravy]

The serialization format is now "in place". So Julia objects that are stored in package images are perma allocated. This is how in 1.8 the system-image worked. In 1.9 we reuse the same serialization format as for the sysimage for package images.

[JeffBezanson]

Could we just not treat package image data as permanent for now? It does not seem to be worth the cost (though an experiment might be in order).

[vchuravy]

Just packages or also sysimg?