staticdata: stop compressing and storing large IR #58172
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Compressing IR adds Methods roots, which tend to become quite expensive to store after some time. We also used to store the inferred code, but since we never actually look at that again now (since now the compiler avoids using --output-ji or --sysimage-native-code=no or --pkgimages=no) it is now quite a bit of wasted space also.
vtjnash
force-pushed
the
jn/58078-reland
branch
from
f5a2e3e to
cda55a4
Compare
vtjnash
added
re-land
|
This caused PackageCompiler to become a lot slower. LinearAlgebra uses it to create a new sysimage with the custom LinearAlgebra in it but that often times out on CI now. The full time for the LinearAlgebra tests on buildkite is: Yesterday: macos 27min: https://buildkite.com/julialang/linearalgebra-dot-jl/builds/576#_ (1.13.0-DEV.429) Running the sysimage build script locally: julia> VERSION
v"1.13.0-DEV.425" # a7b8c833a3
julia> @time create_sysimage_linear_algebra();
✔ [02m:21s] PackageCompiler: creating compiler .ji image (incremental=false)
✔ [01m:23s] PackageCompiler: compiling fresh sysimage (incremental=false)
✔ [00m:45s] PackageCompiler: compiling nonincremental system image
307.620009 seconds (1.81 M allocations: 96.583 MiB, 0.03% gc time, 0.13% compilation time: 32% of which was recompilation)
----------
julia> VERSION
v"1.13.0-DEV.426" # f3264e7e3
julia> @time create_sysimage_linear_algebra()
✔ [02m:21s] PackageCompiler: creating compiler .ji image (incremental=false)
✔ [08m:23s] PackageCompiler: compiling fresh sysimage (incremental=false)
✔ [00m:54s] PackageCompiler: compiling nonincremental system image
736.690612 seconds (4.07 M allocations: 253.563 MiB, 0.01% gc time, 0.08% compilation time: 42% of which was recompilation)This is the command that got slower it seems like: |
DilumAluthge
removed
the
merge me
|
This also seems to have made the On the previous PR: cmdlineargs (5) | 691.43 | 1.32 | 0.2 | 550.58 | 1679.57versus with this PR: cmdlineargs (6) | 2747.45 | 0.30 | 0.0 | 461.12 | 1387.00or the next one: cmdlineargs (15) | 3562.13 | 0.26 | 0.0 | 343.64 | 1220.98This is causing CI timeouts, as here: https://buildkite.com/julialang/julia-master/builds/47120#01968dcc-1924-4cc6-8b04-4c07467d2414/902-1447 |
LebedevRI
pushed a commit
to LebedevRI/julia
that referenced
this pull request
May 2, 2025
Compressing IR adds Methods roots, which tend to become quite expensive to store after some time. We also used to store the inferred code, but since we never actually look at that again now (since now the compiler avoids using --output-ji or --sysimage-native-code=no or --pkgimages=no) it is now quite a bit of wasted space also. Roughly 40 MB saved on disk (165 MB -> 124 MB) if I measured correctly: ``` usr/lib/julia/sys.so : section size addr .note.gnu.build-id 36 680 .gnu.hash 52 720 .dynsym 2760 776 .dynstr 2070 3536 .gnu.version 230 5606 .gnu.version_r 160 5840 .rela.dyn 943440 6000 .rela.plt 2256 949440 .init 27 954368 .plt 1520 954400 .plt.got 8 955920 .text 17207656 955936 .fini 13 18163592 .rodata 18388 18165760 .eh_frame_hdr 315940 18184148 .eh_frame 1525972 18500088 .init_array 8 20032896 .fini_array 8 20032904 .dynamic 496 20032912 .got 128 20033408 .got.plt 776 20033536 .data 8 20034312 .bss 8 20034320 .lbss 80336 20034336 .lrodata 303064 20118768 .ldata 123852168 20425928 .comment 43 0 .debug_info 10028208 0 .debug_abbrev 18197 0 .debug_line 6581659 0 .debug_str 578898 0 .debug_ranges 10854832 0 .debug_gnu_pubnames 3065385 0 .debug_gnu_pubtypes 352332 0 Total 175737082 ```
|
Yes, Jeff and I were noticing that recently too, and he planned to deprecate that command line option, or at least remove one of the two tests for it and put the other in a separate file |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Compressing IR adds Methods roots, which tend to become quite expensive to store after some time. We also used to store the inferred code, but since we never actually look at that again now (since now the compiler avoids using --output-ji or --sysimage-native-code=no or --pkgimages=no) it is now quite a bit of wasted space also.
Roughly 40 MB saved on disk (165 MB -> 124 MB) if I measured correctly: