External Pkg always recompiles upon loading

[KristofferC]

While being in a Pkg.jl repo:

❯ julia +nightly --project -q

julia> using Pkg
[ Info: Precompiling Pkg [44cfe95a-1eb2-52ea-b672-e2afdf69b78f] (cache misses: wrong source (1), mismatched flags (1))

❯ julia +nightly --project -q
julia> using Pkg
[ Info: Precompiling Pkg [44cfe95a-1eb2-52ea-b672-e2afdf69b78f] (cache misses: for different buildid (2), mismatched flags (2))

❯ julia +nightly --project -q
julia> using Pkg
[ Info: Precompiling Pkg [44cfe95a-1eb2-52ea-b672-e2afdf69b78f] (cache misses: for different buildid (2), mismatched flags (2))

The two original misses are from the bundled cache files so those are expected. This is also a bit weird:

julia> pkg = Base.identify_package("Pkg")
Pkg [44cfe95a-1eb2-52ea-b672-e2afdf69b78f]

julia> Base.isprecompiled(pkg)
true

julia> using Pkg
[ Info: Precompiling Pkg [44cfe95a-1eb2-52ea-b672-e2afdf69b78f] (cache misses: for different buildid (2), mismatched flags (2))

[vchuravy]

What does JULIA_DEBUG=loading say?

[KristofferC]

┌ Debug: Rejecting cache file /home/kc/.julia/compiled/v1.12/Pkg/tUTdb_sLFSB.ji because required dependency Base.PkgId(Base.UUID("2a0f44e3-6c83-55bd-87e4-b1978d98bd5f"), "Base64") failed to load from cache file for /home/kc/.julia/juliaup/julia-nightly/share/julia/compiled/v1.12/Base64/D7K0n_Ip9tx.ji.
│   exception = Error reading package image file.

weird..

[KristofferC]

I am also surprised this doesn't print anything at all for Base64:

julia> ENV["JULIA_DEBUG"] = "loading"
"loading"

julia> using Base64

julia> using Example
┌ Debug: Loading object cache file /home/kc/.julia/compiled/v1.12/Example/lLvWP_sLFSB.so for Example [7876af07-990d-54b4-ab0e-23690620f79a]
└ @ Base loading.jl:1227

[KristofferC]

I had some thought that the nightly build juliaup uses was corrupt somehow but building a local Julia on master it gives the exact same behavior.

[IanButterworth]

Base64 is an indirect dep of REPL, so it's already loaded?

[KristofferC]

This method of loading Pkg seems to work:

❯ julia +nightly --project -iqe 'using REPL; using Pkg'
julia> pkgdir(Pkg)
"/home/kc/JuliaPkgs/Pkg.jl"

[KristofferC]

Base64 is an indirect dep of REPL, so it's already loaded?

Okay, that makes sense, I looked in Base.loaded_modules and it wasn't there so I assumed it wasn't loaded already:

julia> Base.loaded_modules
Dict{Base.PkgId, Module} with 12 entries:
  PkgId(UUID("4536629a-c528-5b80-bd46-f80d51c5b363"), "OpenBLAS_jll")          => OpenBLAS_jll
  PkgId(UUID("8f399da3-3557-5675-b5ff-fb832c97cbdb"), "Libdl")                 => Libdl
  PkgId(UUID("ea8e919c-243c-51af-8825-aaa63cd721ce"), "SHA")                   => SHA
  PkgId(UUID("7b1f6079-737a-58dc-b8bc-7a2ca5c1b5ee"), "FileWatching")          => FileWatching
  PkgId(UUID("56f22d72-fd6d-98f1-02f0-08ddc0907c33"), "Artifacts")             => Artifacts
  PkgId(UUID("8e850b90-86db-534c-a0d3-1478176c7d93"), "libblastrampoline_jll") => libblastrampoline_jll
  PkgId(nothing, "Main")                                                       => Main
  PkgId(nothing, "Core")                                                       => Core
  PkgId(UUID("37e2e46d-f89d-539d-b4ee-838fcccc9c8e"), "LinearAlgebra")         => LinearAlgebra
  PkgId(nothing, "Base")                                                       => Base
  PkgId(UUID("9a3f8284-a2c9-5f02-9a11-845980a1fd5c"), "Random")                => Random
  PkgId(UUID("6462fe0b-24de-5631-8697-dd941f90decc"), "Sockets")               => Sockets

so I guess things that are loaded as part of require_stdlib are now invisible but when you do using on them they grab the invisible module and makes it visible?

But that shouldn't make Pkg have to compile over and over due to error when reading the Base64 pkgimage file I guess.

[fatteneder]

┌ Debug: Rejecting cache file /home/kc/.julia/compiled/v1.12/Pkg/tUTdb_sLFSB.ji because required dependency Base.PkgId(Base.UUID("2a0f44e3-6c83-55bd-87e4-b1978d98bd5f"), "Base64") failed to load from cache file for /home/kc/.julia/juliaup/julia-nightly/share/julia/compiled/v1.12/Base64/D7K0n_Ip9tx.ji.
│   exception = Error reading package image file.

weird..

I have seen this too when trying to debug JuliaPy/PythonCall.jl#513.
There @kshyatt nailed down the problem to 71fa11f.

[KristofferC]

There @kshyatt nailed down the problem to 71fa11f.

Same thing happens when I revert that commit.

[kshyatt]

I ran a git bisect for the function I needed not being exported, and it fingered this commit. Maybe multiple things are wrong?

…

On Wed, Jun 26, 2024 at 9:26 AM Kristoffer Carlsson < ***@***.***> wrote: There @kshyatt <https://github.com/kshyatt> nailed down the problem to 71fa11f <71fa11f> . Same thing happens when I revert that commit. — Reply to this email directly, view it on GitHub <#54940 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAGKJY5FPJRFDYJSRJHFI4TZJK6P5AVCNFSM6AAAAABJ5V7TROVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCOJRGY4TSNJQGY> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[KristofferC]

Yes, I think they are unrelated.

[KristofferC]

Same happens with OhMyREPL on nightly:

┌ Debug: Loading object cache file /Users/kristoffercarlsson/.julia/juliaup/julia-nightly/share/julia/compiled/v1.12/StyledStrings/UcVoM_44pG1.dylib for StyledStrings [f489334b-da3d-4c2e-b8f0-e476e12c162b]
└ @ Base loading.jl:1227
┌ Debug: Rejecting cache file /Users/kristoffercarlsson/.julia/compiled/v1.12/OhMyREPL/08e1i_J2Ppx.ji because required dependency Base.PkgId(Base.UUID("f489334b-da3d-4c2e-b8f0-e476e12c162b"), "StyledStrings") failed to load from cache file for /Users/kristoffercarlsson/.julia/juliaup/julia-nightly/share/julia/compiled/v1.12/StyledStrings/UcVoM_44pG1.ji.
│   exception = Error reading package image file.
└ @ Base loading.jl:1962
┌ Debug: Loading object cache file /Users/kristoffercarlsson/.julia/juliaup/julia-nightly/share/julia/compiled/v1.12/StyledStrings/UcVoM_44pG1.dylib for StyledStrings [f489334b-da3d-4c2e-b8f0-e476e12c162b]
└ @ Base loading.jl:1227
┌ Debug: Rejecting cache file /Users/kristoffercarlsson/.julia/compiled/v1.12/OhMyREPL/08e1i_J2Ppx.ji because required dependency Base.PkgId(Base.UUID("f489334b-da3d-4c2e-b8f0-e476e12c162b"), "StyledStrings") failed to load from cache file for /Users/kristoffercarlsson/.julia/juliaup/julia-nightly/share/julia/compiled/v1.12/StyledStrings/UcVoM_44pG1.ji.
│   exception = Error reading package image file.