PythonCall in sysimage

[PallHaraldsson]

I'm trying everything to get startup down, and now:

https://github.com/petvana/AutoSysimages.jl

It works for me. First the good news:

julia> @time using PythonCall
  0.010378 seconds (8.97 k allocations: 565.888 KiB)

and with -O0:

julia> @time using PythonCall
  0.003670 seconds (8.97 k allocations: 565.888 KiB)

Some (but not all, since 5.7 sec otherwise) of the startup-cost has been shifted to startup of Julia itself with the sysimage:

$ hyperfine '~/julia-1.9-DEV-b01fd3367b/bin/julia -J /home/pharaldsson/.julia/asysimg/1.9.0-DEV.1694/30JkAN/asysimg-2022-10-30T22-32-01.991.so -e ""'
Benchmark 1: ~/julia-1.9-DEV-b01fd3367b/bin/julia -J /home/pharaldsson/.julia/asysimg/1.9.0-DEV.1694/30JkAN/asysimg-2022-10-30T22-32-01.991.so -e ""
  Time (mean ± σ):      3.116 s ±  0.031 s    [User: 3.791 s, System: 1.184 s]
  Range (min … max):    3.067 s …  3.151 s    10 runs

And 1.91 sec. for -O0.

I believe some work on the package could improve this, with or without a sysimage. The sysimage is 185 MB vs 209 MB for Julia's own 1.9 sysimage (sys.so, not sure why it's larger) vs a 75 MB sysimage I made for other reasons (to cut startup of Julia in half for benchmarking/scripting reasons).

AutoSysimages is really easy to use (at first I used build_sysimage() directly which likely is incorrect), just follow the docs, then for me this worked:

$ PATH=/home/pharaldsson/.local/bin/:$PATH asysimg

Julia 1.9 alone seems a bit faster than 1.8, at least julia -O0:

julia> @time using PythonCall
  3.970856 seconds (2.55 M allocations: 170.295 MiB, 3.47% gc time, 81.35% compilation time: <1% of which was recompilation)

A question related to it, I got 1.9-DEV to try, and it should also work with PythonCall.jl (and does, on its own), but is this expected:

  Prefix: /home/pharaldsson/.julia/environments/v1.9/.CondaPkg/env

  Updating specs:

   - conda-forge::libstdcxx-ng[version='>=3.4,<11.4']
   - conda-forge::python[version='>=3.7,<4',build=*cpython*]

  + _libgcc_mutex           0.1  conda_forge         conda-forge/linux-64     Cached
  + _openmp_mutex           4.5  2_gnu               conda-forge/linux-64     Cached
  + bzip2                 1.0.8  h7f98852_4          conda-forge/linux-64     Cached
  + ca-certificates   2022.9.24  ha878542_0          conda-forge/linux-64     Cached
  + ld_impl_linux-64       2.39  hc81fddc_0          conda-forge/linux-64      778kB
  + libffi                3.4.2  h7f98852_5          conda-forge/linux-64     Cached
  + libgcc-ng            12.2.0  h65d4601_19         conda-forge/linux-64      954kB
  + libgomp              12.2.0  h65d4601_19         conda-forge/linux-64      466kB
  + libnsl                2.0.0  h7f98852_0          conda-forge/linux-64     Cached
  + libsqlite            3.39.4  h753d276_0          conda-forge/linux-64      822kB
  + libstdcxx-ng         11.3.0  h239ccf8_19         conda-forge/linux-64        4MB
  + libuuid              2.32.1  h7f98852_1000       conda-forge/linux-64     Cached
  + libzlib              1.2.13  h166bdaf_4          conda-forge/linux-64       66kB
  + ncurses                 6.3  h27087fc_1          conda-forge/linux-64        1MB
  + openssl               3.0.5  h166bdaf_2          conda-forge/linux-64     Cached
  + pip                    22.3  pyhd8ed1ab_0        conda-forge/noarch          2MB
  + python               3.11.0  ha86cf86_0_cpython  conda-forge/linux-64       38MB
  + readline              8.1.2  h0f457ee_0          conda-forge/linux-64      298kB
  + setuptools           65.5.0  pyhd8ed1ab_0        conda-forge/noarch        787kB
  + tk                   8.6.12  h27826a3_0          conda-forge/linux-64        3MB
  + tzdata                2022e  h191b570_0          conda-forge/noarch        121kB
  + wheel                0.37.1  pyhd8ed1ab_0        conda-forge/noarch         32kB
  + xz                    5.2.6  h166bdaf_0          conda-forge/linux-64     Cached

I see issue here related to libstdcxx-ng, and I believe maybe you don't need to download it, at least soon (work almost done on julia master).

Good to see Python 3.11.0 is done and getting it with your package. It's not a huge deal to trim the list above, but is e.g. tk, readline, ncurses, libgcc-ng, libsqlite really needed or basically all of it? Is pip redundant with conda/mamba, i.e. as you use it?

Have you looked at CircuitPython/MicroPython? I suppose that can't work with your package (JuliaCall), though maybe StaticCompiler.jl.

[PallHaraldsson]

About your use of libstdcxx maybe it's related to:
JuliaLang/julia#46345

See its (still open) [CompilerSupportLibraries_jll] linked PR fixing (I believe that PR done, except for some minor Windows holdup).

[cjdoris]

I've been experimenting and believe I can reduce the startup time by 1-2 seconds quite easily. Watch this space!

The libstdcxx-ng dependency is there not because it needs to be downloaded, but because we need to ensure that any packages which require libstdc++ are compatible with the one used by Julia.

All those other packages are just what Conda installs when you ask for Python, I can't do anything about those.

Feel free to reopen if you've more questions.