Errors when Python using juliacall and jl.evalfile to call a Julia package that then uses PythonCall

[ScottPJones]

I'll provide more information later - I'm trying to create a MWE, but essentially it fails in init_pointers, saying that PyObject_Call cannot be found, if the julia code is called (via juliacall, with jl.evalfile).
If the same code is started directly from Julia (i.e. simply uses PythonCall), everything works as expected.
This was using various versions of Julia (1.7, 1.8), and various versions of Python (3.7, 3.8)

[cjdoris]

Yeah an MWE would be great thanks.

[mrufsvold]

MWE for me:

On Windows 10, using Julia 1.8.2, PythonCall 0.9.9, and Python 3.9.13:

> pipenv install --python 3.9
> pipenv install juliacall
> julia
> using Pkg
> Pkg.generate("MyProject")
> Pkg.develop("./MyProject")
> Pkg.activate("./MyProject")
> Pkg.add("PythonCall")

In ./MyProject/src/MyProject.jl :

module PythonCallErrorExample
using PythonCall

function test_py_func(input_val::PythonCall.PyArray)
    v = PythonCall.pyconvert(Vector{Int}, input_val)
    return v
end
end

In ./main.py

from juliacall import Main as jl
jl.seval("using PythonCallErrorExample")

Then pipenv run python main.py throws:

ERROR: LoadError: InitError: could not load symbol "PyObject_Call":
The specified module could not be found.
Stacktrace:
  [1] #dlsym#1
    @ .\libdl.jl:59 [inlined]
  [2] dlsym
    @ .\libdl.jl:56 [inlined]
  [3] init_pointers(p::PythonCall.C.CAPIPointers, lib::Ptr{Nothing}) (repeats 2 times)
    @ PythonCall.C C:\Users\mrufsvold\.julia\packages\PythonCall\Td3SH\src\cpython\pointers.jl:284
  [4] init_context()
    @ PythonCall.C C:\Users\mrufsvold\.julia\packages\PythonCall\Td3SH\src\cpython\context.jl:40
  [5] __init__()
    @ PythonCall.C C:\Users\mrufsvold\.julia\packages\PythonCall\Td3SH\src\cpython\CPython.jl:21
  [6] _include_from_serialized(pkg::Base.PkgId, path::String, depmods::Vector{Any})
    @ Base .\loading.jl:831
  [7] _require_search_from_serialized(pkg::Base.PkgId, sourcepath::String, build_id::UInt64)
    @ Base .\loading.jl:1039
  [8] _require(pkg::Base.PkgId)
    @ Base .\loading.jl:1315
  [9] _require_prelocked(uuidkey::Base.PkgId)
    @ Base .\loading.jl:1200
 [10] macro expansion
    @ .\loading.jl:1180 [inlined]
 [11] macro expansion
    @ .\lock.jl:223 [inlined]
 [12] require(into::Module, mod::Symbol)
    @ Base .\loading.jl:1144
 [13] include
    @ .\Base.jl:419 [inlined]
 [14] include_package_for_output(pkg::Base.PkgId, input::String, depot_path::Vector{String}, dl_load_path::Vector{String}, load_path::Vector{String}, concrete_deps::Vector{Pair{Base.PkgId, UInt64}}, source::Nothing)
    @ Base .\loading.jl:1554
 [15] top-level scope
    @ stdin:1
during initialization of module C
in expression starting at C:\Users\mrufsvold\Projects\juliacall_error_mwe\PythonCallErrorExample\src\PythonCallErrorExample.jl:1
in expression starting at stdin:1
Traceback (most recent call last):
  File "C:\Users\mrufsvold\Projects\juliacall_error_mwe\main.py", line 3, in <module>
    jl.seval("using PythonCallErrorExample")
  File "C:\Users\mrufsvold\.julia\packages\PythonCall\Td3SH\src\jlwrap\module.jl", line 25, in seval
    return self._jl_callmethod($(pyjl_methodnum(pyjlmodule_seval)), expr)
juliacall.JuliaError: Failed to precompile PythonCallErrorExample [918b8bb7-8ec8-4388-8dd5-9379565e07cf] to C:\Users\mrufsvold\.julia\compiled\v1.8\PythonCallErrorExample\jl_4E2E.tmp.
Stacktrace:
  [1] error(s::String)
    @ Base .\error.jl:35
  [2] compilecache(pkg::Base.PkgId, path::String, internal_stderr::IO, internal_stdout::IO, keep_loaded_modules::Bool)
    @ Base .\loading.jl:1705
  [3] compilecache
    @ .\loading.jl:1649 [inlined]
  [4] _require(pkg::Base.PkgId)
    @ Base .\loading.jl:1337
  [5] _require_prelocked(uuidkey::Base.PkgId)
    @ Base .\loading.jl:1200
  [6] macro expansion
    @ .\loading.jl:1180 [inlined]
  [7] macro expansion
    @ .\lock.jl:223 [inlined]
  [8] require(into::Module, mod::Symbol)
    @ Base .\loading.jl:1144
  [9] eval
    @ .\boot.jl:368 [inlined]
 [10] eval
    @ .\Base.jl:65 [inlined]
 [11] pyjlmodule_seval(self::Module, expr::PythonCall.Py)
    @ PythonCall C:\Users\mrufsvold\.julia\packages\PythonCall\Td3SH\src\jlwrap\module.jl:13
 [12] _pyjl_callmethod(f::Any, self_::Ptr{PythonCall.C.PyObject}, args_::Ptr{PythonCall.C.PyObject}, nargs::Int64)
    @ PythonCall C:\Users\mrufsvold\.julia\packages\PythonCall\Td3SH\src\jlwrap\base.jl:62
 [13] _pyjl_callmethod(o::Ptr{PythonCall.C.PyObject}, args::Ptr{PythonCall.C.PyObject})
    @ PythonCall.C C:\Users\mrufsvold\.julia\packages\PythonCall\Td3SH\src\cpython\jlwrap.jl:47

FWIW, my Julia code was working just fine being called from Python until I tried to wrap it in a package to get some improved precompilation.

Hope this helps!

[mrufsvold]

Hey, @cjdoris , I'm wondering if you have a hunch about where to start in this. I'm happy to take a crack at a PR if you're busy, but I'm at a loss on this.

Don't mean to rush you if you've got other things on your plate right now, just want to help if I can!

[cjdoris]

Sorry I've been on holiday then was ill. Will take a look in the next week or so.

[mrufsvold]

Sorry to hear that! Not a problem at all. Thanks for all you work!

[ScottPJones]

After coming back to the problem, I've found that making sure it finds a correct juliapkg.json file for the Python script seems to prevent the error. I'm investigating that further.

[brian-dellabetta]

@mrufsvold @cjdoris I am hitting this same error with a julia project that has PythonCall as a dependency.I have been having a terrible time trying to reproduce, but now I believe I know why:

I can circumvent this error by explicitly precompiling the julia project code beforehand, in a separate process outside of juliacall, at which point jl.seval("using MyProject") succeeds. But without explicitly precompiling it will trigger the same error @mrufsvold is reporting -- InitError: could not load symbol "PyObject_Call".

Any advice on how to start on this would be super helpful, I would like to contribute to this project as well. Thanks @cjdoris for creating such a handy tool!

[brian-dellabetta]

For anyone who comes across this issue, there is a pretty simple workaround, just put this before your juliacall imports/commands:

import juliapkg
import subprocess
subprocess.call(
      f"""{juliapkg.executable()} --project={juliapkg.project()} -e 'using MyProject' """,
      # TODO get stdout/stderr to show up
      shell=True,
  )

Will update when i figure out a clean way to display stdout/stderr

[cjdoris]

Thanks for the hints everyone, I figured it out. It's now fixed and I've made a release.

Upgrade to juliapkg 0.1.10 and juliacall 0.9.12 to try it out.

I'm closing this issue now, but please comment if it's still not working for you.

[brian-dellabetta]

I upgraded to PythonCall.jl / juliacall 0.9.12 and everything works as hoped for. Thanks @cjdoris !