pyjulia segfaults with multiprocessing

[kaandocal]

Calling Julia code using Python's multiprocessing module crashes the program (SIGSEGV). I am using Julia 1.2.0, Python 3.7.3 and pyjulia 0.4.1 on Linux.

A minimal example is as follows:

import multiprocessing as mp
import julia

def test(*args):
    j = julia.Julia()

    myfunc = j.eval("function test() return 5; end")
    myfunc()

with mp.Pool(10) as p:
    p.map(test, range(10))

This program crashes with a high probability which increases with the number of children. Based on the actual code I need to run it seems like the crash occurs whenever a Julia-defined function is called for the first time in a process; if this succeeds the program does not seem to crash even after many iterations.

Warning: `getindex(o::PyObject, s::Symbol)` is deprecated in favor of dot overloading (`getproperty`) so elements should now be accessed as e.g. `o.s` instead of `o[:s]`.
│   caller = top-level scope at none:4
└ @ Core none:4

signal (11): Segmentation fault
in expression starting at none:1
unknown function (ip: 0xffffffffffffffff)
jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2197
jl_apply at /buildworker/worker/package_linux64/build/src/julia.h:1614 [inlined]
call_require at /buildworker/worker/package_linux64/build/src/toplevel.c:399 [inlined]
eval_import_path at /buildworker/worker/package_linux64/build/src/toplevel.c:436
jl_toplevel_eval_flex at /buildworker/worker/package_linux64/build/src/toplevel.c:656
jl_toplevel_eval_flex at /buildworker/worker/package_linux64/build/src/toplevel.c:764
jl_toplevel_eval_in at /buildworker/worker/package_linux64/build/src/toplevel.c:844
jl_eval_string at /buildworker/worker/package_linux64/build/src/jlapi.c:94
ffi_call_unix64 at /home/user/.miniconda3/lib/python3.7/lib-dynload/../../libffi.so.6 (unknown line)
ffi_call at /home/user/.miniconda3/lib/python3.7/lib-dynload/../../libffi.so.6 (unknown line)
_call_function_pointer at /usr/local/src/conda/python-3.7.3/Modules/_ctypes/callproc.c:827 [inlined]
_ctypes_callproc at /usr/local/src/conda/python-3.7.3/Modules/_ctypes/callproc.c:1184
PyCFuncPtr_call at /usr/local/src/conda/python-3.7.3/Modules/_ctypes/_ctypes.c:3969
_PyObject_FastCallKeywords at /home/user/.miniconda3/lib/libpython3.7m.so.1.0 (unknown line)
call_function.lto_priv.1536 at /home/user/.miniconda3/lib/libpython3.7m.so.1.0 (unknown line)
_PyEval_EvalFrameDefault at /home/user/.miniconda3/lib/libpython3.7m.so.1.0 (unknown line)
function_code_fastcall at /home/user/.miniconda3/lib/libpython3.7m.so.1.0 (unknown line)
call_function.lto_priv.1536 at /home/user/.miniconda3/lib/libpython3.7m.so.1.0 (unknown line)
_PyEval_EvalFrameDefault at /home/user/.miniconda3/lib/libpython3.7m.so.1.0 (unknown line)
_PyEval_EvalCodeWithName at /home/user/.miniconda3/lib/libpython3.7m.so.1.0 (unknown line)
_PyFunction_FastCallDict at /home/user/.miniconda3/lib/libpython3.7m.so.1.0 (unknown line)
_PyObject_Call_Prepend at /home/user/.miniconda3/lib/libpython3.7m.so.1.0 (unknown line)
slot_tp_init at /home/user/.miniconda3/lib/libpython3.7m.so.1.0 (unknown line)
type_call.lto_priv.151 at /home/user/.miniconda3/lib/libpython3.7m.so.1.0 (unknown line)
PyObject_Call at /home/user/.miniconda3/lib/libpython3.7m.so.1.0 (unknown line)
_PyEval_EvalFrameDefault at /home/user/.miniconda3/lib/libpython3.7m.so.1.0 (unknown line)
_PyEval_EvalCodeWithName at /home/user/.miniconda3/lib/libpython3.7m.so.1.0 (unknown line)
_PyFunction_FastCallDict at /home/user/.miniconda3/lib/libpython3.7m.so.1.0 (unknown line)
_PyObject_Call_Prepend at /home/user/.miniconda3/lib/libpython3.7m.so.1.0 (unknown line)
slot_tp_init at /home/user/.miniconda3/lib/libpython3.7m.so.1.0 (unknown line)
type_call.lto_priv.151 at /home/user/.miniconda3/lib/libpython3.7m.so.1.0 (unknown line)
_PyObject_FastCallKeywords at /home/user/.miniconda3/lib/libpython3.7m.so.1.0 (unknown line)
call_function.lto_priv.1536 at /home/user/.miniconda3/lib/libpython3.7m.so.1.0 (unknown line)
_PyEval_EvalFrameDefault at /home/user/.miniconda3/lib/libpython3.7m.so.1.0 (unknown line)
_PyEval_EvalCodeWithName at /home/user/.miniconda3/lib/libpython3.7m.so.1.0 (unknown line)
_PyFunction_FastCallDict at /home/user/.miniconda3/lib/libpython3.7m.so.1.0 (unknown line)
map_next.lto_priv.527 at /home/user/.miniconda3/lib/libpython3.7m.so.1.0 (unknown line)
list_extend.lto_priv.888 at /home/user/.miniconda3/lib/libpython3.7m.so.1.0 (unknown line)
list___init__.lto_priv.379 at /home/user/.miniconda3/lib/libpython3.7m.so.1.0 (unknown line)
type_call.lto_priv.151 at /home/user/.miniconda3/lib/libpython3.7m.so.1.0 (unknown line)
_PyObject_FastCallKeywords at /home/user/.miniconda3/lib/libpython3.7m.so.1.0 (unknown line)
call_function.lto_priv.1536 at /home/user/.miniconda3/lib/libpython3.7m.so.1.0 (unknown line)
_PyEval_EvalFrameDefault at /home/user/.miniconda3/lib/libpython3.7m.so.1.0 (unknown line)
function_code_fastcall at /home/user/.miniconda3/lib/libpython3.7m.so.1.0 (unknown line)
_PyFunction_FastCallDict at /home/user/.miniconda3/lib/libpython3.7m.so.1.0 (unknown line)
_PyEval_EvalFrameDefault at /home/user/.miniconda3/lib/libpython3.7m.so.1.0 (unknown line)
_PyEval_EvalCodeWithName at /home/user/.miniconda3/lib/libpython3.7m.so.1.0 (unknown line)
_PyFunction_FastCallDict at /home/user/.miniconda3/lib/libpython3.7m.so.1.0 (unknown line)
_PyEval_EvalFrameDefault at /home/user/.miniconda3/lib/libpython3.7m.so.1.0 (unknown line)
function_code_fastcall at /home/user/.miniconda3/lib/libpython3.7m.so.1.0 (unknown line)
call_function.lto_priv.1536 at /home/user/.miniconda3/lib/libpython3.7m.so.1.0 (unknown line)
_PyEval_EvalFrameDefault at /home/user/.miniconda3/lib/libpython3.7m.so.1.0 (unknown line)
function_code_fastcall at /home/user/.miniconda3/lib/libpython3.7m.so.1.0 (unknown line)
call_function.lto_priv.1536 at /home/user/.miniconda3/lib/libpython3.7m.so.1.0 (unknown line)
_PyEval_EvalFrameDefault at /home/user/.miniconda3/lib/libpython3.7m.so.1.0 (unknown line)
function_code_fastcall at /home/user/.miniconda3/lib/libpython3.7m.so.1.0 (unknown line)
call_function.lto_priv.1536 at /home/user/.miniconda3/lib/libpython3.7m.so.1.0 (unknown line)
_PyEval_EvalFrameDefault at /home/user/.miniconda3/lib/libpython3.7m.so.1.0 (unknown line)
function_code_fastcall at /home/user/.miniconda3/lib/libpython3.7m.so.1.0 (unknown line)
_PyFunction_FastCallDict at /home/user/.miniconda3/lib/libpython3.7m.so.1.0 (unknown line)
_PyObject_Call_Prepend at /home/user/.miniconda3/lib/libpython3.7m.so.1.0 (unknown line)
slot_tp_init at /home/user/.miniconda3/lib/libpython3.7m.so.1.0 (unknown line)
type_call.lto_priv.151 at /home/user/.miniconda3/lib/libpython3.7m.so.1.0 (unknown line)
_PyObject_FastCallKeywords at /home/user/.miniconda3/lib/libpython3.7m.so.1.0 (unknown line)
call_function.lto_priv.1536 at /home/user/.miniconda3/lib/libpython3.7m.so.1.0 (unknown line)
_PyEval_EvalFrameDefault at /home/user/.miniconda3/lib/libpython3.7m.so.1.0 (unknown line)
function_code_fastcall at /home/user/.miniconda3/lib/libpython3.7m.so.1.0 (unknown line)
call_function.lto_priv.1536 at /home/user/.miniconda3/lib/libpython3.7m.so.1.0 (unknown line)
_PyEval_EvalFrameDefault at /home/user/.miniconda3/lib/libpython3.7m.so.1.0 (unknown line)
function_code_fastcall at /home/user/.miniconda3/lib/libpython3.7m.so.1.0 (unknown line)
call_function.lto_priv.1536 at /home/user/.miniconda3/lib/libpython3.7m.so.1.0 (unknown line)
_PyEval_EvalFrameDefault at /home/user/.miniconda3/lib/libpython3.7m.so.1.0 (unknown line)
function_code_fastcall at /home/user/.miniconda3/lib/libpython3.7m.so.1.0 (unknown line)
call_function.lto_priv.1536 at /home/user/.miniconda3/lib/libpython3.7m.so.1.0 (unknown line)
_PyEval_EvalFrameDefault at /home/user/.miniconda3/lib/libpython3.7m.so.1.0 (unknown line)
_PyEval_EvalCodeWithName at /home/user/.miniconda3/lib/libpython3.7m.so.1.0 (unknown line)
_PyFunction_FastCallDict at /home/user/.miniconda3/lib/libpython3.7m.so.1.0 (unknown line)
_PyObject_Call_Prepend at /home/user/.miniconda3/lib/libpython3.7m.so.1.0 (unknown line)
slot_tp_init at /home/user/.miniconda3/lib/libpython3.7m.so.1.0 (unknown line)
type_call.lto_priv.151 at /home/user/.miniconda3/lib/libpython3.7m.so.1.0 (unknown line)
_PyObject_FastCallKeywords at /home/user/.miniconda3/lib/libpython3.7m.so.1.0 (unknown line)
call_function.lto_priv.1536 at /home/user/.miniconda3/lib/libpython3.7m.so.1.0 (unknown line)
_PyEval_EvalFrameDefault at /home/user/.miniconda3/lib/libpython3.7m.so.1.0 (unknown line)
_PyEval_EvalCodeWithName at /home/user/.miniconda3/lib/libpython3.7m.so.1.0 (unknown line)
_PyFunction_FastCallKeywords at /home/user/.miniconda3/lib/libpython3.7m.so.1.0 (unknown line)
call_function.lto_priv.1536 at /home/user/.miniconda3/lib/libpython3.7m.so.1.0 (unknown line)
_PyEval_EvalFrameDefault at /home/user/.miniconda3/lib/libpython3.7m.so.1.0 (unknown line)
_PyEval_EvalCodeWithName at /home/user/.miniconda3/lib/libpython3.7m.so.1.0 (unknown line)
PyEval_EvalCodeEx at /home/user/.miniconda3/lib/libpython3.7m.so.1.0 (unknown line)
PyEval_EvalCode at /home/user/.miniconda3/lib/libpython3.7m.so.1.0 (unknown line)
builtin_exec at /home/user/.miniconda3/lib/libpython3.7m.so.1.0 (unknown line)
_PyMethodDef_RawFastCallKeywords at /home/user/.miniconda3/lib/libpython3.7m.so.1.0 (unknown line)
_PyCFunction_FastCallKeywords at /home/user/.miniconda3/lib/libpython3.7m.so.1.0 (unknown line)
call_function.lto_priv.1536 at /home/user/.miniconda3/lib/libpython3.7m.so.1.0 (unknown line)
_PyEval_EvalFrameDefault at /home/user/.miniconda3/lib/libpython3.7m.so.1.0 (unknown line)
_PyEval_EvalCodeWithName at /home/user/.miniconda3/lib/libpython3.7m.so.1.0 (unknown line)
_PyFunction_FastCallKeywords at /home/user/.miniconda3/lib/libpython3.7m.so.1.0 (unknown line)
call_function.lto_priv.1536 at /home/user/.miniconda3/lib/libpython3.7m.so.1.0 (unknown line)
_PyEval_EvalFrameDefault at /home/user/.miniconda3/lib/libpython3.7m.so.1.0 (unknown line)
_PyEval_EvalCodeWithName at /home/user/.miniconda3/lib/libpython3.7m.so.1.0 (unknown line)
_PyFunction_FastCallKeywords at /home/user/.miniconda3/lib/libpython3.7m.so.1.0 (unknown line)
call_function.lto_priv.1536 at /home/user/.miniconda3/lib/libpython3.7m.so.1.0 (unknown line)
_PyEval_EvalFrameDefault at /home/user/.miniconda3/lib/libpython3.7m.so.1.0 (unknown line)
_PyEval_EvalCodeWithName at /home/user/.miniconda3/lib/libpython3.7m.so.1.0 (unknown line)
_PyFunction_FastCallKeywords at /home/user/.miniconda3/lib/libpython3.7m.so.1.0 (unknown line)
call_function.lto_priv.1536 at /home/user/.miniconda3/lib/libpython3.7m.so.1.0 (unknown line)
_PyEval_EvalFrameDefault at /home/user/.miniconda3/lib/libpython3.7m.so.1.0 (unknown line)
_PyEval_EvalCodeWithName at /home/user/.miniconda3/lib/libpython3.7m.so.1.0 (unknown line)
_PyFunction_FastCallDict at /home/user/.miniconda3/lib/libpython3.7m.so.1.0 (unknown line)
_PyEval_EvalFrameDefault at /home/user/.miniconda3/lib/libpython3.7m.so.1.0 (unknown line)
_PyEval_EvalCodeWithName at /home/user/.miniconda3/lib/libpython3.7m.so.1.0 (unknown line)
_PyFunction_FastCallDict at /home/user/.miniconda3/lib/libpython3.7m.so.1.0 (unknown line)
macro expansion at /home/user/.julia/packages/PyCall/ttONZ/src/exception.jl:81 [inlined]
__pycall! at /home/user/.julia/packages/PyCall/ttONZ/src/pyfncall.jl:44
_pycall! at /home/user/.julia/packages/PyCall/ttONZ/src/pyfncall.jl:29
#call#111 at /home/user/.julia/packages/PyCall/ttONZ/src/pyfncall.jl:11 [inlined]
PyObject at /home/user/.julia/packages/PyCall/ttONZ/src/pyfncall.jl:89
jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2197
do_call at /buildworker/worker/package_linux64/build/src/interpreter.c:323
eval_value at /buildworker/worker/package_linux64/build/src/interpreter.c:411
eval_body at /buildworker/worker/package_linux64/build/src/interpreter.c:635
jl_interpret_toplevel_thunk_callback at /buildworker/worker/package_linux64/build/src/interpreter.c:884
unknown function (ip: 0xfffffffffffffffe)
unknown function (ip: 0x7f67cc984f0f)
unknown function (ip: 0x2)
jl_interpret_toplevel_thunk at /buildworker/worker/package_linux64/build/src/interpreter.c:893
jl_toplevel_eval_flex at /buildworker/worker/package_linux64/build/src/toplevel.c:815
jl_toplevel_eval_flex at /buildworker/worker/package_linux64/build/src/toplevel.c:764
jl_toplevel_eval_in at /buildworker/worker/package_linux64/build/src/toplevel.c:844
eval at ./boot.jl:330
jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2191
exec_options at ./client.jl:271
_start at ./client.jl:464
jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2191
unknown function (ip: 0x40192d)
unknown function (ip: 0x401533)
__libc_start_main at /lib64/libc.so.6 (unknown line)
unknown function (ip: 0x4015d4)
Allocations: 11458870 (Pool: 11456484; Big: 2386); GC: 25
^C^C^C^C^C^C^CWARNING: Force throwing a SIGINT

signal (11): Segmentation fault
in expression starting at none:4
tupledealloc.lto_priv.484 at /home/user/.miniconda3/lib/libpython3.7m.so.1.0 (unknown line)
pydecref_ at /home/user/.julia/packages/PyCall/ttONZ/src/PyCall.jl:121
_pycall! at /home/user/.julia/packages/PyCall/ttONZ/src/pyfncall.jl:31
#call#111 at /home/user/.julia/packages/PyCall/ttONZ/src/pyfncall.jl:11 [inlined]
PyObject at /home/user/.julia/packages/PyCall/ttONZ/src/pyfncall.jl:89
jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2197
do_call at /buildworker/worker/package_linux64/build/src/interpreter.c:323
eval_value at /buildworker/worker/package_linux64/build/src/interpreter.c:411
eval_body at /buildworker/worker/package_linux64/build/src/interpreter.c:635
jl_interpret_toplevel_thunk_callback at /buildworker/worker/package_linux64/build/src/interpreter.c:884
unknown function (ip: 0xfffffffffffffffe)
unknown function (ip: 0x7f67cc984f0f)
unknown function (ip: 0x2)
jl_interpret_toplevel_thunk at /buildworker/worker/package_linux64/build/src/interpreter.c:893
jl_toplevel_eval_flex at /buildworker/worker/package_linux64/build/src/toplevel.c:815
jl_toplevel_eval_flex at /buildworker/worker/package_linux64/build/src/toplevel.c:764
jl_toplevel_eval_in at /buildworker/worker/package_linux64/build/src/toplevel.c:844
^Ceval at ./boot.jl:330
jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2191
exec_options at ./client.jl:271
_start at ./client.jl:464
jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2191
unknown function (ip: 0x40192d)^C
unknown function (ip: 0x401533)
__libc_start_main at /lib64/libc.so.6 (unknown line)
unknown function (ip: 0x4015d4)
Allocations: 8918253 (Pool: 8916828; Big: 1425); GC: 19
/bin/bash: line 1: 24040 Segmentation fault 

This seems related to #344 and #306 (where I recently remarked the issue). Using python-jl does not seem to make a difference.

[mattcbro]

I am also seeing segfaults using multiple threads in Julia. But I only see the segfaults on windows. Not sure if it's related. I get this error on windows:

Traceback (most recent call last):
  File "runmetrics.py", line 112, in <module>
    heartrate = Main.besthrates
  File "C:\Users\mattc\AppData\Roaming\Python\Python36\site-packages\julia\core.py", line 176, in __getattr__
    return self.__try_getattr(name)
  File "C:\Users\mattc\AppData\Roaming\Python\Python36\site-packages\julia\core.py", line 189, in __try_getattr
    if self._julia.isamodule(jl_fullname):
  File "C:\Users\mattc\AppData\Roaming\Python\Python36\site-packages\julia\core.py", line 660, in isamodule
    return self.eval("isa({}, Module)".format(julia_name))
  File "C:\Users\mattc\AppData\Roaming\Python\Python36\site-packages\julia\core.py", line 605, in eval
    ans = self._call(src)
  File "C:\Users\mattc\AppData\Roaming\Python\Python36\site-packages\julia\core.py", line 538, in _call
    self.check_exception(src)
  File "C:\Users\mattc\AppData\Roaming\Python\Python36\site-packages\julia\core.py", line 582, in check_exception
    res = self.api.jl_call2(self._convert, self._PyObject, exoc)
OSError: exception: access violation reading 0x0000023BA9CD1000

Please submit a bug report with steps to reproduce this fault, and any error messages that follow (in their entirety). Thanks.
Exception: EXCEPTION_ACCESS_VIOLATION at 0x77997465 -- PyUnicodeWriter_WriteASCIIString at C:\Program Files (x86)\Microsoft Visual Studio\Shared\Python36_64\python36.dll (unknown line)
in expression starting at none:0
PyUnicodeWriter_WriteASCIIString at C:\Program Files (x86)\Microsoft Visual Studio\Shared\Python36_64\python36.dll (unknown line)
Py_DecRef at C:\Program Files (x86)\Microsoft Visual Studio\Shared\Python36_64\python36.dll (unknown line)
pydecref_ at C:\Users\mattc\.julia\packages\PyCall\BcTLp\src\PyCall.jl:118
pydecref at C:\Users\mattc\.julia\packages\PyCall\BcTLp\src\PyCall.jl:123
jl_apply at /cygdrive/c/buildbot/worker/package_win64/build/src\julia.h:1690 [inlined]
run_finalizer at /cygdrive/c/buildbot/worker/package_win64/build/src\gc.c:277
jl_gc_run_finalizers_in_list at /cygdrive/c/buildbot/worker/package_win64/build/src\gc.c:365
run_finalizers at /cygdrive/c/buildbot/worker/package_win64/build/src\gc.c:391 [inlined]
run_finalizers at /cygdrive/c/buildbot/worker/package_win64/build/src\gc.c:370
jl_gc_collect at /cygdrive/c/buildbot/worker/package_win64/build/src\gc.c:3127
maybe_collect at /cygdrive/c/buildbot/worker/package_win64/build/src\gc.c:827 [inlined]
jl_gc_pool_alloc at /cygdrive/c/buildbot/worker/package_win64/build/src\gc.c:1142
jl_gc_alloc_ at /cygdrive/c/buildbot/worker/package_win64/build/src\julia_internal.h:277 [inlined]
_new_array_ at /cygdrive/c/buildbot/worker/package_win64/build/src\array.c:106 [inlined]
_new_array at /cygdrive/c/buildbot/worker/package_win64/build/src\array.c:162 [inlined]
jl_alloc_array_2d at /cygdrive/c/buildbot/worker/package_win64/build/src\array.c:440
Array at .\boot.jl:408 [inlined]
Array at .\boot.jl:416 [inlined]
Array at .\boot.jl:423 [inlined]
similar at .\abstractarray.jl:675 [inlined]
similar at .\abstractarray.jl:674 [inlined]
similar at .\broadcast.jl:197 [inlined]
similar at .\broadcast.jl:196 [inlined]
copy at .\broadcast.jl:862 [inlined]
materialize at .\broadcast.jl:837 [inlined]
broadcast_preserving_zero_d at .\broadcast.jl:826 [inlined]
+ at .\arraymath.jl:47
kronleftmul at C:\Users\mattc\projects\PulseLib\src\PulseAlgos.jl:280
#opsigsub#14 at C:\Users\mattc\projects\PulseLib\src\PulseAlgos.jl:238
opsigsub##kw at C:\Users\mattc\projects\PulseLib\src\PulseAlgos.jl:233
unknown function (ip: 0000023bcbb5270c)
#trigequal#17 at C:\Users\mattc\projects\PulseLib\src\PulseAlgos.jl:455
trigequal##kw at C:\Users\mattc\projects\PulseLib\src\PulseAlgos.jl:423
unknown function (ip: 0000023bcbb55a36)
macro expansion at C:\Users\mattc\projects\PulseLib\src\PulseAlgos.jl:494 [inlined]
#6442#threadsfor_fun at .\threadingconstructs.jl:81
unknown function (ip: 0000023bcbb096f8)
jl_apply at /cygdrive/c/buildbot/worker/package_win64/build/src\julia.h:1690 [inlined]
do_apply at /cygdrive/c/buildbot/worker/package_win64/build/src\builtins.c:655
jl_f__apply at /cygdrive/c/buildbot/worker/package_win64/build/src\builtins.c:669 [inlined]
jl_f__apply_latest at /cygdrive/c/buildbot/worker/package_win64/build/src\builtins.c:705
#invokelatest#1 at .\essentials.jl:710 [inlined]
invokelatest at .\essentials.jl:709 [inlined]
macro expansion at .\threadingconstructs.jl:86 [inlined]
#findtrigperiod#18 at C:\Users\mattc\projects\PulseLib\src\PulseAlgos.jl:490
findtrigperiod##kw at C:\Users\mattc\projects\PulseLib\src\PulseAlgos.jl:479
unknown function (ip: 0000023bcbb085b3)
#slidetrig#22 at C:\Users\mattc\projects\PulseLib\src\PulseAlgos.jl:616
slidetrig##kw at C:\Users\mattc\projects\PulseLib\src\PulseAlgos.jl:566 [inlined]
#55 at .\threadingconstructs.jl:169
unknown function (ip: 0000023bd8b97d73)

I am having trouble producing a simple example . My code spawns a complex task with multiple threads. The code crashes when I try to call a julia function prior to the thread's completion. It's a memory error, so perhaps there is some non-reentrant code on the windows side of the implementation of Pyjulia.

The code works flawlessly in linux however.

[andrewkhardy]

Coming here to say, I also have a bug where this crashes on a linux server. By deleting the variables I called in the Julia function, I could reach a larger number of iterations / threads, but it still crashes.

signal (11): Segmentation fault
in expression starting at none:0
PyObject_RichCompare at /dev/shm/rzon/cooper-install.xbAuSE/Python-3.8.5/Objects/object.c:772
PyObject_RichCompareBool at /dev/shm/rzon/cooper-install.xbAuSE/Python-3.8.5/Objects/object.c:796
lookdict at /dev/shm/rzon/cooper-install.xbAuSE/Python-3.8.5/Objects/dictobject.c:782
PyDict_GetItem at /dev/shm/rzon/cooper-install.xbAuSE/Python-3.8.5/Objects/dictobject.c:1393
_buffer_clear_info at /dev/shm/rzon/cooper-install.xbAuSE/numpy-1.19.2/numpy/core/src/multiarray/buffer.c:665 [inlined]
_dealloc_cached_buffer_info at /dev/shm/rzon/cooper-install.xbAuSE/numpy-1.19.2/numpy/core/src/multiarray/buffer.c:881
array_dealloc at /dev/shm/rzon/cooper-install.xbAuSE/numpy-1.19.2/numpy/core/src/multiarray/arrayobject.c:475
pydecref_ at /home/a/aparamek/andykh/.julia/packages/PyCall/BD546/src/PyCall.jl:118 [inlined]
pydecref at /home/a/aparamek/andykh/.julia/packages/PyCall/BD546/src/PyCall.jl:123
_jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2237 [inlined]
jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2419
jl_apply at /buildworker/worker/package_linux64/build/src/julia.h:1703 [inlined]
run_finalizer at /buildworker/worker/package_linux64/build/src/gc.c:278
jl_gc_run_finalizers_in_list at /buildworker/worker/package_linux64/build/src/gc.c:365
run_finalizers at /buildworker/worker/package_linux64/build/src/gc.c:394 [inlined]
jl_gc_run_all_finalizers at /buildworker/worker/package_linux64/build/src/gc.c:480
jl_atexit_hook at /buildworker/worker/package_linux64/build/src/init.c:240
jl_exit at /buildworker/worker/package_linux64/build/src/jl_uv.c:632
jl_exit_thread0_cb at /buildworker/worker/package_linux64/build/src/signals-unix.c:353
unknown function (ip: (nil))
Allocations: 76865391 (Pool: 76849326; Big: 16065); GC: 114

signal (15): Terminated
in expression starting at none:0
mul_fast at ./fastmath.jl:167 [inlined]
mul_fast at ./fastmath.jl:219 [inlined]
𝒜𝒸𝓉! at /home/a/aparamek/andykh/.julia/packages/Tullio/Px7kN/src/macro.jl:1209 [inlined]
𝒜𝒸𝓉! at /home/a/aparamek/andykh/.julia/packages/Tullio/Px7kN/src/macro.jl:1039 [inlined]
threader at /home/a/aparamek/andykh/.julia/packages/Tullio/Px7kN/src/threads.jl:48 [inlined]
macro expansion at /home/a/aparamek/andykh/.julia/packages/Tullio/Px7kN/src/macro.jl:1002 [inlined]
greenfunction at /gpfs/fs1/home/a/aparamek/andykh/SYK-collaborate/01_Code/fastsum.jl:4
unknown function (ip: 0x2b7396667a37)
_jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2237 [inlined]
jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2419
jl_apply at /buildworker/worker/package_linux64/build/src/julia.h:1703 [inlined]
jl_f__call_latest at /buildworker/worker/package_linux64/build/src/builtins.c:714
_jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2237 [inlined]
jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2419
jl_apply at /buildworker/worker/package_linux64/build/src/julia.h:1703 [inlined]
do_apply at /buildworker/worker/package_linux64/build/src/builtins.c:670
#invokelatest#2 at ./essentials.jl:708
_jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2237 [inlined]
jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2419
jl_apply at /buildworker/worker/package_linux64/build/src/julia.h:1703 [inlined]
do_apply at /buildworker/worker/package_linux64/build/src/builtins.c:670
invokelatest at ./essentials.jl:706
_jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2237 [inlined]
jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2419
jl_apply at /buildworker/worker/package_linux64/build/src/julia.h:1703 [inlined]
do_apply at /buildworker/worker/package_linux64/build/src/builtins.c:670
_pyjlwrap_call at /home/a/aparamek/andykh/.julia/packages/PyCall/BD546/src/callback.jl:28
unknown function (ip: 0x2b738bba3a15)
_jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2237 [inlined]
jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2419
pyjlwrap_call at /home/a/aparamek/andykh/.julia/packages/PyCall/BD546/src/callback.jl:44
unknown function (ip: 0x2b738bb9cd7e)
_PyObject_MakeTpCall at /dev/shm/rzon/cooper-install.xbAuSE/Python-3.8.5/Objects/call.c:159
_PyObject_Vectorcall at /dev/shm/rzon/cooper-install.xbAuSE/Python-3.8.5/./Include/cpython/abstract.h:125 [inlined]
call_function at /dev/shm/rzon/cooper-install.xbAuSE/Python-3.8.5/Python/ceval.c:4963 [inlined]
_PyEval_EvalFrameDefault at /dev/shm/rzon/cooper-install.xbAuSE/Python-3.8.5/Python/ceval.c:3469
function_code_fastcall at /dev/shm/rzon/cooper-install.xbAuSE/Python-3.8.5/Objects/call.c:283 [inlined]
_PyFunction_Vectorcall at /dev/shm/rzon/cooper-install.xbAuSE/Python-3.8.5/Objects/call.c:410
_PyObject_Vectorcall at /dev/shm/rzon/cooper-install.xbAuSE/Python-3.8.5/./Include/cpython/abstract.h:127 [inlined]
call_function at /dev/shm/rzon/cooper-install.xbAuSE/Python-3.8.5/Python/ceval.c:4963 [inlined]
_PyEval_EvalFrameDefault at /dev/shm/rzon/cooper-install.xbAuSE/Python-3.8.5/Python/ceval.c:3500
function_code_fastcall at /dev/shm/rzon/cooper-install.xbAuSE/Python-3.8.5/Objects/call.c:283 [inlined]
_PyFunction_Vectorcall at /dev/shm/rzon/cooper-install.xbAuSE/Python-3.8.5/Objects/call.c:410
PyVectorcall_Call at /dev/shm/rzon/cooper-install.xbAuSE/Python-3.8.5/Objects/call.c:199
do_call_core at /dev/shm/rzon/cooper-install.xbAuSE/Python-3.8.5/Python/ceval.c:5010 [inlined]
_PyEval_EvalFrameDefault at /dev/shm/rzon/cooper-install.xbAuSE/Python-3.8.5/Python/ceval.c:3559
_PyEval_EvalCodeWithName at /dev/shm/rzon/cooper-install.xbAuSE/Python-3.8.5/Python/ceval.c:4298
_PyFunction_Vectorcall at /dev/shm/rzon/cooper-install.xbAuSE/Python-3.8.5/Objects/call.c:435
_PyObject_Vectorcall at /dev/shm/rzon/cooper-install.xbAuSE/Python-3.8.5/./Include/cpython/abstract.h:127 [inlined]
call_function at /dev/shm/rzon/cooper-install.xbAuSE/Python-3.8.5/Python/ceval.c:4963 [inlined]
_PyEval_EvalFrameDefault at /dev/shm/rzon/cooper-install.xbAuSE/Python-3.8.5/Python/ceval.c:3500
function_code_fastcall at /dev/shm/rzon/cooper-install.xbAuSE/Python-3.8.5/Objects/call.c:283 [inlined]
_PyFunction_Vectorcall at /dev/shm/rzon/cooper-install.xbAuSE/Python-3.8.5/Objects/call.c:410
_PyObject_FastCallDict at /dev/shm/rzon/cooper-install.xbAuSE/Python-3.8.5/Objects/call.c:104
_PyObject_Call_Prepend at /dev/shm/rzon/cooper-install.xbAuSE/Python-3.8.5/Objects/call.c:887
slot_tp_call at /dev/shm/rzon/cooper-install.xbAuSE/Python-3.8.5/Objects/typeobject.c:6553
PyObject_Call at /dev/shm/rzon/cooper-install.xbAuSE/Python-3.8.5/Objects/call.c:245
do_call_core at /dev/shm/rzon/cooper-install.xbAuSE/Python-3.8.5/Python/ceval.c:5010 [inlined]
_PyEval_EvalFrameDefault at /dev/shm/rzon/cooper-install.xbAuSE/Python-3.8.5/Python/ceval.c:3559
_PyEval_EvalCodeWithName at /dev/shm/rzon/cooper-install.xbAuSE/Python-3.8.5/Python/ceval.c:4298
_PyFunction_Vectorcall at /dev/shm/rzon/cooper-install.xbAuSE/Python-3.8.5/Objects/call.c:435
_PyObject_FastCallDict at /dev/shm/rzon/cooper-install.xbAuSE/Python-3.8.5/Objects/call.c:104
_PyObject_Call_Prepend at /dev/shm/rzon/cooper-install.xbAuSE/Python-3.8.5/Objects/call.c:887
slot_tp_call at /dev/shm/rzon/cooper-install.xbAuSE/Python-3.8.5/Objects/typeobject.c:6553
PyObject_Call at /dev/shm/rzon/cooper-install.xbAuSE/Python-3.8.5/Objects/call.c:245
do_call_core at /dev/shm/rzon/cooper-install.xbAuSE/Python-3.8.5/Python/ceval.c:5010 [inlined]
_PyEval_EvalFrameDefault at /dev/shm/rzon/cooper-install.xbAuSE/Python-3.8.5/Python/ceval.c:3559
function_code_fastcall at /dev/shm/rzon/cooper-install.xbAuSE/Python-3.8.5/Objects/call.c:283 [inlined]
_PyFunction_Vectorcall at /dev/shm/rzon/cooper-install.xbAuSE/Python-3.8.5/Objects/call.c:410 [inlined]
_PyObject_FastCallDict at /dev/shm/rzon/cooper-install.xbAuSE/Python-3.8.5/Objects/call.c:96
_PyObject_Call_Prepend at /dev/shm/rzon/cooper-install.xbAuSE/Python-3.8.5/Objects/call.c:887
slot_tp_call at /dev/shm/rzon/cooper-install.xbAuSE/Python-3.8.5/Objects/typeobject.c:6553
_PyObject_MakeTpCall at /dev/shm/rzon/cooper-install.xbAuSE/Python-3.8.5/Objects/call.c:159
_PyObject_Vectorcall at /dev/shm/rzon/cooper-install.xbAuSE/Python-3.8.5/./Include/cpython/abstract.h:125 [inlined]
call_function at /dev/shm/rzon/cooper-install.xbAuSE/Python-3.8.5/Python/ceval.c:4963 [inlined]
_PyEval_EvalFrameDefault at /dev/shm/rzon/cooper-install.xbAuSE/Python-3.8.5/Python/ceval.c:3500
function_code_fastcall at /dev/shm/rzon/cooper-install.xbAuSE/Python-3.8.5/Objects/call.c:283 [inlined]
_PyFunction_Vectorcall at /dev/shm/rzon/cooper-install.xbAuSE/Python-3.8.5/Objects/call.c:410
PyVectorcall_Call at /dev/shm/rzon/cooper-install.xbAuSE/Python-3.8.5/Objects/call.c:199
do_call_core at /dev/shm/rzon/cooper-install.xbAuSE/Python-3.8.5/Python/ceval.c:5010 [inlined]
_PyEval_EvalFrameDefault at /dev/shm/rzon/cooper-install.xbAuSE/Python-3.8.5/Python/ceval.c:3559
function_code_fastcall at /dev/shm/rzon/cooper-install.xbAuSE/Python-3.8.5/Objects/call.c:283 [inlined]
_PyFunction_Vectorcall at /dev/shm/rzon/cooper-install.xbAuSE/Python-3.8.5/Objects/call.c:410
_PyObject_Vectorcall at /dev/shm/rzon/cooper-install.xbAuSE/Python-3.8.5/./Include/cpython/abstract.h:127 [inlined]
call_function at /dev/shm/rzon/cooper-install.xbAuSE/Python-3.8.5/Python/ceval.c:4963 [inlined]
_PyEval_EvalFrameDefault at /dev/shm/rzon/cooper-install.xbAuSE/Python-3.8.5/Python/ceval.c:3486
_PyEval_EvalCodeWithName at /dev/shm/rzon/cooper-install.xbAuSE/Python-3.8.5/Python/ceval.c:4298
_PyFunction_Vectorcall at /dev/shm/rzon/cooper-install.xbAuSE/Python-3.8.5/Objects/call.c:435
_PyObject_Vectorcall at /dev/shm/rzon/cooper-install.xbAuSE/Python-3.8.5/./Include/cpython/abstract.h:127 [inlined]
call_function at /dev/shm/rzon/cooper-install.xbAuSE/Python-3.8.5/Python/ceval.c:4963 [inlined]
_PyEval_EvalFrameDefault at /dev/shm/rzon/cooper-install.xbAuSE/Python-3.8.5/Python/ceval.c:3486
_PyEval_EvalCodeWithName at /dev/shm/rzon/cooper-install.xbAuSE/Python-3.8.5/Python/ceval.c:4298
PyEval_EvalCodeEx at /dev/shm/rzon/cooper-install.xbAuSE/Python-3.8.5/Python/ceval.c:4327
PyEval_EvalCode at /dev/shm/rzon/cooper-install.xbAuSE/Python-3.8.5/Python/ceval.c:718
builtin_exec_impl at /dev/shm/rzon/cooper-install.xbAuSE/Python-3.8.5/Python/bltinmodule.c:1033 [inlined]
builtin_exec at /dev/shm/rzon/cooper-install.xbAuSE/Python-3.8.5/Python/clinic/bltinmodule.c.h:396
cfunction_vectorcall_FASTCALL at /dev/shm/rzon/cooper-install.xbAuSE/Python-3.8.5/Objects/methodobject.c:422
_PyObject_Vectorcall at /dev/shm/rzon/cooper-install.xbAuSE/Python-3.8.5/./Include/cpython/abstract.h:127 [inlined]
call_function at /dev/shm/rzon/cooper-install.xbAuSE/Python-3.8.5/Python/ceval.c:4963 [inlined]
_PyEval_EvalFrameDefault at /dev/shm/rzon/cooper-install.xbAuSE/Python-3.8.5/Python/ceval.c:3500
_PyEval_EvalCodeWithName at /dev/shm/rzon/cooper-install.xbAuSE/Python-3.8.5/Python/ceval.c:4298
_PyFunction_Vectorcall at /dev/shm/rzon/cooper-install.xbAuSE/Python-3.8.5/Objects/call.c:435
_PyObject_Vectorcall at /dev/shm/rzon/cooper-install.xbAuSE/Python-3.8.5/./Include/cpython/abstract.h:127 [inlined]
call_function at /dev/shm/rzon/cooper-install.xbAuSE/Python-3.8.5/Python/ceval.c:4963 [inlined]
_PyEval_EvalFrameDefault at /dev/shm/rzon/cooper-install.xbAuSE/Python-3.8.5/Python/ceval.c:3500
_PyEval_EvalCodeWithName at /dev/shm/rzon/cooper-install.xbAuSE/Python-3.8.5/Python/ceval.c:4298
_PyFunction_Vectorcall at /dev/shm/rzon/cooper-install.xbAuSE/Python-3.8.5/Objects/call.c:435
PyVectorcall_Call at /dev/shm/rzon/cooper-install.xbAuSE/Python-3.8.5/Objects/call.c:199
pymain_run_module at /dev/shm/rzon/cooper-install.xbAuSE/Python-3.8.5/Modules/main.c:302
pymain_run_python at /dev/shm/rzon/cooper-install.xbAuSE/Python-3.8.5/Modules/main.c:600 [inlined]
Py_RunMain at /dev/shm/rzon/cooper-install.xbAuSE/Python-3.8.5/Modules/main.c:685
Py_BytesMain at /dev/shm/rzon/cooper-install.xbAuSE/Python-3.8.5/Modules/main.c:739
__libc_start_main at /lib64/libc.so.6 (unknown line)
_start at /gpfs/fs1/home/a/aparamek/andykh/syknematic/bin/python (unknown line)
unknown function (ip: (nil))
Allocations: 76861098 (Pool: 76845028; Big: 16070); GC: 116

signal (11): Segmentation fault
in expression starting at none:0
PyObject_RichCompare at /dev/shm/rzon/cooper-install.xbAuSE/Python-3.8.5/Objects/object.c:772
PyObject_RichCompareBool at /dev/shm/rzon/cooper-install.xbAuSE/Python-3.8.5/Objects/object.c:796
lookdict at /dev/shm/rzon/cooper-install.xbAuSE/Python-3.8.5/Objects/dictobject.c:782
PyDict_GetItem at /dev/shm/rzon/cooper-install.xbAuSE/Python-3.8.5/Objects/dictobject.c:1393
_buffer_clear_info at /dev/shm/rzon/cooper-install.xbAuSE/numpy-1.19.2/numpy/core/src/multiarray/buffer.c:665 [inlined]
_dealloc_cached_buffer_info at /dev/shm/rzon/cooper-install.xbAuSE/numpy-1.19.2/numpy/core/src/multiarray/buffer.c:881
array_dealloc at /dev/shm/rzon/cooper-install.xbAuSE/numpy-1.19.2/numpy/core/src/multiarray/arrayobject.c:475
pydecref_ at /home/a/aparamek/andykh/.julia/packages/PyCall/BD546/src/PyCall.jl:118 [inlined]
pydecref at /home/a/aparamek/andykh/.julia/packages/PyCall/BD546/src/PyCall.jl:123
_jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2237 [inlined]
jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2419
jl_apply at /buildworker/worker/package_linux64/build/src/julia.h:1703 [inlined]
run_finalizer at /buildworker/worker/package_linux64/build/src/gc.c:278
jl_gc_run_finalizers_in_list at /buildworker/worker/package_linux64/build/src/gc.c:365
run_finalizers at /buildworker/worker/package_linux64/build/src/gc.c:394 [inlined]
jl_gc_run_all_finalizers at /buildworker/worker/package_linux64/build/src/gc.c:480
jl_atexit_hook at /buildworker/worker/package_linux64/build/src/init.c:240
jl_exit at /buildworker/worker/package_linux64/build/src/jl_uv.c:632
jl_exit_thread0_cb at /buildworker/worker/package_linux64/build/src/signals-unix.c:353
unknown function (ip: (nil))
Allocations: 76881097 (Pool: 76864955; Big: 16142); GC: 115
Traceback (most recent call last):
  File "parallelsyk.py", line 34, in <module>
    observables  = Parallel(n_jobs = 20, verbose = 10)(delayed(thermodynamics)(intrline[i], hoppline[i],param,inputfile,outlocation) \
  File "/scinet/niagara/software/2019b/opt/base/python/3.8.5/lib/python3.8/site-packages/joblib/parallel.py", line 1042, in __call__
    self.retrieve()
  File "/scinet/niagara/software/2019b/opt/base/python/3.8.5/lib/python3.8/site-packages/joblib/parallel.py", line 921, in retrieve
    self._output.extend(job.get(timeout=self.timeout))
  File "/scinet/niagara/software/2019b/opt/base/python/3.8.5/lib/python3.8/site-packages/joblib/_parallel_backends.py", line 542, in wrap_future_result
    return future.result(timeout=timeout)
  File "/scinet/niagara/software/2019b/opt/base/python/3.8.5/lib/python3.8/concurrent/futures/_base.py", line 439, in result
    return self.__get_result()
  File "/scinet/niagara/software/2019b/opt/base/python/3.8.5/lib/python3.8/concurrent/futures/_base.py", line 388, in __get_result
    raise self._exception
joblib.externals.loky.process_executor.TerminatedWorkerError: A worker process managed by the executor was unexpectedly terminated. This could be caused by a segmentation fault while calling the function or by an excessive memory usage causing the Operating System to kill the worker.

[sibyjackgrove]

I believe this is related to #477 and SciML/diffeqpy#98

@mattcbro @kaandocal Did you find any solution for this?

[kaandocal]

Sadly not, I have stopped using pyjulia since...

[davidavdav]

This is a bummer --- I get a similar error ReadOnlyMemoryError() in a similar setup, using PyJulia in a multiprocessing.Process instance.

Has anyone found a way around this, or should I use one of the alternatives:

• just recode everything in Python again
• setup a completely separate Julia process and communicate using some ad hoc RPC protocol

[davidavdav]

Small update: when I use context = multiprocessing.get_context("spawn") or "forkserver", I am able to run a python-jl process without crashing.

In a basic application I can run the python-jl process as subprocess, and communicate through the context.Queue() objects, but in the full application with a complex subprocessing/queue arrangement I still run into trouble, there might indeed be a dependence on the number of subprocesses or somewthing like that.

[Mark2000]

@davidavdav You seem to be the only person online who has figured this out. Spent my afternoon figuring out why my multiprocessed Python code using pyjulia was working on my Mac but not on a linux server; forcing the linux machine to use "spawn" (default on Mac, non-default on linux) made things work. I think it would be good to update the limitations page in the documentation to reflect this workaround, however unsure if it has hidden issues that make this unkosher.

[mkitti]

At least one aspect of this seems actionable, even if it is only documentation. Can someone send me a pull request? Tag me.

[davidavdav]

Well, even in pure python the multiprocessing communication is difficult (but that is off topic). In combination with asyncio the Queues become quite complex, every kind of multiprocessing and asyncio have their own Queue object, and making them work together is not trivial IMHO. In a complicated setup (several GPU processes, multiple CPU processes) messages travel around fine, but at a certain level of complexion error messages completely vanish from all forms of logging / stderr, which make debugging hard. Perhaps a pure Julia implementation would resolve some of these difficulties, but we are far from that.