Crash when trying to use Julia from Dask multiprocessing

[lungben]

Hello,

PyJulia crashes when I try to callJulia from inside a Python process using Dask multiprocessing (dask.distributed.Client) throwing the following exceptions:

Exception: OSError('exception: access violation reading 0x00000000000019A8')
Exception: AttributeError("function 'was_initialized' not found")

Calling Julia from Python's built-in multiprocessing works however fine.

I am using Windows 10 (64 bit), Python 3.7, Dask 2.9 and Julia 1.3.

Test Code:

from julia import Julia
import dask.distributed
import concurrent.futures


def jl_calc(id):
    func_text = """
        function myfunc(n:: Int64)
            z:: Int64=1
            for i=1:n
                z += 2i^2
            end
            z
        end
        """
    jl = Julia()
    test_func = jl.eval(func_text)
    res = test_func(id+1)
    return res


def submit_tasks(executor):
    futures = []
    for i in range(10):
        futures.append(executor.submit(jl_calc, i))
    results = [future.result() for future in futures]
    print(results)


if __name__ == '__main__':
    print('calling Julia from Pythons bulit-in Multiprocessing')
    executor_multiprocessing = concurrent.futures.ProcessPoolExecutor()
    try:
        submit_tasks(executor_multiprocessing)
    finally:
        executor_multiprocessing.shutdown()
    print('calling Julia from Pythons bulit-in Multiprocessing successful')

    print('calling Julia from Dask Client')
    executor_dask = dask.distributed.Client()
    try:
        submit_tasks(executor_dask)
    finally:
        executor_dask.shutdown()
    print('calling Julia from Dask Client successful')

System setup and stack trace is given in the attachment.
Dask_Julia_error.txt

Edit:

The same issue has been raised at Dask, because I do not know in which component the issue is:

dask/dask#5732

Best Regards
Benjamin

[sibyjackgrove]

I think this is related to #477 and SciML/diffeqpy#98