Skip to content

Support virtualenv/venv without separate cache #190

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 19 commits into from
Sep 5, 2018
Merged

Support virtualenv/venv without separate cache #190

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
19 commits
Select commit Hold shift + click to select a range
5290c8e
Avoid the cryptic error about fake-julia/../lib/julia/sys.so
tkf Aug 22, 2018
d49d819
More robust pre-compilation cache handling for Julia 0.6
tkf Aug 22, 2018
a748dea
Run doctest via pytest
tkf Aug 22, 2018
54d1116
Don't install pytest 3.7.3 to avoid doctest failures in macOS
tkf Aug 29, 2018
1825d34
Make PyJulia usable in virtual environments
tkf Aug 29, 2018
1da6243
Python < 3.7 does not support namedtuple(..., defaults=...)
tkf Aug 29, 2018
252cd55
Print debug message from is_compatible_exe
tkf Aug 29, 2018
2ab64e7
More permissive test for finding_libpython
tkf Aug 29, 2018
1f47404
Run find_libpython in tox
tkf Aug 29, 2018
9adb074
Use absolute_import in find_libpython.py
tkf Aug 29, 2018
6e91cf9
Print jlinfo.libpython
tkf Aug 29, 2018
435ccf4
Find the linked libpython using dladdr
tkf Aug 29, 2018
25631af
Manually set SHLIB_SUFFIX in non-Linux platforms
tkf Aug 29, 2018
e98c265
Use sysconfig.get_config_var("VERSION")
tkf Aug 29, 2018
f09ae42
Directly compare executable paths
tkf Aug 29, 2018
96429c6
Print PyCall/deps/build.log
tkf Aug 29, 2018
6982ae8
Test against 1.0 instead of nightly in AppVeyor
tkf Aug 29, 2018
fe0cd97
Do not use platform.system()
tkf Aug 29, 2018
cdbefb1
More manual suffix handling for macOS
tkf Aug 29, 2018
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 14 additions & 4 deletions appveyor.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,22 +13,32 @@ environment:
BATDIR: ci\appveyor\win32
CROSS_VERSION: 1

# 32 julia latest Python-35
- JULIA_URL: "https://julialangnightlies-s3.julialang.org/bin/winnt/x86/julia-latest-win32.exe"
# 32 julia-1.0 Python-35
- JULIA_URL: "https://julialang-s3.julialang.org/bin/winnt/x86/1.0/julia-1.0-latest-win32.exe"
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Testing with Julia nightly yields some bizarre failures (see below). The same code works with Julia 1.0. I suggest to test with stable release.

From Python 3.5:

ErrorException("ccall: could not find function PyString_AsStringAndSize in library C:\projects\pyjulia\.tox\py\Scripts\python35")
--- https://ci.appveyor.com/project/Keno/pyjulia/build/1.0.175/job/wwr318to7jr1b5h6#L433

From Python 2.7:

    def test_call_julia_function_with_python_args(self):
        self.assertEqual(['A', 'B', 'C'],
                         list(julia.map(julia.uppercase,
>                                       array.array('u', [u'a', u'b', u'c']))))
E       RuntimeError: Julia exception: MethodError(uppercase, (PyObject u'a',), 0x000061ca)

--- https://ci.appveyor.com/project/Keno/pyjulia/build/1.0.175/job/wwr318to7jr1b5h6#L547

PYTHONDIR: "C:\\Python35"
BATDIR: ci\appveyor\win32

# 32 julia latest Python-35
# - JULIA_URL: "https://julialangnightlies-s3.julialang.org/bin/winnt/x86/julia-latest-win32.exe"
# PYTHONDIR: "C:\\Python35"
# BATDIR: ci\appveyor\win32

# 64 julia-0.6 Python-35
- JULIA_URL: "https://julialang-s3.julialang.org/bin/winnt/x64/0.6/julia-0.6-latest-win64.exe"
PYTHONDIR: "C:\\Python35-x64"
BATDIR: ci\appveyor\win64
CROSS_VERSION: 1

# 64 julia latest Python-35
- JULIA_URL: "https://julialangnightlies-s3.julialang.org/bin/winnt/x64/julia-latest-win64.exe"
# 64 julia-1.0 Python-35
- JULIA_URL: "https://julialang-s3.julialang.org/bin/winnt/x64/1.0/julia-1.0-latest-win64.exe"
PYTHONDIR: "C:\\Python35-x64"
BATDIR: ci\appveyor\win64

# 64 julia latest Python-35
# - JULIA_URL: "https://julialangnightlies-s3.julialang.org/bin/winnt/x64/julia-latest-win64.exe"
# PYTHONDIR: "C:\\Python35-x64"
# BATDIR: ci\appveyor\win64

matrix:
allow_failures:
- JULIA_URL: "https://julialangnightlies-s3.julialang.org/bin/winnt/x86/julia-latest-win32.exe"
Expand Down
106 changes: 87 additions & 19 deletions julia/core.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -32,6 +32,8 @@
# this is python 3.3 specific
from types import ModuleType, FunctionType

from .find_libpython import find_libpython, normalize_path

#-----------------------------------------------------------------------------
# Classes and funtions
#-----------------------------------------------------------------------------
Expand All @@ -42,6 +44,12 @@ def iteritems(d): return iter(d.items())
else:
iteritems = dict.iteritems


# As setting up Julia modifies os.environ, we need to cache it for
# launching subprocesses later in the original environment.
_enviorn = os.environ.copy()


class JuliaError(Exception):
pass

Expand Down Expand Up @@ -254,7 +262,9 @@ def determine_if_statically_linked():

JuliaInfo = namedtuple(
'JuliaInfo',
['JULIA_HOME', 'libjulia_path', 'image_file', 'pyprogramname'])
['JULIA_HOME', 'libjulia_path', 'image_file',
# Variables in PyCall/deps/deps.jl:
'pyprogramname', 'libpython'])


def juliainfo(runtime='julia'):
Expand All @@ -277,24 +287,61 @@ def juliainfo(runtime='julia'):
if PyCall_depsfile !== nothing && isfile(PyCall_depsfile)
include(PyCall_depsfile)
println(pyprogramname)
println(libpython)
end
"""])
"""],
# Use the original environment variables to avoid a cryptic
# error "fake-julia/../lib/julia/sys.so: cannot open shared
# object file: No such file or directory":
env=_enviorn)
args = output.decode("utf-8").rstrip().split("\n")
if len(args) == 3:
args.append(None) # no pyprogramname set
args.extend([None] * (len(JuliaInfo._fields) - len(args)))
return JuliaInfo(*args)


def is_same_path(a, b):
a = os.path.normpath(os.path.normcase(a))
b = os.path.normpath(os.path.normcase(b))
a = os.path.realpath(os.path.normcase(a))
b = os.path.realpath(os.path.normcase(b))
return a == b


def is_different_exe(pyprogramname, sys_executable):
if pyprogramname is None:
def is_compatible_exe(jlinfo, _debug=lambda *_: None):
"""
Determine if Python used by PyCall.jl is compatible with this Python.

Current Python executable is considered compatible if it is dynamically
linked to libpython (usually the case in macOS and Windows) and
both of them are using identical libpython. If this function returns
`True`, PyJulia use the same precompilation cache of PyCall.jl used by
Julia itself.

Parameters
----------
jlinfo : JuliaInfo
A `JuliaInfo` object returned by `juliainfo` function.
"""
_debug("jlinfo.libpython =", jlinfo.libpython)
if jlinfo.libpython is None:
_debug("libpython cannot be read from PyCall/deps/deps.jl")
return False

if determine_if_statically_linked():
_debug(sys.executable, "is statically linked.")
return False

# Note that the following check is OK since statically linked case
# is already excluded.
if is_same_path(jlinfo.pyprogramname, sys.executable):
# In macOS and Windows, find_libpython does not work as good
# as in Linux. We add this shortcut so that PyJulia can work
# in those environments.
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I needed this shortcut to make it work since find_libpython.py and PyCall/deps/build.jl sometimes disagree:

For example, in macOS, PyCall/deps/build.jl finds:

L957 jlinfo.libpython = /usr/local/opt/python/Frameworks/Python.framework/Versions/3.7/lib/libpython3.7m

but find_libpython.py finds:

L958 py_libpython = /usr/local/Cellar/python/3.7.0/Frameworks/Python.framework/Versions/3.7/Python

Another example in Windows: PyCall/deps/build.jl cannot find a fullpath:

L517 jlinfo.libpython = python27

but find_libpython.py "finds" it:

L518 py_libpython = C:\Windows\system32\python27.dl

(But it looks like the same path was found in win32 and win64. Ultimately, we need someone to write a function that uses dladdr equivalent in Windows. It should be possible by just extending: https://stackoverflow.com/a/16659821)

I think it's better to use the same script to discover libpython in PyCall.jl and PyJulia. We can then avoid a hack like this. Since find_libpython.py can call libdl.dladdr (though it works only if libpython is dynamically linked), I think it's better approach.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In the cases where they disagree, which is correct? Should deps/build.jl be fixed?

Copy link
Member Author

@tkf tkf Aug 29, 2018 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't have access to macOS so I'm not sure. But PyJulia printed

jlinfo.libpython = /usr/local/opt/python/Frameworks/Python.framework/Versions/3.7/lib/libpython3.7m
py_libpython = /usr/local/Cellar/python/3.7.0/Frameworks/Python.framework/Versions/3.7/Python
jl_libpython = None

where jlinfo.libpython is libpython saved in deps/deps.jl and jl_libpython is the path I tried to resolved by

pyjulia/julia/find_libpython.py

Lines 176 to 182 in 6982ae8

if not os.path.isabs(path):
return None
if os.path.exists(path):
return os.path.realpath(path)
if os.path.exists(path + suffix):
return os.path.realpath(path + suffix)
return None

So jl_libpython = None means that libpython3.7m.dylib did not exist (I set suffix = ".dylib" in macOS). Maybe we should try libpython3.7m.so in macOS too?

In the case of Windows, deps/build.jl just gave up (jlinfo.libpython = python27) whereas ctypes.util.find_library finds C:\Windows\system32\python27.dl.

Copy link
Member Author

@tkf tkf Aug 29, 2018 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm trying it: 1d599be

return True
return not is_same_path(pyprogramname, sys_executable)

py_libpython = find_libpython()
jl_libpython = normalize_path(jlinfo.libpython)
_debug("py_libpython =", py_libpython)
_debug("jl_libpython =", jl_libpython)
return py_libpython == jl_libpython


_julia_runtime = [False]
Expand Down Expand Up @@ -349,11 +396,10 @@ def __init__(self, init_julia=True, jl_runtime_path=None, jl_init_path=None,
runtime = jl_runtime_path
else:
runtime = 'julia'
JULIA_HOME, libjulia_path, image_file, depsjlexe = juliainfo(runtime)
jlinfo = juliainfo(runtime)
JULIA_HOME, libjulia_path, image_file, depsjlexe = jlinfo[:4]
self._debug("pyprogramname =", depsjlexe)
self._debug("sys.executable =", sys.executable)
exe_differs = is_different_exe(depsjlexe, sys.executable)
self._debug("exe_differs =", exe_differs)
self._debug("JULIA_HOME = %s, libjulia_path = %s" % (JULIA_HOME, libjulia_path))
if not os.path.exists(libjulia_path):
raise JuliaError("Julia library (\"libjulia\") not found! {}".format(libjulia_path))
Expand All @@ -371,7 +417,8 @@ def __init__(self, init_julia=True, jl_runtime_path=None, jl_init_path=None,
else:
jl_init_path = JULIA_HOME.encode("utf-8") # initialize with JULIA_HOME

use_separate_cache = exe_differs or determine_if_statically_linked()
use_separate_cache = not is_compatible_exe(jlinfo, _debug=self._debug)
self._debug("use_separate_cache =", use_separate_cache)
if use_separate_cache:
PYCALL_JULIA_HOME = os.path.join(
os.path.dirname(os.path.realpath(__file__)),"fake-julia").replace("\\","\\\\")
Expand Down Expand Up @@ -441,16 +488,37 @@ def __init__(self, init_julia=True, jl_runtime_path=None, jl_init_path=None,
# configuration and so do any packages that depend on it.
self._call(u"unshift!(Base.LOAD_CACHE_PATH, abspath(Pkg.Dir._pkgroot()," +
"\"lib\", \"pyjulia%s-v$(VERSION.major).$(VERSION.minor)\"))" % sys.version_info[0])
# If PyCall.ji does not exist, create an empty file to force
# recompilation

# If PyCall.jl is already pre-compiled, for the global
# environment, hide it while we are loading PyCall.jl
# for PyJulia which has to compile a new cache if it
# does not exist. However, Julia does not compile a
# new cache if it exists in Base.LOAD_CACHE_PATH[2:end].
# https://github.com/JuliaPy/pyjulia/issues/92#issuecomment-289303684
self._call(u"""
isdir(Base.LOAD_CACHE_PATH[1]) ||
mkpath(Base.LOAD_CACHE_PATH[1])
depsfile = joinpath(Base.LOAD_CACHE_PATH[1],"PyCall.ji")
isfile(depsfile) || touch(depsfile)
for path in Base.LOAD_CACHE_PATH[2:end]
cache = joinpath(path, "PyCall.ji")
backup = joinpath(path, "PyCall.ji.backup")
if isfile(cache)
mv(cache, backup; remove_destination=true)
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a bit orthogonal to this PR but I figured fixing #175 is required for passing the test. Merging this fix should also resolve #109 and #169.

In .travis.yml we have:

pyjulia/.travis.yml

Lines 62 to 64 in 138d47d

- if [ "$CROSS_VERSION" = "1" ]; then
$PYTHON -m tox -e py,py27 -- -s;
fi

Previously, use_separate_cache was True for both py and py27 virtual environments. Thus, no global cache ~/.julia/lib/v0.6/PyCall.ji existed while running the above test. However, with this patch, use_separate_cache is False in py because the code I added sees that PYTHON configured for PyCall uses the same libpython used by py virtual environment prepared by tox. So, after running tox -e py, there is a file ~/.julia/lib/v0.6/PyCall.ji. Now then running tox -e py27 hits the bug #175 because julia 0.6 does not recompile PyCall when it finds ~/.julia/lib/v0.6/PyCall.ji in Base.LOAD_CACHE_PATH[2].

end
end
""")

self._call(u"using PyCall")

if use_separate_cache:
self._call(u"""
for path in Base.LOAD_CACHE_PATH[2:end]
cache = joinpath(path, "PyCall.ji")
backup = joinpath(path, "PyCall.ji.backup")
if !isfile(cache) && isfile(backup)
mv(backup, cache)
end
rm(backup; force=true)
end
""")

# Whether we initialized Julia or not, we MUST create at least one
# instance of PyObject and the convert function. Since these will be
# needed on every call, we hold them in the Julia object itself so
Expand Down
Loading