ccall does not support non-const library names

[stevengj]

For PyCall, I need to determine the Python shared library name at runtime (by calling python and querying its LDLIBRARY), but ccall does not allow the library name to be a non-const variable:

libpython = "libpython2.6"
ccall((:Py_Initialize, libpython), Void, ())

gives ERROR: type: anonymous: in ccall: function argument not a pointer or valid, constant expression, expected BitsKind, got (AbstractKind,)

However, as an ugly hack:

const libpython = ""
libpython = "libpython2.6"
ccall((:Py_Initialize, libpython), Void, ())

seems to sort of work (Julia allows me to modify libpython even though I declared it const, which seems mildly horrifying and likely to break in future versions). But from struggling to use this hack in a module, it seems like it breaks easily even now.

Please provide a way to do this properly.

PS. The more I use ccall the more I hate that it behaves so differently from an ordinary function call. No splicing (#1313), apparently no macros or functions of any kind in some of the arguments, and probably there are other limitations I haven't noticed yet. All of this goofiness seems like a bug to me.

[pao]

A slightly less ugly hack would be to fall back on dlopen, which I don't believe has such a restriction.

[stevengj]

Good point, thanks for the tip! I still find it frustrating that ccall is so unlike most Julia functions in how it handles it arguments, but this is a good workaround.

[vtjnash]

part of the problem with ccall is that a lot of it must be known at compile time to be efficient. the restriction on having the symbol lookup be const is supposed to hint that changing the variable might not have an effect, and would be much slower if allowed (the library name is cached at compile-time for efficiency). it's possible to allow the library to be non-constant, but this would force a call to dlsym every time, since we don't know when or if the library name will ever change.

c-functions jl_wrap_raw_dl_handle(dlopen()) or julia's dlopen and dlsym can be used now for this effect, but require extra work on the part of the programmer to manage the method lookup cache

I'm interested to hear if you have ideas for how to make this easier, without introducing another performance trap

[toivoh]

Even though the python shared library name is not known until runtime, there's nothing that prevents you from forming the code that needs to know that name afterwards, and then compiling it, e.g. by passing function definitions to eval. That's the beauty of JIT compilation in a homoiconic language: runtime and compile time can be intermixed.

[stevengj]

Fortunately, the dlopen/dlsym stuff, now that I've noticed it, turns out to be fairly easy to use for this purpose, and also allow me to cache the library for repeated calls. I'm going to close this issue for now, but it would be good to document dlopen and dlsym in the section of the manual on calling external C code.

[pao]

Since dlopen/dlsym used to be the only way to use ccall, you can probably recover some text from a prior version of the documentation.

[StefanKarpinski]

dlopen could also be given an optional argument indicating whether to load symbols globally or not.

[Keno]

Tk and Cairo have a similar problem, which I solved using this quick hack at the time. We should probably either make it official/document it or think of a better way.