Corner cases that fail for lack of `LD_LIBRARY_PATH`

[ocaisa]

Python packages may use ctypes to load libraries:

cdll.LoadLibrary("libprotoc.so.3.13.0.0")

but this relies on the linker being able to find the libraries, which can fail in our current setup.

[mboisson]

This is how we fix this in the Compute Canada software stack: ComputeCanada/software-stack#100

[mboisson]

Relevant issue in cpython upstream: python/cpython#65241

[ocaisa]

@mboisson I'm thinking about making a trivial update to ctypes via a sed command that would change https://github.com/python/cpython/blob/main/Lib/ctypes/util.py#L316 to

libpath = os.environ.get('LD_LIBRARY_PATH', os.environ.get('LIBRARY_PATH'))

that should also solve the issue in our case I think.

[boegel]

@ocaisa Why don't we install that as a proper patch for the Python installations we have in EESSI (or even always, I guess it wouldn't do any harm)?

[ocaisa]

Yes sure, I was thinking of doing it via sed ... && in preconfigopts for a Python parse hook

[ocaisa]

I guess I could actually go ahead and change this in the easyblock. The only scenario I can see this being a problem is where you break the detection by defining LD_LIBRARY_PATH . I could make the change conditional on LD_LIBRARY_PATH being an excluded envvar.

[ocaisa]

See easybuilders/easybuild-easyblocks#3352

[mboisson]

@mboisson I'm thinking about making a trivial update to ctypes via a sed command that would change https://github.com/python/cpython/blob/main/Lib/ctypes/util.py#L316 to

libpath = os.environ.get('LD_LIBRARY_PATH', os.environ.get('LIBRARY_PATH'))

that should also solve the issue in our case I think.

that would not fix anything if both LD_LIBRARY_PATH and LIBRARY_PATH are defined. While we usually don't define LD_LIBRARY_PATH, it is sometimes necessary, and users routinely do it.

[mboisson]

In my opinion, the correct fix is that find_library needs to return the full path. It actually does find the library through _findLib_gcc, since it calls the compilers which relies on LIBRARY_PATH, but it then returns only the file name of the SO instead of the full path as it should. Since it only returns the SO name and not the full path, it is not usable down the line.

See:
ComputeCanada/software-stack#100 (comment)
python/cpython#65241

This is what we do in our custom ctypes library:
ComputeCanada/custom_ctypes@b3a0288

Unfortunately, the patch is slightly different for each version of python.

[ocaisa]

PR is at easybuilders/easybuild-easyblocks#3352 , I've only triggered this in specific circumstances and respect the use of both LD_LIBRARY_PATH and LIBRARY_PATH in those circumstances