extra-lib-dirs does not propagate to dependencies when building

[diasbruno]

Describe the bug
extra-lib-dirs does not propagate to dependencies when building.

Dependency A uses a library that is installed on a non-standard path (in this case is brew on macOS).
Project B used A and tries to compile with

cabal v2-build --extra-lib-dirs=/opt/homebrew/Cellar/unac/1.8.0/lib

To Reproduce
Steps to reproduce the behavior:

• Install a library on a non-standard location
• Write a package that has extra-libraries: {library}
• Add the package to a project
• Try to compile using --extra-lib-dirs so the compiler can find the library

$ cabal v2-build --extra-lib-dirs=/opt/homebrew/Cellar/unac/1.8.0/lib

Expected behavior
extra-libraries should be found at --extra-lib-dirs paths.

System information

• macOS (latest version)
• cabal 3.10.1.0 , ghc 9.2.8 (nix)

Additional context
cabal and ghc might be outdated. I'll see if it also happens on latest versions.

Logs:

$ make build
cabal v2-build --extra-lib-dirs=/opt/homebrew/Cellar/unac/1.8.0/lib
Build profile: -w ghc-9.2.8 -O1
In order, the following will be built (use -v for more details):
 - unac-bindings-0.1.0.0 (lib) (requires build)
 - Slugify-0.1.0.0 (lib) (first run)
 - Slugify-0.1.0.0 (test:Slugify-test) (first run)
Starting     unac-bindings-0.1.0.0 (lib)

Failed to build unac-bindings-0.1.0.0. The failure occurred during the
configure step.
Build log (
/Users/dias/.cache/cabal/logs/ghc-9.2.8/nc-bndngs-0.1.0.0-5665bd12.log ):
Configuring library for unac-bindings-0.1.0.0..
Error: .cabal-wrapped: Missing dependency on a foreign library:
* Missing (or bad) C library: unac
This problem can usually be solved by installing the system package that
provides this library (you may need the "-dev" version). If the library is
already installed but in a non-standard location then you can use the flags
--extra-include-dirs= and --extra-lib-dirs= to specify where it is.If the
library file does exist, it may contain errors that are caught by the C
compiler at the preprocessing stage. In this case you can re-run configure
with the verbosity flag -v3 to see the error messages.

Error: cabal: Failed to build unac-bindings-0.1.0.0 (which is required by
test:Slugify-test from Slugify-0.1.0.0). See the build log above for details.

make: *** [makefile:13: build] Error 1

[diasbruno]

Also, appreciate any suggestions to linking the library more reliably.

[gbaz]

This is by design for now. See: https://cabal.readthedocs.io/en/stable/cabal-project.html#package-configuration-options

There's discussion at #2997 on various alternatives that might be possible, but its all a bit tricky.

[gbaz]

(note that the ticket provides a recommended way to specify extra-lib-dirs for dependencies, specifically through use of a cabal.project file.)

[andreabedini]

tl;dr: --extra-lib-dirs applies only the local packages (e.g. the ones mentioned directly in cabal.project, or the package in the currect directory in absence of cabal.project).
You want to pass --extra-lib-dirs to unac-bindings which is a dependency of Slugify and not a local package.
This requires using a cabal.project file, which allows to specify which package to pass that option. If I get the syntax right, you can write

package unac-bindings
  extra-lib-dirs: /opt/homebrew/Cellar/unac/1.8.0/lib

either in cabal.project if you are already using one, or in cabal.project.local. (The only difference being that if cabal.project is missing packages: . is implied, so if you create a cabal.project with the snippet above you need to remember to add also packages: .).

EDIT: actually, as @gbaz mentions in the linked thread, the best practice is to put that snippet in cabal.project.local, since that path is system dependent and cabal.project.local is typically ignored doing source version control).

[diasbruno]

Thanks, @gbaz @andreabedini. I'll have a look on the references.
I'm closing this to let #2997 opened (it has more info and solutions).

[diasbruno]

@andreabedini Ah, the package unac-bindings worked. 👏

[mouse07410]

I'm against closing this issue, because #2997 addresses only some use cases. For me, an important use case is when I'm installing a package from (e.g.) Hackage, that package pulls a pile of dependencies, and all of them must link to a library that's not on the standard system search path.

cabal.project won't work here, because the top package is not local.

I mentioned all this in #2997.

[andreabedini]

@mouse07410 I wrote a summary in #2997, I hope I have captured your use case.

Hackage, that package pulls a pile of dependencies, and all of them must link to a library that's not on the standard system search path.
cabal.project won't work here, because the top package is not local.

I am not sure what problem you are running into, what do you mean by "top-package"? (I did try by compiling Z-Botan but it worked without any extra library 🤷).

I listed the workarounds I know of in my summary. I suggest we leave this closed and move any discussion to the other ticket.

[geekosaur]

I see "homebrew" in there so I presume they're on a Mac. Homebrew installs most things in /usr/local where the compiler will find them by default, but things which are available in multiple versions are installed elsewhere so they don't conflict with each other, among other issues. (Also, X11 is in a weird place.) Anyway, I had to use extra lib dirs a lot when I was on a Mac.

[diasbruno]

@geekosaur I had previous installations of homebrew that create /Cellar under the /usr folder, but this one (fresh install) store everything on /opt 🤷‍♂️.

But good point. I'll check if installing packages from brew on a nix shell can influence where the libraries are installed...

[diasbruno]

Another option I tried was pkg-config, but I gave up because, for some reason, it wasn't finding the .pc after including the correct path to PKG_CONFIG_PATH.

[mouse07410]

@mouse07410 I wrote a summary in #2997, I hope I have captured your use case.

Hackage, that package pulls a pile of dependencies, and all of them must link to a library that's not on the standard system search path.

cabal.project won't work here, because the top package is not local.

I am not sure what problem you are running into, what do you mean by "top-package"? (I did try by compiling Z-Botan but it worked without any extra library 🤷).

I listed the workarounds I know of in my summary. I suggest we leave this closed and move any discussion to the other ticket.

Your post in the other ticket summed it up: my use case will not be supported.

Specifically, if a library, like OpenSSL libs, that is not in a standard location like /usr/lib or /usr/local/lib that I want the package from Hackage and all of its dependencies to use, they're no way to do that.

[andreabedini]

Your post in the other ticket summed it up: my use case will not be supported.

Did I say it will not be supported? If I did it was not what I had meant to say. My intention was to describe the current status quo, issues and workarounds.

Specifically, if a library, like OpenSSL libs, that is not in a standard location like /usr/lib or /usr/local/lib that I want the package from Hackage and all of its dependencies to use, they're no way to do that.

I listed two options: 1) adding extra-lib-dirs to the global cabal config 2) setting the LIBRARY_PATH environment variable. I am afraid that's all the help I can offer.

[mouse07410]

I listed two options: 1) adding extra-lib-dirs to the global cabal config

This is pretty much what I'm doing now. And it's fine for libraries that are truly system-wide (though in non-standard locations).

However, in (admittedly rate) cases when some groups of packages need to use a "standard" (usually, OS-provided) library, while other groups require the "non-standard" (e.g., Macports- or Brew-installed), there's a problem.

2. setting the LIBRARY_PATH environment variable.

No, this doesn't work. LIBRARY_PATH tells the loader where to find the already-linked dynamic/shared libraries the executable needs to load. It does not tell the linker where to look for the libraries (static or dynamic) to build/link the executable.

I am afraid that's all the help I can offer.

That's what I meant in my previous post. One partial solution (admittedly, good enough for majority of my use cases, and already employed), and one no-solution.

[andreabedini]

LIBRARY_PATH tells the loader where to find the already-linked dynamic/shared libraries the executable needs to load. It does not tell the linker where to look for the libraries (static or dynamic) to build/link the executable.

Unless I got the names mixed up, I think gcc uses LIBRARY_PATH to inform the linker while the loader uses LD_LIBRARY_PATH. This, of course, assumes gcc is doing the linking.

[mouse07410]

This, of course, assumes gcc is doing the linking.

GCC to the best of my knowledge invokes system linker (from XCode) on MacOS, but I'm not sure.

In any case, thanks for the useful information - let me test it. I assume LIBRARY_PATH behavior should follow your interpretation with Clang as well? Or only for GCC?

[andreabedini]

It's described in gcc documentation
https://gcc.gnu.org/onlinedocs/gcc/Environment-Variables.html
and it appears in clang's source code
https://github.com/llvm/llvm-project/blob/99fb65fa7a65b9a3f961d6ef715324c2f8637766/clang/lib/Driver/ToolChains/CommonArgs.cpp#L265-L268

Let us know if it works!