Inconsistent ability to build cargo on OS X

[ehiggs]

Issue #2295 reported a link error on OS X due to OpenSSL being installed by brew. The fix is to define the following variables:

export OPENSSL_ROOT_DIR=$(brew --prefix openssl)
export OPENSSL_LIB_DIR=$(brew --prefix openssl)"/lib"
export OPENSSL_INCLUDE_DIR=$(brew --prefix openssl)"/include"

However, even this gives spotty results. I find that with these variables defined, I still get link errors to openssl symbols maybe 4/5 times. And sometimes it works.

The link errors are:

  = note: Undefined symbols for architecture x86_64:
  "_EVP_aes_256_ctr", referenced from:
      _libssh2_crypt_method_aes256_ctr in liblibssh2_sys-5b58702936280067.rlib(crypt.c.o)
     (maybe you meant: __libssh2_EVP_aes_256_ctr)
  "_EVP_aes_192_ctr", referenced from:
      _libssh2_crypt_method_aes192_ctr in liblibssh2_sys-5b58702936280067.rlib(crypt.c.o)
     (maybe you meant: __libssh2_EVP_aes_192_ctr)
  "_EVP_aes_128_ctr", referenced from:
      _libssh2_crypt_method_aes128_ctr in liblibssh2_sys-5b58702936280067.rlib(crypt.c.o)
     (maybe you meant: __libssh2_EVP_aes_128_ctr)
ld: symbol(s) not found for architecture x86_64

$ uname -a
Darwin myhostname 15.6.0 Darwin Kernel Version 15.6.0: Thu Jun 23 18:25:34 PDT 2016; root:xnu-3248.60.10~1/RELEASE_X86_64 x86_64
$ brew info openssl| head -1
openssl: stable 1.0.2h (bottled) [keg-only]

[alexcrichton]

To clarify, you cleaned out the build directory after setting the environment variables, right?

[ehiggs]

Yes, when I begin working on #1747, I clean the directory and then run source env.sh where env.sh contains the appropriate exports. Then sometimes it builds and sometimes it fails. I end up just repeatedly building until it links.

[ehiggs]

Log of failed build: http://sprunge.us/YRiQ

Repeated builds can succeed: http://sprunge.us/BUdd

This works/fails with stable and nightly.

[alexcrichton]

It looks like this may be because "-L" "/usr/lib" is getting passed to the linker before "-L" "/usr/local/opt/openssl/lib" which is where OpenSSL is. That typically comes up when pkg-config gets overzealous and finds libs in that directory.

I'm not sure what build script would be asking for linkage to that directory, though? Could you try digging around in target/debug/*/output to see which build script is asking for that?

[ehiggs]

Looks like libz may be the culprit.

$ fgrep '/usr/lib' target/debug/build/*/output 
target/debug/build/libgit2-sys-7e806e051f3bd634/output:-- Found ZLIB: /usr/lib/libz.dylib (found version "1.2.5") 
target/debug/build/libgit2-sys-7e806e051f3bd634/output:-- Found Iconv: -L/usr/lib -liconv
target/debug/build/libssh2-sys-a1010066147c936a/output:-- Found ZLIB: /usr/lib/libz.dylib (found version "1.2.5") 
target/debug/build/libz-sys-b246b9f3989628d7/output:cargo:rustc-link-search=native=/usr/lib


$ cat target/debug/build/libz-sys-b246b9f3989628d7/output 
cargo:rustc-link-search=native=/usr/lib
cargo:rustc-link-lib=z

The build script for libz-sys doesn't ever print only those two options, so I'm intrigued how this works.

[alexcrichton]

That's probably printed by pkg-config. The script for libz-sys probably shouldn't run pkg-config at all on OSX :(

[ehiggs]

The script for libz-sys probably shouldn't run pkg-config at all on OSX :(

I'm not sure. It exists in brew and can be used to find things like libreadline, etc. But OS X has a tonne of packaging headaches for other systems. e.g. Python. OpenSSL is another place where OS X is all screwy.

Either way, I think the main issue here is that the build is not deterministic. If that can be solved, then we can figure out a shim environment to setup so builds can work reliably.

[alexcrichton]

Ok, I've pushed rust-lang/libz-sys@043a4cd. @ehiggs could you try that out?

[ehiggs]

I added the following to my cargo/Cargo.toml:

[replace]
"libz-sys:1.0.5" = {git = "https://github.com/alexcrichton/libz-sys" } 

I can no longer to get the build to break even if I don't have the environment variables set. This is probably because /usr/lib is no longer on the link line at all. So this fix was even better than expected! 🎉 🎉 🎉

Thanks!