`exp10(::BigFloat)` is inaccurate on FreeBSD AArch64 unless MPFR is preloaded

[ararslan]

On FreeBSD AArch64, exp10(::BigFloat) is inaccurate unless:

1. Julia is run with its own bundled libmpfr.so LD_PRELOADed, or
2. Julia is invoked from C with -lmpfr passed when compiling, even if MPFR is not used at all in the C code.

Example of case 1:

$ ./usr/bin/julia -e 'show(exp10(BigFloat(-100.0)))'
8.572405434617062366618376612290057195889324961170989959113443824207244769300536e-101

$ LD_PRELOAD="./usr/lib/libmpfr.so" ./usr/bin/julia -e 'show(exp10(BigFloat(-100.0)))'
9.999999999999999999999999999999999999999999999999999999999999999999999999999955e-101

The latter value is correct. The former is not and causes some math tests to fail.

Example of case 2:

// call this test.c
#include "julia.h"
int main(void) {
    jl_init();
    jl_eval_string("show(exp10(BigFloat(-100.0)))");
    jl_atexit_hook(0);
    return 0;
}

$ cc test.c -o test1 -std=gnu11 -I/root/julia/usr/include/julia -I/root/julia/usr/include -L/root/julia/usr/lib -L/root/julia/usr/lib/julia -fPIC -Wl,--export-dynamic -Wl,-rpath,/root/julia/usr/lib -Wl,-rpath,/root/julia/usr/lib/julia -ljulia
$ ./test1
8.572405434617062366618376612290057195889324961170989959113443824207244769300536e-101
$ cc test.c -o test2 -std=gnu11 -I/root/julia/usr/include/julia -I/root/julia/usr/include -L/root/julia/usr/lib -L/root/julia/usr/lib/julia -fPIC -Wl,--export-dynamic -Wl,-rpath,/root/julia/usr/lib -Wl,-rpath,/root/julia/usr/lib/julia -ljulia -lmpfr
$ ./test2
9.999999999999999999999999999999999999999999999999999999999999999999999999999955e-101

I've attempted quite a bit of debugging, none of which has been particularly fruitful, but I did notice something odd. jl_dlfind can't find mpfr_exp10 (unless MPFR is preloaded) but we still end up calling it:

$ ./usr/bin/julia -E 'ccall(:jl_dlfind, Ptr{UInt8}, (Ptr{UInt8},), "mpfr_exp10")'
Ptr{UInt8}(0x0000000000000000)

$ LD_PRELOAD="./usr/lib/libmpfr.so" ./usr/bin/julia -E 'ccall(:jl_dlfind, Ptr{UInt8}, (Ptr{UInt8},), "mpfr_exp10")'
Ptr{UInt8}(0x0000000000000001)

The latter corresponds to JL_EXE_LIBNAME, and is the result I get on macOS AArch64. The former means we couldn't find the symbol in libjulia-internal, libjulia, or the executable (searched in that order). I would have assumed this means we couldn't find it at all—however, running under LLDB with a breakpoint on mpfr_exp10, we do actually call it.

I'm pretty stumped; this is among the stranger things I've encountered in my time spent using computers. Does anybody know what the issue could be, or have any suggestions for further debugging?

[Keno]

The default explanation is that julia somehow finds a different copy of mpfr somewhere else and that copy is bad. What does Libdl.dllist() show?

[giordano]

It points to julia's libmpfr

julia> using Libdl

julia> filter(contains("libmpfr"), dllist())[]
"/home/mose/repo/julia/usr/bin/../lib/libmpfr.so.6"

julia> exp10(BigFloat(-100.0))
8.572405434617062366618376612290057195889324961170989959113443824207244769300536e-101

Preloading the same library, makes it work, as Alex showed above:

julia> run(addenv(`$(Base.julia_cmd()) -e '@show(exp10(BigFloat(-100))); '`, "LD_PRELOAD" => filter(contains("libmpfr"), dllist())[]));
exp10(BigFloat(-100)) = 9.999999999999999999999999999999999999999999999999999999999999999999999999999955e-101

[giordano]

Uhm, compiling libmpfr locally and moving it to the libs directory with

gmake -j -C deps/ compile-mpfr USE_BINARYBUILDER_MPFR=0
cp deps/scratch/mpfr-4.2.1/src/.libs/libmpfr.so.6.2.1 usr/lib/libmpfr.so.6.2.1

I get

julia> exp10(BigFloat(-100.0))
9.999999999999999999999999999999999999999999999999999999999999999999999999999955e-101

Sounds like we have a borked build? 😥

[ararslan]

Confirmed with a fresh rebuild using USE_BINARYBUILDER_MPFR=0: that produces the correct answer, and all math tests pass with that build. So it seems there is something odd with the library itself, but it's not at all clear to me what that could be, since it actually does function properly outside of the context of Julia. That is, interacting with it from C without involving Julia at all works as expected (and as stated in the issue, it works as expected from Julia if it's preloaded).

[ararslan]

Some observations:

• BinaryBuilder's FreeBSD environment is based on FreeBSD 13.2, whereas I'm running FreeBSD 14.1 locally and Mosè mentioned in Slack that he was running on 15.0 (bleeding edge, not even released yet as of this writing—Mosè ultimate FreeBSD user confirmed). Binaries compiled on some version of FreeBSD should be forward- but not backward-compatible with other versions, so this shouldn't be a factor in theory, but can't be ruled out entirely at this point.
• The RUNPATH of the compiled library is $ORIGIN for the BinaryBuilder-built library and /root/julia/usr/lib:$ORIGIN for the local build. This shouldn't matter, as the build path is the same as the origin for libmpfr.so.
• The locally built libmpfr.so has a PT_PHDR ELF header, but BinaryBuilder's does not. However, this is also true of e.g. libgmp.so, which exhibits no issues, and it seems unlikely that this would consistently cause issues for libmpfr.so and nothing else.

[ararslan]

Mosè tried swapping in the libmpfr.so that I built on FreeBSD 14.1 with USE_BINARYBUILDER=0 on his 15.0 system, and he gets:

$ ./julia -E 'exp10(BigFloat(-100.0))'
9.999999999999999999999999999999999999999999999999999999999999999999999999999955e-101

This suggests to me that the problem is likely at least one of the following:

• The library built on FreeBSD 13.2 is somehow incompatible with FreeBSD ≥14.1. If that's the case, updating the BinaryBuilder environment for FreeBSD AArch64 to 14.1 may fix it. Unfortunately, that may require rebuilding many things in Yggdrasil.
• The BinaryBuilder environment for FreeBSD AArch64 is somehow wonky in a way that's not related to the particular FreeBSD version. I'm not sure how we'd determine that.

An ideal next step would be to try a FreeBSD 13.2 AArch64 VM, but I don't have enough disk space on my machine for another VM. 😐 I have a feeling we'll need to update to 14.1.

Mosè built libmpfr.so on 15.0 and I'll be trying that on 14.1 to see what problems arise, if any, as a semi-analogous situation to updating the BinaryBuilder environment, rebuilding MPFR only, and using the resulting JLL alongside the existing JLLs. (I'm not sure this will tell us a lot for the current situation though given that there may be fewer differences between 14.1 and 15.0 than between 13.2 and 14.1, but it's worth trying.)

[ararslan]

The BinaryBuilder environment for FreeBSD AArch64 has been updated from 13.2 to 14.1, and GMP and MPFR have been rebuilt. However, building Julia locally on 14.1 with the new libgmp and libmpfr libraries swapped in still exhibits this issue.