Julia fails to build on on aarch64 because libquadmath.so is not available

[ggardet]

Julia fails to build on aarch64 because libquadmath.so is not available on this arch.
Log:

[ 4706s] ln -sf /lib64/libgfortran.so.5 /home/abuild/rpmbuild/BUILD/julia-1.6.1/usr/lib/julia/libgfortran.so.5
[ 4706s] System library symlink failure: Unable to locate libquadmath.so.0 on your system!
[ 4706s] make[1]: *** [Makefile:206: /home/abuild/rpmbuild/BUILD/julia-1.6.1/usr/lib/julia/libquadmath.so.0] Error 1
[ 4706s] make[1]: Leaving directory '/home/abuild/rpmbuild/BUILD/julia-1.6.1/base'
[ 4706s] make: *** [Makefile:63: julia-base] Error 2

[Seelengrab]

Are you sure you have all required dependencies installed? There's also #36617. What system are you trying to build on?

[ggardet]

I am trying to build on openSUSE Tumbleweed aarch64, with GCC11. libquadmath is available on other archs, but not on aarch64.

[Seelengrab]

I just noticed you're trying to build 1.6.1 - I don't think 1.6.1 had even build support on aarch64. The issue I linked is about getting it to work in the future (motivated by Apple's recent move to that architecture) and has only recently begun to work. Do the official 1.7 beta binaries work? If so, you can try building 1.7 from source, which will probably work better than 1.6.1.

[Seelengrab]

Fiddling around with the download links on julialang.org/downloads, there were binaries available for aarch64 and 1.6.1, so I suspect there's something going on with your setup. Perhaps the libraries are in a non-standard place, making julia unable to find them?

[giordano]

with GCC11

Any chance to try with GCC 9 or 10? Also, why don't you simply use the official binaries (aarch64 is Tier 1, since it's also tested in CI)?

[ggardet]

with GCC11

Any chance to try with GCC 9 or 10? Also, why don't you simply use the official binaries (aarch64 is Tier 1, since it's also tested in CI)?

The policy of the Linux distribution is to rebuild from sources, not using pre-built binaries.
I may try an older GCC.

[ggardet]

It also fails with GCC7

[giordano]

What if you remove line

julia/base/Makefile

Line 213 in 8c97995

$(eval $(call symlink_system_library,CSL,libquadmath,0))

? But I'm confused, because this platform is Tier 1 and tested regularly 😕

[imciner2]

From some digging, I don't think libquadmath is supposed to exist on aarch64 platforms. At first glance, checking the Fedora GCC builds shows that libquadmath is only enabled on x86_64/ppc64le/ia64:

%ifarch %{ix86} x86_64 ia64 ppc64le
%global build_libquadmath 1
%else
%global build_libquadmath 0
%endif

Second, there was this somewhat recent bug in upstream GCC (https://gcc.gnu.org/bugzilla/show_bug.cgi?id=96016) that says libquadmath fails to configure on aarch64 because it tries to detect if __float128 is a supported type on the architecture - and it isn't for aarch64.

This begs the question, what is different about the CI builds for aarch64 and the original reporter's build? @ggardet can you please post your entire configure line so we can see the options you are building with?

[ggardet]

%ifarch aarch64
  %define julia_march armv8-a
%endif

# We need these compilation flags to avoid error when building MBEDTLS with
# GCC-11.
# Ref.: https://build.opensuse.org/package/show/security:tls/mbedtls
export CFLAGS="%{optflags} -Wno-stringop-overflow -Wno-maybe-uninitialized"
export CXXLAGS="%{optflags} -Wno-stringop-overflow -Wno-maybe-uninitialized"

%define julia_builddir %{_builddir}/%{name}/
make %{?_smp_mflags} \
                     MARCH=%{julia_march} \
%ifarch aarch64
                     JULIA_CPU_TARGET="generic" \
%endif
                     prefix=%{_prefix} \
                     bindir=%{_bindir} \
                     libdir=%{_libdir} \
                     libexecdir=%{_libexecdir} \
                     datarootdir=%{_datarootdir} \
                     includedir=%{_includedir} \
                     sysconfdir=%{_sysconfdir} \
                     %{juliabuildopts} \
                     release debug

With juliabuildopts being:

BUNDLE_DEBUG_LIBS=1 \
NO_GIT=1 \
USE_SYSTEM_LLVM=0 \
USE_SYSTEM_LIBUNWIND=1 \
USE_SYSTEM_READLINE=1 \
USE_SYSTEM_PCRE=1 \
USE_SYSTEM_LIBM=0 \
USE_SYSTEM_OPENLIBM=0 \
USE_SYSTEM_OPENSPECFUN=1 \
USE_SYSTEM_DSFMT=0 \
USE_SYSTEM_BLAS=1 \
LIBBLAS=-lopenblas \
LIBBLASNAME=libopenblas \
USE_SYSTEM_LAPACK=1 \
LIBLAPACK=-libopenblas \
LIBLAPACKNAME=libopenblas \
USE_SYSTEM_FFTW=1 \
LIBFFTWNAME=libfftw3_threads.so.3 \
LIBFFTWFNAME=libfftw3f_threads.so.3 \
USE_SYSTEM_GMP=1 \
USE_SYSTEM_MPFR=1 \
USE_SYSTEM_ARPACK=1 \
USE_SYSTEM_SUITESPARSE=1 \
USE_SYSTEM_ZLIB=1 \
USE_SYSTEM_GRISU=1 \
USE_SYSTEM_LIBUV=0 \
USE_SYSTEM_UTF8PROC=0 \
USE_SYSTEM_PATCHELF=1 \
USE_SYSTEM_P7ZIP=1 \
USE_SYSTEM_MBEDTLS=0 \
USE_SYSTEM_LIBSSH2=1 \
USE_SYSTEM_LIBGIT2=0 \
USE_SYSTEM_CURL=1 \
USE_SYSTEM_NGHTTP2=1 \
USE_SYSTEM_CSL=1 \
USE_MKL=0 \
USE_BLAS64=0 \
USE_LLVM_SHLIB=1 \
VERBOSE=1 \
USECLANG=0 \
USE_BINARYBUILDER=0

[ggardet]

What if you remove line

julia/base/Makefile

Line 213 in 8c97995

$(eval $(call symlink_system_library,CSL,libquadmath,0))

? But I'm confused, because this platform is Tier 1 and tested regularly 😕

If I drop this line, the build of julia 1.6.3 succeeds on aarch64 as well as on x86_64.

[giordano]

And what if you replace that line with

$(eval $(call symlink_system_library,CSL,libquadmath,0,ALLOW_FAILURE))

? Allowing the library not to be there is used in

julia/base/Makefile

Lines 215 to 218 in 8c97995

	# We allow libssp, libatomic and libgomp to fail as they are not available on all systems
	$(eval $(call symlink_system_library,CSL,libssp,0,ALLOW_FAILURE))
	$(eval $(call symlink_system_library,CSL,libatomic,1,ALLOW_FAILURE))
	$(eval $(call symlink_system_library,CSL,libgomp,1,ALLOW_FAILURE))

which are other libraries which are not available on all platforms. Again, no idea how the build bot currently works at all.

[imciner2]

It is probably succeeding on the buildbot because the official packaging is using the BinaryBuilder-provided CSL, which is automatically extracted into the local lib directory. The build showing the issue is using the system CSL (it defines USE_SYSTEM_CSL=1 on the build options), so the makefile then tries to make a symlink in the local library directory to the system's version of libquadmath. So the buildbot build step isn't actually doing this linking.

The way to fix this would be to guard this line inside an exclusion for aarch64 (I don't think we want to unconditionally make it allowed to fail, since in all other cases it being missing is probably a problem).

julia/base/Makefile

Line 213 in 8c97995

$(eval $(call symlink_system_library,CSL,libquadmath,0))

And now that I think about it some more, shouldn't that line be inside this check as well?

julia/base/Makefile

Line 210 in 8c97995

ifneq (,$(LIBGFORTRAN_VERSION))

Since I think libquadmath is only available if gfortran is available (or at least it is packaged in the gfortran CSL tar).

And as a note, this line should probably get a different comment (since it currently reads that the libquadmath is available on all platforms).

julia/deps/csl.mk

Lines 64 to 65 in 27c0291

	# These are all libraries that we should always have
	$(eval $(call copy_csl,$(call versioned_libname,libquadmath,0)))

[giordano]

It is probably succeeding on the buildbot because the official packaging is using the BinaryBuilder-provided CSL

But CSL doesn't provide libquadmath for aarch64

[imciner2]

It is probably succeeding on the buildbot because the official packaging is using the BinaryBuilder-provided CSL

But CSL doesn't provide libquadmath for aarch64

Yes, but because the build isn't being told to use the system version (via USE_SYSTEM_CSL=1), the symlink step of the build is never getting run. If you trace through the makefile, you find that the line

julia/base/Makefile

Line 213 in 8c97995

$(eval $(call symlink_system_library,CSL,libquadmath,0))

will only be run when this condition is satisfied, with $1 being CSL for libquadmath - making this line only run when the variable USE_SYSTEM_CSL is not equal to 0.

julia/base/Makefile

Lines 183 to 185 in 8c97995

	ifneq ($$(USE_SYSTEM_$1),0)
	SYMLINK_SYSTEM_LIBRARIES += symlink_$2
	endif

Because that line actually adds the symlink target to the list of targets to run in the full symlink target here

julia/base/Makefile

Line 268 in 8c97995

symlink_system_libraries: $(SYMLINK_SYSTEM_LIBRARIES)

So because USE_SYSTEM_CSL=1 is given in the build options, the symlink_system_libraries step of the build will try to symlink libquadmath.so into the local directory. On the buildbot version though, USE_SYSTEM_CSL=0 is used, so symlink_system_libraries will not try to make a symlink.