Error Building Android, CLang, MacOS, NDK 20: Final linker can't find crtbegin_so.o or crtend_so.o

[dugsmith]

Hello,

I've worked through many things to make it almost all the way through a build of OpenBLAS for armeabi-v7a and arm64-v8a with the above listed platform and NDK, but am unable to overcome the what appears to be the last linker problem.

Here's the error I'm seeing:

 "/Users/doug.smith/Library/Android/sdk/ndk/20.1.5948944/toolchains/llvm/prebuilt/darwin-x86_64/bin/aarch64-linux-android-ld" --sysroot=/Users/doug.smith/Library/Android/sdk/ndk/20.1.5948944/toolchains/llvm/prebuilt/darwin-x86_64/sysroot -EL --fix-cortex-a53-843419 -z now -z relro --hash-style=both --enable-new-dtags --eh-frame-hdr -m aarch64linux -shared -o ../libopenblas_armv8p-r0.3.9.so crtbegin_so.o -L/Users/doug.smith/Library/Android/sdk/ndk/20.1.5948944/toolchains/aarch64-linux-android-4.9/prebuilt/darwin-x86_64/lib/gcc/aarch64-linux-android/4.9.x -L/Users/doug.smith/Library/Android/sdk/ndk/20.1.5948944/toolchains/llvm/prebuilt/darwin-x86_64/lib64/clang/8.0.7/lib/linux/aarch64 -L/Users/doug.smith/Library/Android/sdk/ndk/20.1.5948944/toolchains/llvm/prebuilt/darwin-x86_64/bin/../lib/gcc/aarch64-linux-android/4.9.x -L/Users/doug.smith/Library/Android/sdk/ndk/20.1.5948944/toolchains/llvm/prebuilt/darwin-x86_64/bin/../lib/gcc/aarch64-linux-android/4.9.x/../../../../aarch64-linux-android/lib/../lib64 -L/Users/doug.smith/Library/Android/sdk/ndk/20.1.5948944/toolchains/llvm/prebuilt/darwin-x86_64/sysroot/usr/lib/aarch64-linux-android -L/Users/doug.smith/Library/Android/sdk/ndk/20.1.5948944/toolchains/llvm/prebuilt/darwin-x86_64/bin/../lib/gcc/aarch64-linux-android/4.9.x/../../../../aarch64-linux-android/lib -L/Users/doug.smith/Library/Android/sdk/ndk/20.1.5948944/toolchains/llvm/prebuilt/darwin-x86_64/sysroot/usr/lib -lm --whole-archive ../libopenblas_armv8p-r0.3.9.a --no-whole-archive -soname libopenblas.so -lm -lm -lm -lgcc -ldl -lc -lgcc -ldl crtend_so.o
/Users/doug.smith/Library/Android/sdk/ndk/20.1.5948944/toolchains/llvm/prebuilt/darwin-x86_64/bin/aarch64-linux-android-ld: cannot find crtbegin_so.o: No such file or directory
/Users/doug.smith/Library/Android/sdk/ndk/20.1.5948944/toolchains/llvm/prebuilt/darwin-x86_64/bin/aarch64-linux-android-ld: cannot find crtend_so.o: No such file or directory
clang: error: linker command failed with exit code 1 (use -v to see invocation)
make[1]: *** [../libopenblas_armv8p-r0.3.9.so] Error 1
make: *** [shared] Error 2

Here's the shell script I'm using:

#!/bin/bash

# Starting from: https://github.com/xianyi/OpenBLAS/wiki/How-to-build-OpenBLAS-for-Android#building-with-android-ndk-using-clang-compiler

echo "Starting OpenBLAS Build for Android"
echo

# OpenBLAS must be cloned under this name and checked out at the right version - see README.md
cd OpenBLAS || exit $?

NDK_BUNDLE_DIR=$1

echo "START building arm64-v8a #####################################################:"

abi=arm64-v8a

# Set the PATH to contain paths to clang and aarch64-linux-android-* utilities
export PATH=${NDK_BUNDLE_DIR}/toolchains/aarch64-linux-android-4.9/prebuilt/darwin-x86_64/aarch64-linux-android/bin:${NDK_BUNDLE_DIR}/toolchains/llvm/prebuilt/darwin-x86_64/bin:$PATH

echo "PATH=${PATH}"
echo

echo "which clang: "
which clang
echo

echo "which ld: "
which ld
echo

# Set LDFLAGS so that the linker finds what it needs

# Recommended path in OpenBLAS Wiki
# export LDFLAGS="-L${NDK_BUNDLE_DIR}/toolchains/aarch64-linux-android-4.9/prebuilt/darwin-x86_64/lib/gcc/aarch64-linux-android/4.9.x -lm "

# Path where files are actually located in NDK 20 (neither this nor the above work)
export LDFLAGS="-L${NDK_BUNDLE_DIR}/toolchains/llvm/prebuilt/darwin-x86_64/sysroot/usr/lib/aarch64-linux-android/23 -L${NDK_BUNDLE_DIR}/platforms/android-23/arch-arm64/usr/lib -lm "

CLANG_FLAGS="-v -target aarch64-linux-android -marm -mfpu=vfp -mfloat-abi=softfp --sysroot=${NDK_BUNDLE_DIR}/toolchains/llvm/prebuilt/darwin-x86_64/sysroot "

AR="${NDK_BUNDLE_DIR}/toolchains/aarch64-linux-android-4.9/prebuilt/darwin-x86_64/bin/aarch64-linux-android-ar"

command="make TARGET=ARMV8 ONLY_CBLAS=1 CC='clang ${CLANG_FLAGS}' AR=${AR} HOSTCC=gcc -j4"

echo "make clean"
echo
make clean || exit $?

echo $command
echo
eval $command || exit $?

I appreciate any help you can offer!

[brada4]

You have to export NDK_BUNDLE_DIR so that NDK finds its own C runtime.

[dugsmith]

@brada4 Thank you for your fast response.

I tried export NDK_BUNDLE_DIR=$1, but I see the same error.

I think the NDK is finding its own runtime successfully given the output of the which statements:

which clang: 
/Users/doug.smith/Library/Android/sdk/ndk/20.1.5948944/toolchains/llvm/prebuilt/darwin-x86_64/bin/clang

which ar: 
/Users/doug.smith/Library/Android/sdk/ndk/20.1.5948944/toolchains/aarch64-linux-android-4.9/prebuilt/darwin-x86_64/aarch64-linux-android/bin/ar

which ld: 
/Users/doug.smith/Library/Android/sdk/ndk/20.1.5948944/toolchains/aarch64-linux-android-4.9/prebuilt/darwin-x86_64/aarch64-linux-android/bin/ld

[martin-frbg]

Could you check where in your NDK installation path the crtbegin_so.o is located ? Perhaps that would provide a hint to what is missing

[dugsmith]

@martin-frbg: There is a crtbegin_so.o file located underneath both of the directories listed in LDFLAGS above. (I put them both in, have tried one or the other, it still says it can't find it.) Specifically (targeting API 23):

{NDK_BUNDLE_DIR}/toolchains/llvm/prebuilt/darwin-x86_64/sysroot/usr/lib/aarch64-linux-android/23
{NDK_BUNDLE_DIR}/platforms/android-23/arch-arm64/usr/lib

[martin-frbg]

#2005 (comment) looks related though not on darwin

[dugsmith]

@martin-frbg: Thank you for pointing me to #2005, that was related and very helpful! I was able to make it work starting from there.

Here's what worked (NDK r20b, targeting API 23 instead of 21 as in the #2005). The difference between #2005 and mine (besides mine being Darwin) is that clang was not located under $TOOLCHAIN. I also added other CLANG args we wanted from our previous usage of OpenBLAS.

armeabi-v7a:

#!/bin/bash

export NDK=$1
export TOOLCHAIN=$NDK/toolchains/arm-linux-androideabi-4.9/prebuilt/darwin-x86_64
export CLANG=$NDK/toolchains/llvm/prebuilt/darwin-x86_64/bin/armv7a-linux-androideabi23-clang
export AR=$TOOLCHAIN/bin/arm-linux-androideabi-ar
export SOURCE_DIR=OpenBLAS
export INSTALL_DIR=../android-openblas/armeabi-v7a

rm -rf $INSTALL_DIR
cd $SOURCE_DIR || exit $?

make clean || exit $?

make \
    TARGET=ARMV7 \
    ONLY_CBLAS=1 \
    CC="$CLANG -g -marm -mfpu=vfp -mfloat-abi=softfp " \
    AR=$AR \
    HOSTCC=gcc \
    ARM_SOFTFP_ABI=1 \
    -j4 \
    || exit $?

make PREFIX=$INSTALL_DIR install || exit $?

arm64-v8a:

#!/bin/bash

export NDK=$1
export TOOLCHAIN=$NDK/toolchains/aarch64-linux-android-4.9/prebuilt/darwin-x86_64
export CLANG=$NDK/toolchains/llvm/prebuilt/darwin-x86_64/bin/aarch64-linux-android23-clang
export AR=$TOOLCHAIN/bin/aarch64-linux-android-ar
export SOURCE_DIR=OpenBLAS
export INSTALL_DIR=../android-openblas/arm64-v8a

rm -rf $INSTALL_DIR
cd $SOURCE_DIR || exit $?

make clean || exit $?

make \
    TARGET=CORTEXA57 \
    ONLY_CBLAS=1 \
    CC="$CLANG -g -marm -mfpu=vfp -mfloat-abi=softfp " \
    AR=$AR \
    HOSTCC=gcc \
    -j4 \
    || exit $?

make PREFIX=$INSTALL_DIR install || exit $?

It might help another user if you updated your Wiki. Particularly, this is what I was trying to make work and couldn't ever get it to work: https://github.com/xianyi/OpenBLAS/wiki/How-to-build-OpenBLAS-for-Android#building-with-android-ndk-using-clang-compiler.

[martin-frbg]

Unfortunately that wiki page becomes outdated every now and then (and probably confused/confusing in between) as I currently do not have the environments to test Android builds myself. I usually try to update it with information gleaned from tickets such as this but I must have forgotten to do this with GavinAndre's contribution, only dimly remembering that we had a related ticket somewhere. Btw you have write access to the wiki as well.