Building for Android doesn't generate shared lib (.so)

[scorpeeon]

I made a standalone NDK toolchain with Fortran support using the latest NDK (r14b) following these instructions:
https://github.com/buffer51/android-gfortran

Using latest develop branch of OpenBLAS, followed this guide to build OpenBLAS with Fortran:
https://github.com/xianyi/OpenBLAS/wiki/How-to-build-OpenBLAS-for-Android

The build itself seemingly goes fine (without error) but the install command says it can't find the .so file - I checked, and it's not generated for some reason (can't find any .so files within the folders). So the install command only generates the header files and the static (.a) lib, but not the shared (.so) lib. Tried both arm and arm64, happens in both versions.

The relevant error message (when running the install command) :

Copying the shared library to /home/scrpn/blas/OpenBLAS/install/lib
install: cannot stat 'libopenblas_armv7p-r0.2.20.dev.so': No such file or directory
make[1]: *** [install] Error 1

Also attached build log.
log.txt

Any ideas what's wrong?

[martin-frbg]

Build log looks a bit short, is that from the complete build (after "make clean") or continuing after an earlier interruption ?
See if you set NO_SHARED=1 somewhere, otherwise the shared library should be built by the toplevel Makefile (which should be running make -C exports so and adding a few softlinks afterwards)

[scorpeeon]

@martin-frbg Yeah you're right, it's probably not the full log, it's the log after a build was already done (or at least attempted). The original log was too long to copy out of the console as I couldn't scroll back to the beginning - is the build log saved somewhere or should I just increase the scrollback size of the terminal?

I only ran the commands from the guides I referenced and I don't thinks there's a NO_SHARED=1 in any of those. Gonna try to get a full log tomorrow.

[martin-frbg]

The build log is not saved, what you could do is redirect the console output to a file.
Looking at the wiki recipe it could be that the problem comes simply from building only the "libs" target, try repeating that command line with "shared" instead of "libs" at the end.

[brada4]

You can collect output by typing 'script'
Pobably add -s parameter to make at first attempt to shorten the output to read
Once done with demonstrating error, exit script wrapper (Ctrl-D/exit/whatever) and examine file 'typescript' for your output.

[scorpeeon]

Thanks for the tips.
I successfully grabbed a full log, attached it.

log.txt

Meanwhile I also tried the "shared" option instead of "libs" as suggested by @martin-frbg, first it gave an error saying I first need static libs to do that. I suppose one could use both the libs and shared options to get both (or just run it with first the libs. then the shared option?), but while at it, I tried the "all" option to see what happens and it actually ran all the way through with no errors and I could also successfully run the install command and saw the shared lib files were generated.
Actually 3 shared lib files have been generated: libopenblas.so, libopenblas.so.0, libopenblas_armv7p-r0.2.20.dev.so - I'm not sure why - I think all 3 are binary equal).

Just to have it in case it's needed I also included to full log for building with the "all" option:
log_all.txt

So it looks like @martin-frbg was on point, it was just because of the options that the .so didn't got generated.

I haven't actually tested it yet though and the Android guide says:
"Since we are cross-compiling, we make the libs recipe, not all. Otherwise you will get errors when trying to link/run tests."

So I probably shouldn't use the all option after all. I checked in the Makefile and it looks like the all option means these: libs netlib tests shared
We know we need libs and shared (well, technically one of them could be enough, but I'm not sure yet if we'd use at as a shared or static lib so it's handy to have both), and according to the Android guide we shouldn't include tests. What is netlib, do we need that? After a quick search I found it probably refers to Netlib Lapack, but I'm still unsure what this option exactly does. Does it mean it includes the full Lapack lib if this option is set?

[martin-frbg]

Yeah, my suggestion was assuming that you had already run make ... libs before. It could be that the wiki entry is from a time where the build system did not yet detect cross-compilation, or did not know to avoid running the tests in the cross-compilation case. (Obviously it would not make sense to try to run the newly built Android code on the desktop computer used for compilation).
The 3 shared lib files are indeed equal - actually I think you will find that only the one with the longest name is an actual file, while the other two are symbolic links to it so that one can refer to it as just "openblas" for linking or when running code linked against it.
Finally as you assumed the make ... netlib would build the lapack part (but only if you have the gfortran compiler installed, as LAPACK is written in FORTRAN). So in effect running make all should not hurt.

[scorpeeon]

@martin-frbg Thanks for the quick reply.

Indeed it looks like the multiple files are symbolic links to the same file (Ubuntu file manager didn't display this for some reason).

Yeah, so one of my goals would be to also include Lapack, so I'm using a custom standalone Android NDK toolchain (referenced in my first post) with Fortran support, so compiling Fortran (and Lapack) shouldn't be an issue.

One thing I don't quite get (sorry, I'm pretty new at this) yet is how can I combine these options (libs, shared, netlib). Can i just call it with multiple options like make ... libs netlib shared?
Or first I would have to call "libs", then "shared" - and also "netlib" at some point? This is confusing to me because even when I ran the "all" option, in the install folder I found no separate file/lib for Lapack - I suspect it is included in the same libopenblas static/shared files, right? But in this case what happens if I just run make ... netlib without either "libs" or "shared" options, where is Lapack then placed? (I guess it doesn't make sense to only run it with "netlib" option, only if you later also call it with "libs" or "shared"?)

I found that if I call it with options make ... libs shared, I get several errors like these:

/tmp/ccjsikY2.o:linktest.c:function main: error: undefined reference to 'spotri_'
/tmp/ccjsikY2.o:linktest.c:function main: error: undefined reference to 'dpotri_'
/tmp/ccjsikY2.o:linktest.c:function main: error: undefined reference to 'cpotri_'
[...]
/tmp/ccjsikY2.o:linktest.c:function main: error: undefined reference to 'LAPACKE_cgb_nancheck'
/tmp/ccjsikY2.o:linktest.c:function main: error: undefined reference to 'LAPACKE_cgb_trans'
[...]

So it looks like the shared libs needs netlib/lapack.
But if I run it with make ... libs netlib shared it seemingly compiles fine - so it's probably my best option, right?

Or you think I can just use the "all" option (which also includes tests) and ignore the warning regarding the link/run errors in the guide? (of course I probably wouldn't want to run tests on Android, I would just use the library)

[martin-frbg]

Yes, just use the "all" option. The LAPACK functions will be included in the library.

[scorpeeon]

@martin-frbg great, thanks! Now I'm trying to use the shared library in an Android project. The problem is that the generated shared lib (libopenblas_armv8p-r0.2.20.dev.so) has a versioned SONAME:
0x000000000000000e (SONAME) Library soname: [libopenblas.so.0]

(see: the .0 after the .so part). Android doesn't support versioned SONAMEs, so there shouldn't be anything after the ".so" part.

Is there any easy way to modify the build process so that the SONAME entry would be libopenblas.so?

[martin-frbg]

Try removing the first occurence of .$(MAJOR_VERSION) in exports/Makefile - if this turns out to be the right spot we can try to put some form of "if android" around it.

[scorpeeon]

@martin-frbg Thanks, that indeed looks like it's working!
First I found a part similar to that in the Makefile of the root directory where Android is handled the same way as Linux and SunOS, but that part looks like it's only creating the symbolic links. In this case it's creating two links one of them being $(LIBSONAME) $(LIBPREFIX).so.$(MAJOR_VERSION) - which doesn't make much sense on Android but having an additional unused symbolic link is no real issue.

Meanwhile I also got the Android app to work by using the static lib instead of the shared one. Using a static lib allows Android build to strip away unused code, so maybe it can be even preferable to using a shared lib, especially since the full shared lib size is at around 10 MB (the full static lib would be 20 MB, but I think most of the time only a small portion of that is needed)

[martin-frbg]

Looking at the commit log, shared library support for Android was added only a few weeks ago (by xianyi on Apr 11 ), keeping the soname versioning was probably just an oversight there. I'll submit a fix over the weekend if nobody beats me to it.

[scorpeeon]

Yeah OK, great!

I also tested the shared library in the Android app, but apparently I get a crash on startup when loading the lib because of some missing symbol:

java.lang.UnsatisfiedLinkError: dlopen failed: cannot locate symbol "_gfortran_compare_string" referenced by "/data/app/com.my.package-1/lib/arm64/libopenblas.so"...

Not sure why it's missing from shared while it's OK with static.

[martin-frbg]

Do you have a libgfortran.so on the Android ? You may have to modify the linker flags so that the shared openblas gets built with its dependencies all linked statically (at which point there may be even less point in having a share libopenblas, except perhaps to be able to replace it as needed).

[brada4]

Where is that build log? It is apparent that your cross-compiler does not find (or have) target fortran library and cannot link resulting object. It whould be above linktest, something like gcc/gfortran/ld *.o -o libopenblas.so
What functions you intend to use from OpenBLAS? e.g if no lapack is needed dont build it...

[scorpeeon]

@martin-frbg Oh so it's also trying to load libgfortran.so dynamically? I'm pretty sure Android doesn't have that by default, so I think the only way to load it would be to also include that with the app - and I certainly see no such library packed in the app currently.

I'm not even sure if this problem should be handled when compiling openblas itself, or when compiling the native Android app (with cmake) and linking agains openblas...
But yeah, since the static lib seems to be working without issues, maybe we really shouldn't bother with the shared lib, and just use the static one. :)

Also I have a few additional questions (a bit offtopic, as they are more of general questions regarding Android)

I noticed that the openblas_config.h file inside the "include" folder differs in 32 and 64 bit builds as it uses different flags to indicate specific platforms like:
#define OPENBLAS_ARCH_ARM64 1
My question is what are the consequences of these configuration changes in this file, is there anz significance to it in the way the lib works? Or can I just use the same (32 bit) configuration file on all targets (including 64 bit, etc)? The reason why I ask is that in most Android native apps, the native headers are the same for all different architectures (arm, arm64, mips, x86, etc.).

The other question is: how can I verify that Lapack is included in the lib? Is there any method to query this, or maybe a specific method I can call that wouldn't work if Lapack was not included?

[brada4]

First the linker does not find android libgfortran.so, and does not generate android libopenblas.so
gfortran is not part of NDK, so you must include at least copying SO in your cmakes.

[scorpeeon]

@brada4 yeah that's what I figured.
I think I'll just use the static lib for now, until I can get things working.

I still don't quite get is even though I supposedly compiled OpenBLAS with Lapack, I'm not sure how can I use it or even verify that it's indeed compiled and working. The include folder obtained after running the install command only contains 3 files for blas, but none for lapacke. Are there some additional steps needed to properly compile/include Lapack - or I should just manually include/use the lapacke headers?

[martin-frbg]

Looks as if your "make install" assumed NO_LAPACKE=1 for some reason (perhaps it did not find a fortran compiler if you omitted the FC= and other options from the build run ?). Try using all the options from the "make ... all" on the "make ... install" as well.

[scorpeeon]

In the build run I did specify FC, the full command (for 64 bit) I used is:
make TARGET=ARMV8 BINARY=64 HOSTCC=gcc CC=aarch64-linux-android-gcc FC=aarch64-linux-android-gfortran all

Running install with all these options again though seems to do the trick as suggested, it creates 4 additional lapacke headers in the include folder. I suppose I should be able to use Lapack through these headers, gonna try that soon.
Thanks!

[brada4]

You can examine elf libraries with nm (most likely $ aarch64-linux-android-nm libopenblas.so)

[vargamarton]

@brada4 tried that to and definitely see a lot of methods that has Lapack included in their name, I guess that's a good sign :)

BTW the default android compiler uses clang since a while, and not gcc. I think openblas was compiled with gcc though. Should I set the Android compiler to also use gcc, or clang should be fine too?

[brada4]

clang is fine (think OSX), but you need fortran compiler for LAPACK.

[scorpeeon]

@brada4 what I meant was that I compiled OpenBLAS using gcc and I'm compiling the rest of the app with clang - should that be also OK?

Now I'm trying to use the lapacke functions through the header lapacke.h, using the static library, but when compiling it seems like I get very similar errors I got when I tried to use shared lib with armadillo issues: "undefined reference to `_gfortran_concat_string".

The full error message:

FAILED: cmd.exe /C "cd . && C:\Android\sdk\ndk-bundle\toolchains\llvm\prebuilt\windows-x86_64\bin\clang++.exe  --target=aarch64-none-linux-android --gcc-toolchain=C:/Android/sdk/ndk-bundle/toolchains/aarch64-linux-android-4.9/prebuilt/windows-x86_64 --sysroot=C:/Android/sdk/ndk-bundle/platforms/android-21/arch-arm64 -fPIC -g -DANDROID -ffunction-sections -funwind-tables -fstack-protector-strong -no-canonical-prefixes -Wa,--noexecstack -Wformat -Werror=format-security -std=c++11 -g -DANDROID -ffunction-sections -funwind-tables -fstack-protector-strong -no-canonical-prefixes -Wa,--noexecstack -Wformat -Werror=format-security -std=c++11 -std=c++11 -frtti -fexceptions -O0 -fno-limit-debug-info -O0 -fno-limit-debug-info  -Wl,--build-id -Wl,--warn-shared-textrel -Wl,--fatal-warnings -LC:/Android/sdk/ndk-bundle/sources/cxx-stl/llvm-libc++/libs/arm64-v8a -Wl,--no-undefined -Wl,-z,noexecstack -Qunused-arguments -Wl,-z,relro -Wl,-z,now -Wl,--build-id -Wl,--warn-shared-textrel -Wl,--fatal-warnings -LC:/Android/sdk/ndk-bundle/sources/cxx-stl/llvm-libc++/libs/arm64-v8a -Wl,--no-undefined -Wl,-z,noexecstack -Qunused-arguments -Wl,-z,relro -Wl,-z,now -shared -Wl,-soname,libnative-lib.so -o ..\..\..\..\build\intermediates\cmake\debug\obj\mips64\libnative-lib.so CMakeFiles/native-lib.dir/src/main/cpp/native-lib.cpp.o  -landroid -llog ../../../../build/intermediates/cmake/debug/obj/mips64/libarmadillo.so ../../../../../distribution/openblas/lib/arm64-v8a/libopenblas.a -lm "C:/Android/sdk/ndk-bundle/sources/cxx-stl/llvm-libc++/libs/arm64-v8a/libc++.so" && cd ."
../../../../../distribution/openblas/lib/arm64-v8a/libopenblas.a(dormlq.o): In function `dormlq_':
  dormlq.f:(.text+0x278): undefined reference to `_gfortran_concat_string'
  dormlq.f:(.text+0x5f0): undefined reference to `_gfortran_concat_string'
  ../../../../../distribution/openblas/lib/arm64-v8a/libopenblas.a(dormqr.o): In function `dormqr_':
  dormqr.f:(.text+0x29c): undefined reference to `_gfortran_concat_string'
  dormqr.f:(.text+0x5f8): undefined reference to `_gfortran_concat_string'
  ../../../../../distribution/openblas/lib/arm64-v8a/libopenblas.a(iparam2stage.o): In function `iparam2stage_':
  iparam2stage.F:(.text+0x28c): undefined reference to `_gfortran_compare_string'
  clang++.exe: error: linker command failed with exit code 1 (use -v to see invocation)
  ninja: build stopped: subcommand failed.

Any idea what could be wrong? Maybe the linking flags? I still haven't set the ldflags specified here:
https://github.com/xianyi/OpenBLAS/wiki/How-to-build-OpenBLAS-for-Android
(I'm currently trying to do that in cmake)

[martin-frbg]

Seems even your static libopenblas.a actually does not have libgfortran built into it after all. Try adding -static-libgfortran to the ldflags (and/or see if your toolchain path contains static libraries libgfortran.a and libquadmath.a that you could supply in place of the -lgfortran)

[scorpeeon]

@martin-frbg yeah, this is pretty strange. It looks like the toolchain I used does contain a libgfortran.a (the 32 bit version even contains multiple versions in different folders like armv7-a/hard, armv7-a/thumb) and if I include it, it compiles. Thanks for the tip!

[scorpeeon]

I tried one sample that used LAPACKE_dgels() function and it worked, but when I tried another, I got errors - it looks like there are still a few things it can't resolve:

undefined reference to 'lapack_make_complex_float(float, float)'
Any idea what can be the problem here?

[martin-frbg]

Is this from the testing suite supplied with lapacke ? The README in lapack-netlib/LAPACKE suggests you may be expected to supply a suitable implementation of this function yourself.

[scorpeeon]

@martin-frbg I found that sample code here: http://www.netlib.org/lapack/lapacke.html
But yeah, according to that README you are right indeed, and those functions need implementations provided. Other Lapacke functions I tested work fine, so it seems Lapacke is indeed properly included now. Thanks again!

[martin-frbg]

I have merged a change that should remove the soname versioning from the android builds, so I assume this issue can now be closed (?)

[scorpeeon]

@martin-frbg that's great, just tested it and can confirm that the SONAME is now generated correctly on Android by default.

Yeah so there were multiple problems I ran into mentioned in this thread, so the original was just that no shared lib was generated was that because in the Android guide I linked in the first post the "libs" parameter is used instead of "all" - so this was really the only original issue I had, and this was solved. (maybe the guide could be updated to reflect this.)
Then I had the SONAME issue which now seems to be solved thanks to you - great work BTW.
I also had an issue with install (missing lapacke headers) which can be just solved by using all the same options there.

BTW I still haven't tested the shared libs yet as the static was working fine - last time I checked there were the unresolved references because of missing libgfortran - maybe I can still test it by linking with the same libgfortran.a I did when using openblas as static library.

Some other issues I now encounter (sorry for bringing in more off-topic questions, I realize this is probably not the best place to do this) : in some combinations I get an error when linking that the openblas lib "uses VFP register arguments, output does not". I guess this is because the guide mentions you have to set the flags to use hard floating points. So both openblas and lapacke use hard floating points, so I should set all the flags in my programs to do the same also, right? I was trying the set the suggested flags in the C flags part of the Android build (gradle) file but it didn't help much so far. Maybe I'll try to do that in the cmake file. The strange thing is that when compiling for arm64, it works without setting any flags for hard floats - while 32 bit arm doesn't.

[martin-frbg]

Arm64 uses completely different assembly kernels (and I suspect there are no arm64 cpus that lack a floating point unit, making the choice between dedicated hardware and slower emulation unnecessary). From my (limited) understanding, one of the differences between the modes lies in how (in which registers) the values are passed to the functions so mixing modules that were built with different flags will only work as long as one set does not do any floating point computations at all. "Historically" the OpenBLAS codes for arm32 were written for designs with dedicated floating point hardware, xianyi has started only recently to modify them.
If you still encounter new and unrelated problems, I think it would indeed be better to create a new issue thread.

[scorpeeon]

@martin-frbg yes, you are right that this is a separate issue, I think this one could be closed.
I'll keep trying to make it compile on armv7 and if I can't figure it out, I'll open a new issue. Thanks for all the help!

[podilaaditya]

Building for Armv7 32 bit architecture with android tool chain is throwing error during make install
make TARGET=ARMV7 HOSTCC=gcc CC=arm-linux-androideabi-gcc NOFORTRAN=1 libs

make PREFIX=./install install

/Applications/Xcode.app/Contents/Developer/usr/bin/make -j 8 -f Makefile.install install
Generating openblas_config.h in ./install/include
Generating f77blas.h in ./install/include
Generating cblas.h in ./install/include
Copying LAPACKE header files to ./install/include
Copying the static library to ./install/lib
Copying the shared library to ./install/lib
install: libopenblas_armv7p-r0.2.20.dev.so: No such file or directory
make[1]: *** [install] Error 71
make: *** [install] Error 2

[martin-frbg]

See above - if you want the shared library (and you are using the current "develop" version from github, not some earlier release), do not specify the "libs" option on the make command line as that tells it to run only the subset of tasks necessary for building the static version.

[vinayak618]

did anyone try to load the shared library in android and got it working.?
@martin-frbg @scorpeeon, i built libopenblas.so using lapack support and while linking it to android application, i'm getting the error same as above: _cannot locate symbol "gfortran_concat_string" referenced by "libopenblas.so".
Any help would be appreciated.
Thank you

[brada4]

@vinayak618 please see #1871 , it looks like x86 and arm compilers were mixed together.

[ghost]

I tried one sample that used LAPACKE_dgels() function and it worked, but when I tried another, I got errors - it looks like there are still a few things it can't resolve:

undefined reference to 'lapack_make_complex_float(float, float)'
Any idea what can be the problem here?

Dear scorpeeon, I have the same problem in c++. Did you resolve it? What should I do?
Any function works, but I can not initialize the arrays using lapack_make_complex_double(double, double)

[brada4]

That function (actually macro) compensates absent complex support in C compilerwhen including lapacke.h. It is internal/private to the degree it is hidden when compiler supports complex numbers.

[ghost]

That function (actually macro) compensates absent complex support in C compilerwhen including lapacke.h. It is internal/private to the degree it is hidden when compiler supports complex numbers.

Uhhhh, You mean it is enough writing like this? for example: lapack_complex_double x = {1.0 , 2.0};

[brada4]

Use lapack_complex_double which is not hidden by macros. I think other's thread is not the right place to discuss unrelated API.