Build OpenBLAS 0.3.6 for iOS

[L1onKing]

Hello! I'm trying to build Open BLAS 0.3.5 version for iOS. Here's my shell script I'm using for building:

TOOLCHAIN_PATH=/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin
SYSROOT_PATH=/Applications/Xcode.app/Contents/Developer/Platforms/iPhoneOS.platform/Developer/SDKs/iPhoneOS.sdk
make TARGET=ARMV8 BINARY=64 HOSTCC=clang CC="$TOOLCHAIN_PATH/clang -isysroot $SYSROOT_PATH -arch arm64 -miphoneos-version-min=10.0 -O2" NOFORTRAN=1 libs

When I execute it, I'm happen to have this error:

clang: error: unknown argument: '-ru'
clang: error: no such file or directory: '../libopenblas_armv8p-r0.3.6.a'
clang: warning: no such sysroot directory: '/Applications/Xcode.app/Contents/Developer/Platforms/iPhoneOS.platform/Developer/SDKs/-ar' [-Wmissing-sysroot]

How can this be fixed? Thanks!

[martin-frbg]

Try adding "AR=ar" to your make command line (the "-ru" is an option for the ar program which creates the .a file). Not sure why this goes wrong though...

[L1onKing]

Sorry, my bad. I found my mistake. I forgot to add to c_check this line:

$cross_suffix = "/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/";

But new problem arised. Now I have those errors:

<instantiation>:7:2: note: while in macro instantiation
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/clang -isysroot /Applications/Xcode.app/Contents/Developer/Platforms/iPhoneOS.platform/Developer/SDKs/iPhoneOS.sdk -arch arm64 -miphoneos-version-min=10.0 -O2 -O2 -DMAX_STACK_ALLOC=2048 -Wall -DF_INTERFACE_GFORT -fPIC -DNO_LAPACK -DNO_LAPACKE -DSMP_SERVER -DNO_WARMUP -DMAX_CPU_NUMBER=8 -DMAX_PARALLEL_NUMBER=1 -DVERSION=\"0.3.6\" -march=armv8-a -DASMNAME=_sgemm_oncopy -DASMFNAME=_sgemm_oncopy_ -DNAME=sgemm_oncopy_ -DCNAME=sgemm_oncopy -DCHAR_NAME=\"sgemm_oncopy_\" -DCHAR_CNAME=\"sgemm_oncopy\" -DNO_AFFINITY -I.. -UDOUBLE  -UCOMPLEX -c -UDOUBLE -UCOMPLEX ../kernel/arm64/../generic/gemm_ncopy_4.c -o sgemm_oncopy.o
 KERNEL_F1
 ^
/var/folders/xt/7v6bxk2n19zft1y3yzymdc580000gq/T/nrm2-bd3346.s:146:2: note: while in macro instantiation
 KERNEL_F8
 ^
<instantiation>:14:22: error: unknown token in expression
KERNEL_F1_SCALE_GE_X_\@:
                     ^
<instantiation>:7:2: note: while in macro instantiation
 KERNEL_F1
 ^
/var/folders/xt/7v6bxk2n19zft1y3yzymdc580000gq/T/nrm2-bd3346.s:146:2: note: while in macro instantiation
 KERNEL_F8
 ^
<instantiation>:14:22: error: invalid operand
KERNEL_F1_SCALE_GE_X_\@:
                     ^
<instantiation>:7:2: note: while in macro instantiation
 KERNEL_F1
 ^
/var/folders/xt/7v6bxk2n19zft1y3yzymdc580000gq/T/nrm2-bd3346.s:146:2: note: while in macro instantiation
 KERNEL_F8
 ^
../kernel/arm64/nrm2.S:87:16: error: unknown token in expression
KERNEL_F1_NEXT_\@:
               ^
<instantiation>:7:2: note: while in macro instantiation
 KERNEL_F1
 ^
/var/folders/xt/7v6bxk2n19zft1y3yzymdc580000gq/T/nrm2-bd3346.s:146:2: note: while in macro instantiation
 KERNEL_F8
 ^
../kernel/arm64/nrm2.S:87:16: error: invalid operand
KERNEL_F1_NEXT_\@:
               ^
<instantiation>:7:2: note: while in macro instantiation
 KERNEL_F1
 ^
/var/folders/xt/7v6bxk2n19zft1y3yzymdc580000gq/T/nrm2-bd3346.s:146:2: note: while in macro instantiation
 KERNEL_F8
 ^
<instantiation>:4:21: error: unexpected token in argument list
 beq KERNEL_F1_NEXT_\@
                    ^
<instantiation>:8:2: note: while in macro instantiation
 KERNEL_F1
 ^
/var/folders/xt/7v6bxk2n19zft1y3yzymdc580000gq/T/nrm2-bd3346.s:146:2: note: while in macro instantiation
 KERNEL_F8
 ^
<instantiation>:7:27: error: unexpected token in argument list
 bge KERNEL_F1_SCALE_GE_X_\@
                          ^
<instantiation>:8:2: note: while in macro instantiation
 KERNEL_F1
 ^
/var/folders/xt/7v6bxk2n19zft1y3yzymdc580000gq/T/nrm2-bd3346.s:146:2: note: while in macro instantiation
 KERNEL_F8
 ^
<instantiation>:13:19: error: unexpected token in argument list
 b KERNEL_F1_NEXT_\@
                  ^
<instantiation>:8:2: note: while in macro instantiation
 KERNEL_F1
 ^
/var/folders/xt/7v6bxk2n19zft1y3yzymdc580000gq/T/nrm2-bd3346.s:146:2: note: while in macro instantiation
 KERNEL_F8
 ^
<instantiation>:14:22: error: unknown token in expression
KERNEL_F1_SCALE_GE_X_\@:
                     ^
<instantiation>:8:2: note: while in macro instantiation
 KERNEL_F1
 ^
/var/folders/xt/7v6bxk2n19zft1y3yzymdc580000gq/T/nrm2-bd3346.s:146:2: note: while in macro instantiation
 KERNEL_F8
 ^
<instantiation>:14:22: error: invalid operand
KERNEL_F1_SCALE_GE_X_\@:
                     ^
<instantiation>:8:2: note: while in macro instantiation
 KERNEL_F1
 ^
/var/folders/xt/7v6bxk2n19zft1y3yzymdc580000gq/T/nrm2-bd3346.s:146:2: note: while in macro instantiation
 KERNEL_F8
 ^
../kernel/arm64/nrm2.S:87:16: error: unknown token in expression
KERNEL_F1_NEXT_\@:
               ^
<instantiation>:8:2: note: while in macro instantiation
 KERNEL_F1
 ^
/var/folders/xt/7v6bxk2n19zft1y3yzymdc580000gq/T/nrm2-bd3346.s:146:2: note: while in macro instantiation
 KERNEL_F8
 ^
../kernel/arm64/nrm2.S:87:16: error: invalid operand
KERNEL_F1_NEXT_\@:
               ^
<instantiation>:8:2: note: while in macro instantiation
 KERNEL_F1
 ^
/var/folders/xt/7v6bxk2n19zft1y3yzymdc580000gq/T/nrm2-bd3346.s:146:2: note: while in macro instantiation
 KERNEL_F8
 ^
<instantiation>:4:21: error: unexpected token in argument list
 beq KERNEL_F1_NEXT_\@
                    ^
/var/folders/xt/7v6bxk2n19zft1y3yzymdc580000gq/T/nrm2-bd3346.s:159:2: note: while in macro instantiation
 KERNEL_F1
 ^
<instantiation>:7:27: error: unexpected token in argument list
 bge KERNEL_F1_SCALE_GE_X_\@
                          ^
/var/folders/xt/7v6bxk2n19zft1y3yzymdc580000gq/T/nrm2-bd3346.s:159:2: note: while in macro instantiation
 KERNEL_F1
 ^
<instantiation>:13:19: error: unexpected token in argument list
 b KERNEL_F1_NEXT_\@
                  ^
/var/folders/xt/7v6bxk2n19zft1y3yzymdc580000gq/T/nrm2-bd3346.s:159:2: note: while in macro instantiation
 KERNEL_F1
 ^
<instantiation>:14:22: error: unknown token in expression
KERNEL_F1_SCALE_GE_X_\@:
                     ^
/var/folders/xt/7v6bxk2n19zft1y3yzymdc580000gq/T/nrm2-bd3346.s:159:2: note: while in macro instantiation
 KERNEL_F1
 ^
<instantiation>:14:22: error: invalid operand
KERNEL_F1_SCALE_GE_X_\@:
                     ^
/var/folders/xt/7v6bxk2n19zft1y3yzymdc580000gq/T/nrm2-bd3346.s:159:2: note: while in macro instantiation
 KERNEL_F1
 ^
../kernel/arm64/nrm2.S:87:16: error: unknown token in expression
KERNEL_F1_NEXT_\@:
               ^
/var/folders/xt/7v6bxk2n19zft1y3yzymdc580000gq/T/nrm2-bd3346.s:159:2: note: while in macro instantiation
 KERNEL_F1
 ^
../kernel/arm64/nrm2.S:87:16: error: invalid operand
KERNEL_F1_NEXT_\@:
               ^
/var/folders/xt/7v6bxk2n19zft1y3yzymdc580000gq/T/nrm2-bd3346.s:159:2: note: while in macro instantiation
 KERNEL_F1

That's a part of it, but it's basically all the same. I think I saw a fix for that somewhere in Issues list

[martin-frbg]

Pretty sure it was in the closed ticket #1531 you commented on earlier...though I think I had put in a fix for this bac then (maybe it was affected by later changes)

[L1onKing]

#569 I found a fix for that here. But now when I try to build my XCode project with new library, I have those errors:

ld: warning: directory not found for option '-L/Users/user/Documents/Work/AlgofaceWork/algoface-ios-209-landmarks-tracker/Algoface-Landmarks-Tracker/tracker/3rdparty/eigen/OpenBLAS_iOS_0.2.21'

Undefined symbols for architecture arm64:
  "_sdot_k", referenced from:
      _strmv_TLN in libopenblas.a(strmv_TLN.o)
      _strmv_TLU in libopenblas.a(strmv_TLU.o)
      _strmv_TUN in libopenblas.a(strmv_TUN.o)
      _strmv_TUU in libopenblas.a(strmv_TUU.o)
      _trmv_kernel in libopenblas.a(strmv_thread_TLN.o)
      _trmv_kernel in libopenblas.a(strmv_thread_TLU.o)
      _trmv_kernel in libopenblas.a(strmv_thread_TUN.o)
      ...
  "_scopy_k", referenced from:
      _strmv_NLN in libopenblas.a(strmv_NLN.o)
      _strmv_NLU in libopenblas.a(strmv_NLU.o)
      _strmv_NUN in libopenblas.a(strmv_NUN.o)
      _strmv_NUU in libopenblas.a(strmv_NUU.o)
      _strmv_TLN in libopenblas.a(strmv_TLN.o)
      _strmv_TLU in libopenblas.a(strmv_TLU.o)
      _strmv_TUN in libopenblas.a(strmv_TUN.o)
      ...
  "_strmm_kernel_RN", referenced from:
      _strmm_RNUN in libopenblas.a(strmm_RNUN.o)
      _strmm_RNUU in libopenblas.a(strmm_RNUU.o)
      _strmm_RTLN in libopenblas.a(strmm_RTLN.o)
      _strmm_RTLU in libopenblas.a(strmm_RTLU.o)
  "_strmm_kernel_LN", referenced from:
      _strmm_LNUN in libopenblas.a(strmm_LNUN.o)
      _strmm_LNUU in libopenblas.a(strmm_LNUU.o)
      _strmm_LTLN in libopenblas.a(strmm_LTLN.o)
      _strmm_LTLU in libopenblas.a(strmm_LTLU.o)
  "_strmm_kernel_LT", referenced from:
      _strmm_LNLN in libopenblas.a(strmm_LNLN.o)
      _strmm_LNLU in libopenblas.a(strmm_LNLU.o)
      _strmm_LTUN in libopenblas.a(strmm_LTUN.o)
      _strmm_LTUU in libopenblas.a(strmm_LTUU.o)
  "_sscal_k", referenced from:
      _sgemv_ in libopenblas.a(sgemv.o)
      _trmv_kernel in libopenblas.a(strmv_thread_NLN.o)
      _trmv_kernel in libopenblas.a(strmv_thread_NLU.o)
      _trmv_kernel in libopenblas.a(strmv_thread_NUN.o)
      _trmv_kernel in libopenblas.a(strmv_thread_NUU.o)
      _trmv_kernel in libopenblas.a(strmv_thread_TLN.o)
      _trmv_kernel in libopenblas.a(strmv_thread_TLU.o)
      ...
  "_sgemm_kernel", referenced from:
      _sgemm_nn in libopenblas.a(sgemm_nn.o)
      _sgemm_nt in libopenblas.a(sgemm_nt.o)
      _inner_thread in libopenblas.a(sgemm_thread_nn.o)
      _inner_thread in libopenblas.a(sgemm_thread_nt.o)
      _inner_thread in libopenblas.a(sgemm_thread_tn.o)
      _inner_thread in libopenblas.a(sgemm_thread_tt.o)
      _sgemm_tn in libopenblas.a(sgemm_tn.o)
      ...
  "_strmm_kernel_RT", referenced from:
      _strmm_RNLN in libopenblas.a(strmm_RNLN.o)
      _strmm_RNLU in libopenblas.a(strmm_RNLU.o)
      _strmm_RTUN in libopenblas.a(strmm_RTUN.o)
      _strmm_RTUU in libopenblas.a(strmm_RTUU.o)
  "_sgemv_n", referenced from:
      l___const.sgemv_.gemv in libopenblas.a(sgemv.o)
      _gemv_kernel in libopenblas.a(sgemv_thread_n.o)
      _strmv_NLN in libopenblas.a(strmv_NLN.o)
      _strmv_NLU in libopenblas.a(strmv_NLU.o)
      _strmv_NUN in libopenblas.a(strmv_NUN.o)
      _strmv_NUU in libopenblas.a(strmv_NUU.o)
      _trmv_kernel in libopenblas.a(strmv_thread_NLN.o)
      ...
  "_sgemv_t", referenced from:
      l___const.sgemv_.gemv in libopenblas.a(sgemv.o)
      _gemv_kernel in libopenblas.a(sgemv_thread_t.o)
      _strmv_TLN in libopenblas.a(strmv_TLN.o)
      _strmv_TLU in libopenblas.a(strmv_TLU.o)
      _strmv_TUN in libopenblas.a(strmv_TUN.o)
      _strmv_TUU in libopenblas.a(strmv_TUU.o)
      _trmv_kernel in libopenblas.a(strmv_thread_TLN.o)
      ...
     (maybe you meant: _sgemv_thread_n, _sgemv_thread_t )
  "_saxpy_k", referenced from:
      _saxpy_ in libopenblas.a(saxpy.o)
      _strmv_NLN in libopenblas.a(strmv_NLN.o)
      _strmv_NLU in libopenblas.a(strmv_NLU.o)
      _strmv_NUN in libopenblas.a(strmv_NUN.o)
      _strmv_NUU in libopenblas.a(strmv_NUU.o)
      _strmv_thread_NLN in libopenblas.a(strmv_thread_NLN.o)
      _trmv_kernel in libopenblas.a(strmv_thread_NLN.o)
      ...
ld: symbol(s) not found for architecture arm64
clang: error: linker command failed with exit code 1 (use -v to see invocation)

Can that be an issue that XCode can't compile assembly files?

I think it's pretty significant moment, because I forced XCode to build C versions of functions previously and the library is not fast as I think it should be

[martin-frbg]

Hard to tell from the partial log (there should have been earlier errors from trying to compile the respective OpenBLAS sources), but the previous reports (e.g. #1531) made it look like xcode only has problems with some of the assembly files. On the other hand, some enhancements have been made since #1531 was last updated (in particular, #1821 brought in more assembly kernels including one for SAXPY) so the situation may have become worse on IOS since it was last tried.

[L1onKing]

@martin-frbg are there any suggestions of what should I do? Maybe I can download a clang compiler and compile assembly files with those?

I understand that the best solution would be to optimize assembly files just so XCode compiler could compile them, but unfortunately I don't have such experience.

[L1onKing]

I thought that I might try older version of XCode. But I highly doubt the older version of compiler will do a better job

[martin-frbg]

No suggestion at the moment unfortunately - I am still trying to set up an equivalent cross-build on the Travis service for experimenting. As far as I know, the standard compiler in xcode is clang, so if anything one would have to try building with gcc (and the assembler from gnu binutils).
Note the whole issue could be something silly - actually i do not know if there is any special meaning to the use of the @ sign as the last character in a label (which seems to be what xcode hates most about our code) or if this was just one contributor's coding style.

[L1onKing]

As far as I'm aware, that XCode's clang is somehow optimized/changed. I saw few articles where developers prefer to use manually downloaded clang, like here for instance: https://embeddedartistry.com/blog/2017/2/20/installing-clangllvm-on-osx

So I will try to intall clang via Homebrew and compile with it. Let's see what happens

[martin-frbg]

Probably easier to just strip all the "@" from the offending .S files and see what blows up.

[L1onKing]

@martin-frbg I noticed that all @ files are used along with \ dash. Does it mean I should strip the dash as well?

Sorry, I know almost nothing about Assembler :(

[L1onKing]

I removed \@ stuff and now I have such errors:

<instantiation>:14:1: error: invalid symbol redefinition
KERNEL_F1_SCALE_GE_X_:
^
../kernel/arm64/nrm2.S:87:1: error: invalid symbol redefinition
KERNEL_F1_NEXT_:
^
<instantiation>:14:1: error: invalid symbol redefinition
KERNEL_F1_SCALE_GE_X_:
^
../kernel/arm64/nrm2.S:87:1: error: invalid symbol redefinition
KERNEL_F1_NEXT_:
^
<instantiation>:14:1: error: invalid symbol redefinition
KERNEL_F1_SCALE_GE_X_:
^
../kernel/arm64/nrm2.S:87:1: error: invalid symbol redefinition
KERNEL_F1_NEXT_:
^
<instantiation>:14:1: error: invalid symbol redefinition
KERNEL_F1_SCALE_GE_X_:
^
../kernel/arm64/nrm2.S:87:1: error: invalid symbol redefinition
KERNEL_F1_NEXT_:
^
/usr/local/opt/llvm/bin/clang -isysroot /Applications/Xcode.app/Contents/Developer/Platforms/iPhoneOS.platform/Developer/SDKs/iPhoneOS.sdk -arch arm64 -miphoneos-version-min=8.0 -O2 -c -O2 -DMAX_STACK_ALLOC=2048 -Wall -DF_INTERFACE_GFORT -fPIC -DNO_LAPACK -DNO_LAPACKE -DSMP_SERVER -DNO_WARMUP -DMAX_CPU_NUMBER=8 -DMAX_PARALLEL_NUMBER=1 -DVERSION=\"0.3.5\" -march=armv8-a -DASMNAME=_ssymv_L -DASMFNAME=_ssymv_L_ -DNAME=ssymv_L_ -DCNAME=ssymv_L -DCHAR_NAME=\"ssymv_L_\" -DCHAR_CNAME=\"ssymv_L\" -DNO_AFFINITY -I.. -UDOUBLE  -UCOMPLEX -UCOMPLEX -UDOUBLE -DLOWER ../kernel/arm64/../generic/symv_k.c -o ssymv_L.o
<instantiation>:14:1: error: invalid symbol redefinition
KERNEL_F1_SCALE_GE_X_:
^
../kernel/arm64/nrm2.S:87:1: error: invalid symbol redefinition
KERNEL_F1_NEXT_:
^
<instantiation>:14:1: error: invalid symbol redefinition
KERNEL_F1_SCALE_GE_X_:
^
../kernel/arm64/nrm2.S:87:1: /usr/local/opt/llvm/bin/clang -isysroot /Applications/Xcode.app/Contents/Developer/Platforms/iPhoneOS.platform/Developer/SDKs/iPhoneOS.sdk -arch arm64 -miphoneos-version-min=8.0 -O2 -c -O2 -DMAX_STACK_ALLOC=2048 -Wall -DF_INTERFACE_GFORT -fPIC -DNO_LAPACK -DNO_LAPACKE -DSMP_SERVER -DNO_WARMUP -DMAX_CPU_NUMBER=8 -DMAX_PARALLEL_NUMBER=1 -DVERSION=\"0.3.5\" -march=armv8-a -DASMNAME=_sger_k -DASMFNAME=_sger_k_ -DNAME=sger_k_ -DCNAME=sger_k -DCHAR_NAME=\"sger_k_\" -DCHAR_CNAME=\"sger_k\" -DNO_AFFINITY -I.. -UDOUBLE  -UCOMPLEX -UDOUBLE ../kernel/arm64/../generic/ger.c -o sger_k.o
error: invalid symbol redefinition
KERNEL_F1_NEXT_:
^
<instantiation>:14:1: error: invalid symbol redefinition
KERNEL_F1_SCALE_GE_X_:
^
../kernel/arm64/nrm2.S:87:1: error: invalid symbol redefinition
KERNEL_F1_NEXT_:
^
<instantiation>:14:1: error: invalid symbol redefinition
KERNEL_F1_SCALE_GE_X_:
^
../kernel/arm64/nrm2.S:87:1: error: invalid symbol redefinition
KERNEL_F1_NEXT_:
^
/usr/local/opt/llvm/bin/clang -isysroot /Applications/Xcode.app/Contents/Developer/Platforms/iPhoneOS.platform/Developer/SDKs/iPhoneOS.sdk -arch arm64 -miphoneos-version-min=8.0 -O2 -O2 -DMAX_STACK_ALLOC=2048 -Wall -DF_INTERFACE_GFORT -fPIC -DNO_LAPACK -DNO_LAPACKE -DSMP_SERVER -DNO_WARMUP -DMAX_CPU_NUMBER=8 -DMAX_PARALLEL_NUMBER=1 -DVERSION=\"0.3.5\" -march=armv8-a -DASMNAME=_sgemm_kernel -DASMFNAME=_sgemm_kernel_ -DNAME=sgemm_kernel_ -DCNAME=sgemm_kernel -DCHAR_NAME=\"sgemm_kernel_\" -DCHAR_CNAME=\"sgemm_kernel\" -DNO_AFFINITY -I.. -UDOUBLE  -UCOMPLEX -c -UDOUBLE -UCOMPLEX ../kernel/arm64/sgemm_kernel_16x4.S -o sgemm_kernel.o
make[1]: *** [snrm2_k.o] Error 1
make[1]: *** Waiting for unfinished jobs....
make: *** [libs] Error 1
MacBook-Pro-Mojo:OpenBLAS-release_0.3.5 user$ 

Do you have any comments on how it can be fixed in Assembly?

[martin-frbg]

Sorry, probably that was just a silly suggestion. Not clear to me where the "redefinition" happens (unless the labels are local to the file when the @ is present, and become global ones when not).

[martin-frbg]

Found a possible explanation (though for an entirely different compiler/assembler):

If a macro containing a label is used more than once, unique label names need to be generated to avoid multiple definition errors. A backslash followed by an at sign (@) appearing in a label within a macro expansion is replaced with a macro expansion serial number.

if this is widely used, wonder why the xcode assembler does not understand it (and what its Apple equivalent would be)

[L1onKing]

Just tried to build with gcc instead of clang. And the same mistake :

../kernel/arm64/nrm2.S:87:16: error: unknown token in expression
KERNEL_F1_NEXT_\@:
               ^
/var/folders/xt/7v6bxk2n19zft1y3yzymdc580000gq/T/nrm2-e25b4c.s:159:2: note: while in macro instantiation
 KERNEL_F1
 ^
../kernel/arm64/nrm2.S:87:16: error: invalid operand
KERNEL_F1_NEXT_\@:

A lot of these errors. I won't provide the full text, it was in issues list too many times now. I'm trying to figure out what this @ do in Assembly, but no luck so far.