*GEMM_DEFAULT_UNROLL_* errors

[aytekinar]

Hi,

For my project, I am working on providing multi-architecture Docker images that are shipped with either reference LAPACK or OpenBLAS. The former seems to be working smoothly for amd64, arm32v6 and arm64v8 architectures. However, I have problems building the Docker image (with OpenBLAS) for the ARM architectures.

Below is the excerpt from my Dockerfile (see attached for the full version) that builds and installs OpenBLAS:

# Build and install reference LAPACK from source
RUN git clone https://github.com/xianyi/OpenBLAS /tmp/openblas
WORKDIR /tmp/openblas
RUN git checkout v0.3.3
RUN mkdir build
WORKDIR /tmp/openblas/build
RUN cmake -D CMAKE_INSTALL_PREFIX=/usr/local  \
          -D CMAKE_BUILD_TYPE=Release         \
          -D BUILD_SHARED_LIBS=ON             \
          -D BUILD_WITHOUT_LAPACK=OFF         \
          -D BUILD_WITHOUT_CBLAS=ON           \
          -D DYNAMIC_ARCH=ON                  \
          ../
RUN cmake --build .
RUN cmake --build . --target install

I am using an amd64 host with qemu-user-static handles registered via the Docker image. Basically, I am building the Docker images in an emulator, using alpine as the base image. When I run

docker run --rm --privileged multiarch/qemu-user-static:register --reset
docker build --tag my-image:openblas-arm32v6 .

I get the long list of error messages:

CMake Error at cmake/prebuild.cmake:229 (MESSAGE):
  Compiling getarch_2nd failed Change Dir:
  /tmp/openblas/build/getarch2_build/CMakeFiles/CMakeTmp

  

  Run Build Command:"/usr/bin/make" "cmTC_10582/fast"

  /usr/bin/make -f CMakeFiles/cmTC_10582.dir/build.make
  CMakeFiles/cmTC_10582.dir/build

  make[1]: Entering directory
  '/tmp/openblas/build/getarch2_build/CMakeFiles/CMakeTmp'

  Building C object CMakeFiles/cmTC_10582.dir/getarch_2nd.c.o

  /usr/bin/cc -DGEMM_MULTITHREAD_THRESHOLD=4 -DNO_PARALLEL_MAKE=0
  -I/tmp/openblas/build/getarch2_build -I"/tmp/openblas"
  -I"/tmp/openblas/build" -o CMakeFiles/cmTC_10582.dir/getarch_2nd.c.o -c
  /tmp/openblas/getarch_2nd.c

  /tmp/openblas/getarch_2nd.c: In function 'main':

  /tmp/openblas/getarch_2nd.c:12:35: error: 'SGEMM_DEFAULT_UNROLL_M'
  undeclared (first use in this function)

       printf("SGEMM_UNROLL_M=%d\n", SGEMM_DEFAULT_UNROLL_M);
                                     ^~~~~~~~~~~~~~~~~~~~~~

  /tmp/openblas/getarch_2nd.c:12:35: note: each undeclared identifier is
  reported only once for each function it appears in

  /tmp/openblas/getarch_2nd.c:13:35: error: 'SGEMM_DEFAULT_UNROLL_N'
  undeclared (first use in this function)

       printf("SGEMM_UNROLL_N=%d\n", SGEMM_DEFAULT_UNROLL_N);
                                     ^~~~~~~~~~~~~~~~~~~~~~

  /tmp/openblas/getarch_2nd.c:14:35: error: 'DGEMM_DEFAULT_UNROLL_M'
  undeclared (first use in this function)

       printf("DGEMM_UNROLL_M=%d\n", DGEMM_DEFAULT_UNROLL_M);
                                     ^~~~~~~~~~~~~~~~~~~~~~

  /tmp/openblas/getarch_2nd.c:15:35: error: 'DGEMM_DEFAULT_UNROLL_N'
  undeclared (first use in this function)

       printf("DGEMM_UNROLL_N=%d\n", DGEMM_DEFAULT_UNROLL_N);
                                     ^~~~~~~~~~~~~~~~~~~~~~

  /tmp/openblas/getarch_2nd.c:19:35: error: 'CGEMM_DEFAULT_UNROLL_M'
  undeclared (first use in this function)

       printf("CGEMM_UNROLL_M=%d\n", CGEMM_DEFAULT_UNROLL_M);
                                     ^~~~~~~~~~~~~~~~~~~~~~

  /tmp/openblas/getarch_2nd.c:20:35: error: 'CGEMM_DEFAULT_UNROLL_N'
  undeclared (first use in this function)

       printf("CGEMM_UNROLL_N=%d\n", CGEMM_DEFAULT_UNROLL_N);
                                     ^~~~~~~~~~~~~~~~~~~~~~

  /tmp/openblas/getarch_2nd.c:21:35: error: 'ZGEMM_DEFAULT_UNROLL_M'
  undeclared (first use in this function)

       printf("ZGEMM_UNROLL_M=%d\n", ZGEMM_DEFAULT_UNROLL_M);
                                     ^~~~~~~~~~~~~~~~~~~~~~

  /tmp/openblas/getarch_2nd.c:22:35: error: 'ZGEMM_DEFAULT_UNROLL_N'
  undeclared (first use in this function)

       printf("ZGEMM_UNROLL_N=%d\n", ZGEMM_DEFAULT_UNROLL_N);
                                     ^~~~~~~~~~~~~~~~~~~~~~

  make[1]: *** [CMakeFiles/cmTC_10582.dir/build.make:66:
  CMakeFiles/cmTC_10582.dir/getarch_2nd.c.o] Error 1

  make[1]: Leaving directory
  '/tmp/openblas/build/getarch2_build/CMakeFiles/CMakeTmp'

  make: *** [Makefile:126: cmTC_10582/fast] Error 2

Call Stack (most recent call first):
  cmake/system.cmake:122 (include)
  CMakeLists.txt:44 (include)

Is there an easy fix for this problem? Should I switch to using make-based installations? Is it possible to stay with cmake at all?

[brada4]

You have to specify TARGET= for ARM architectures.
ARMV6 and ARMV8 respectively.
There is ARM DYNAMIC_ARCH support in works in develop branch, nice if you can try it out on real system, but for 0.3.3 you have to stick to generic architectures targets.

[aytekinar]

Hi Andrew,

Thank you for your prompt response. Just to be clear, I have also tried DYNAMIC_ARCH=OFF with the installation, which resulted in the same set of error messages.

Do you think I should stick to the make-based installation, then? Do I simply issue:

make TARGET=ARMV6 # ARMV8 for ARM64
make PREFIX=/usr/local install

Does this method install the corresponding CMake config and target files?

[martin-frbg]

Both build systems should install all the required files, the cmake one happens to be newer and may
not support all the less common options from Makefile.rule yet.
For your project the cmake build should work fine, but you will need to add the -D TARGET=ARMV6 (or ARMV8 respectively) together with DYNAMIC_ARCH=OFF as the build system has no chance to guess the target cpu in your cross-compiling environment. (If this fails, we need to be told)

[brada4]

You need to repeat TARGET= (and preferably all other) options for install as they determine header files to be pulled along and name of main library to be copied (CMAKE build stores those options)

[aytekinar]

Hi again,

Changing the -D DYNAMIC_ARCH=ON \ line above with -D DYNAMIC_ARCH=OFF -D TARGET=ARMV6 seems to have has fixed the problem. At least, it is at 54% at the time of writing this post, and I do not expect any problems thereafter.

I don't know if we should close the issue, but as far as my problem is concerned, your suggestions have helped me solve it. I am still using the v0.3.3 tag , and I will re-open this issue if the build fails.

Thank you, both, for your prompt responses.

[brada4]

@aytekinar by chance do you know how to launch arm32/arm64 alpine on ubuntu14 provided by travis CI ?

[aytekinar]

@brada4, that will be my next step tonight and tomorrow. I will share the results with you here, under this thread, if you'd like.

I think you need sudo: required and services: docker in the YAML file. Then, you need to

1. mkdir $HOME/.docker
2. echo "{ \"experimental\": \"enabled\" }" > $HOME/.docker/config.json
3. sudo service docker restart (or, sudo systemctl restart docker)

(EDIT. You need the above only if you need experimental features of Docker such as, e.g., manifest)

in your before_install. Basically, for the emulated builds like I had, you can follow the README at pologrp/docker-ci. The README is for Alpine, even though the Dockerfile's are using debian:stretch-slim (I will modify the README, soon). build.sh in that repository is working, but maybe you need to prepend the

docker run --rm --privileged multiarch/qemu-user-static:register --reset

line with sudo. Then, everything should be fine.

I will be using such an approach to use these prebuilt multi-architecture Docker images to build and test my library for arm32, arm64 and amd64 architectures. But just a reminder --- Travis limits jobs to 50 minutes, which seems to be a bit too low for building OpenBLAS using qemu-user-static.

EDIT. I have created a dummy travis-arm-ci repository summarizing the above. It seems to be working. If you check the logs there, you will notice that the Docker images give the correct uname -m response.

[brada4]

It will be worth a heap to make a pull request regarding travis.yml in top level.
There are no arm build tests yet...

[martin-frbg]

Closing as the original build issue was resolved. Thanks a lot for the arm CI tests from PR#1864.

[aytekinar]

Thank you, for your help and fast response on the original issue. I am glad that it did help you with ARM builds :)