USE_TLS=1 causes segfault for 0.3.20

[arungiridhar]

Building OpenBlas with USE_TLS=1 causes a segfault when used with matrices larger than 64x64 and using multiple threads.

Downstream crash in GNU Octave where this bug was noticed and isolated: https://savannah.gnu.org/bugs/index.php?62144

OpenBlas 0.3.20 was installed from here https://aur.archlinux.org/packages/openblas-lapack/ and USE_TLS=1 was switched on by default as of package version "openblas-lapack 0.3.20-1"

Workaround: change it to USE_TLS=0 when installing OpenBlas. That prevents a crash when exiting GNU Octave.

[brada4]

Here, upstream, the default is USE_TLS=0, please report the bug to archlinux to revert the parameter.

[martin-frbg]

a rerun of #3503 ? this was supposed to be fixed by #3504 ...

[martin-frbg]

So far not reproduced on Ryzen5-4600 (12 threads) with current develop, are you sure your octave binary picks up the 0.3.20 OpenBLAS at runtime ?

[arungiridhar]

There is only one OpenBLAS version on my system at any given time and currently it is 0.3.20.

OpenBLAS options, all default options from the distro:

_config="FC=gfortran USE_OPENMP=1 USE_THREAD=1 \
  USE_TLS=1 \
  NO_LAPACK=0 BUILD_LAPACK_DEPRECATED=1 \
  MAJOR_VERSION=${_lapackver:0:1} NO_STATIC=1 "

Result is a crash on exit:

octave:1> n = 65; A = ones(n,n); tic; B = A^2; toc, assert (all (B(:) == n))
Elapsed time is 0.000589848 seconds.
octave:2> exit
fatal: caught signal Segmentation fault -- stopping myself...
Segmentation fault (core dumped)

Change USE_TLS to 0, leave everything else the same:

_config="FC=gfortran USE_OPENMP=1 USE_THREAD=1 \
  USE_TLS=0 \
  NO_LAPACK=0 BUILD_LAPACK_DEPRECATED=1 \
  MAJOR_VERSION=${_lapackver:0:1} NO_STATIC=1 "

Result is normal exit:

octave:1> n = 65; A = ones(n,n); tic; B = A^2; toc, assert (all (B(:) == n))
Elapsed time is 0.000334024 seconds.
octave:2> exit
### normal exit, no crash

The only difference between the two is USE_TLS.

Octave was unchanged in both cases:

octave:1> version
ans = 8.0.0
octave:2> version -blas
ans = OpenBLAS (config: OpenBLAS 0.3.20 NO_AFFINITY USE_OPENMP ZEN MAX_THREADS=12)

and OMP_NUM_THREADS was set to 6 in all use cases, being the number of physical cores.

[martin-frbg]

Alright, thanks. Which compiler and glibc version then ? (Both the build options and hardware seem to match mine, yet I can your reproducer in octave just fine with n>64. I do notice however that my octave version is only 5.2.0)

[martin-frbg]

Also slightly confused by your octave reporting version "8.0.0" when latest stable is apparently still 6.4.0 and the latest tag in their mercurial vcs appears to be release candidate 7.0.90

[arungiridhar]

Octave 8.0.0 is the version built from dev sources, since I contribute code to Octave.

But I could replicate it with Octave 5.2.0 as well. This is with USE_TLS=1:

octave:1> version
ans = 5.2.0
octave:2> version -blas
ans = OpenBLAS (config: OpenBLAS 0.3.20 NO_AFFINITY USE_OPENMP ZEN MAX_THREADS=12)
octave:3> n = 65; A = ones(n,n); tic; B = A^2; toc, assert (all (B(:) == n))
Elapsed time is 0.00657392 seconds.
octave:4> exit
fatal: caught signal Segmentation fault -- stopping myself...
Segmentation fault (core dumped)

This is with USE_TLS=0, no crash on exit:

octave:1> version
ans = 5.2.0
octave:2> version -blas
ans = OpenBLAS (config: OpenBLAS 0.3.20 NO_AFFINITY USE_OPENMP ZEN MAX_THREADS=12)
octave:3> n = 65; A = ones(n,n); tic; B = A^2; toc, assert (all (B(:) == n))
Elapsed time is 0.000365973 seconds.
octave:4> exit

In all cases, OpenBLAS was built and installed with gcc 11.2.0 and gfortran 11.2.0 (system defaults).

The glibc version is 2.35 (system default).

All versions of Octave used clang++ 13.0.1 for C++ and gfortran 11.2.0 for Fortran.

Update: Just to check if this was a Clang/GCC problem, I rebuilt Octave 8.0.0 with GCC and DID NOT get the error with USE_TLS=1 or USE_TLS=0. I also verified the same with Octave 5.2.0 built with GCC.

Summary so far: For the segfault on exit, Octave (any version) must be built with Clang and OpenBLAS must be built with GCC and the library must set USE_TLS=1. If any of those conditions is not met, there is no segfault. Tested with GCC 11.2.0 and Clang 13.0.1.

[martin-frbg]

Can you check if there are two OpenMP libraries linked to Octave in that case (I think we saw some bad interactions between gcc's libgomp and llvm's libomp in the past already) ?

[brada4]

(in /proc/12345/maps)

[arungiridhar]

Octave built with Clang uses both:

7f331be29000-7f331be33000 r--p 00000000 08:02 39857844                   /usr/lib/libgomp.so.1.0.0
7f331be33000-7f331be60000 r-xp 0000a000 08:02 39857844                   /usr/lib/libgomp.so.1.0.0
7f331be60000-7f331be6c000 r--p 00037000 08:02 39857844                   /usr/lib/libgomp.so.1.0.0
7f331be6c000-7f331be6d000 r--p 00042000 08:02 39857844                   /usr/lib/libgomp.so.1.0.0
7f331be6d000-7f331be6e000 rw-p 00043000 08:02 39857844                   /usr/lib/libgomp.so.1.0.0
7f331ce31000-7f331ce4f000 r--p 00000000 08:02 39863722                   /usr/lib/libomp.so
7f331ce4f000-7f331cef2000 r-xp 0001e000 08:02 39863722                   /usr/lib/libomp.so
7f331cef2000-7f331cf1b000 r--p 000c1000 08:02 39863722                   /usr/lib/libomp.so
7f331cf1b000-7f331cf1c000 r--p 000e9000 08:02 39863722                   /usr/lib/libomp.so
7f331cf1c000-7f331cf22000 rw-p 000ea000 08:02 39863722                   /usr/lib/libomp.so

Octave built with GCC uses only gomp:

7fb415e20000-7fb415e2a000 r--p 00000000 08:02 39857844                   /usr/lib/libgomp.so.1.0.0
7fb415e2a000-7fb415e57000 r-xp 0000a000 08:02 39857844                   /usr/lib/libgomp.so.1.0.0
7fb415e57000-7fb415e63000 r--p 00037000 08:02 39857844                   /usr/lib/libgomp.so.1.0.0
7fb415e63000-7fb415e64000 r--p 00042000 08:02 39857844                   /usr/lib/libgomp.so.1.0.0
7fb415e64000-7fb415e65000 rw-p 00043000 08:02 39857844                   /usr/lib/libgomp.so.1.0.0

[brada4]

Some franken-blas that could work:

• build openblas with NOFORTRAN=1 and clang, complement with netlib lapack and run octave on top of that
• build openblas with USE_THREADS=1 USE_LOCKING=1 and it works with omp and gomp
• if arch packages use mix of clang and gcc in dependencies it is best to not USE_TLS in blas....

This is out of reach of OpenBLAS, best can be said here - "use same (version of) toolkit everywhere", OpenMP is headers + library, abstracting from backend, mixing implementations with same function names "yields unexpected results"

e.g:

• you acquire mutex in one implementation
• you do the exclusive code
• you unlock in other implementation

What could happen:

• lock of first impelemntation stays forever, next locking attempt never succeeds
• unlock fails with some ERR because lock handle is moot
• call prototypes differ and you get something like sigsegv on either call <--- you are here likely

[arungiridhar]

That makes sense. For me the easiest solution is to set USE_TLS=0, since that mode is more widely tested than USE_TLS=1. I need to use both Clang and GCC for different software projects, so doing a toolchain monoculture is not possible for me right now.

It's OK to close this report since there is nothing pending.

Out of curiosity, why does the OMP/GOMP clash only get exposed with USE_TLS=1? Wouldn't that clash collide with more things across the board?

[brada4]

Layout of actual TLS storage differs between implementations?
You can close this, there is nothing we can fix or improve. If you think arch is affected in common scenarios pls tell them.

[arungiridhar]

Closing issue since nothing can be done realistically other than setting USE_TLS=0. I will let the Arch AUR package maintainer know so they can make an informed choice.