Segmentation fault using serial OpenBLAS with OpenMP on Windows

[hjbreg]

Updated test results (OpenBLAS 0.3.3)

Windows 64-bit

gcc version 4.9.3 20150626 (Fedora MinGW 4.9.3-1.el7)

• USE_THREAD=0 NUM_THREADS=1
  segmentation fault

• USE_THREAD=0 NUM_THREADS=16
  incorrect result

• USE_THREAD=1 NUM_THREADS=2 USE_OPENMP=1 NUM_PARALEL=16
  incorrect result

• USE_THREAD=1 NUM_THREADS=2 openblas_set_num_threads(1)
  incorrect result

Linux 64-bit

gcc version 4.4.7 20120313 (Red Hat 4.4.7-4)

• USE_THREAD=0 NUM_THREADS=1
  too many memory regions

• USE_THREAD=0 NUM_THREADS=16
  OK

---

Segmentation fault if OMP_NUM_THREADS > 4, but linking to netlib-lapack is ok, so I think this may be OpenBLAS side problem. I also tested it on Linux, no segmentation fault.

GCC: x86_64-w64-mingw32-gcc 4.9.3

Single threaded OpenBLAS is built

USE_THREAD=0 DYNAMIC_ARCH=1 DYNAMIC_OLDER=0 NO_CBLAS=1 NO_LAPACKE=1 NO_SHARED=1

Test code: segfault-win.cpp

#include <cstdlib>
#include <vector>
#include <iostream>

// g++ segfault-win.cpp -lopenblas -lgfortran -lquadmath -fopenmp

#define bint int

extern "C"
void dgels_(char *trans, bint *m, bint *n, bint *nrhs, double *a, bint *lda, double *b, bint *ldb,
            double *work, bint *lwork, bint *info);

extern "C" void openblas_set_num_threads(int);
extern "C" int openblas_get_num_threads();
extern "C" int omp_get_max_threads();

int C_dgels(char trans, bint m, bint n, bint nrhs, double *a, bint lda, double *b, bint ldb)
{
    bint info = 0;
    double wkopt;
    bint lwork = -1;
    dgels_(&trans, &m, &n, &nrhs, a, &lda, b, &ldb, &wkopt, &lwork, &info);
    if (info == 0) {
        lwork = static_cast<bint>( wkopt );
        double *work = new double[lwork];
        dgels_(&trans, &m, &n, &nrhs, a, &lda, b, &ldb, work, &lwork, &info);
        delete[] work;
    }
    return info;
}

int main()
{
    // USE_THREAD=1 NUM_THREADS=2
    // openblas_set_num_threads(1);

    int n = 300;
    int q = 5;
    int r = 1000;

    std::vector<double> x, y;
    for (int i = 0; i < n; ++i) {
        y.push_back( static_cast<double>(std::rand()) / RAND_MAX );
        for (int j = 0; j < q; ++j)
            x.push_back( static_cast<double>(std::rand()) / RAND_MAX );
    }

    std::cerr << "omp_get_max_threads: " << omp_get_max_threads() << "\n";
    std::cerr << "openblas_get_num_threads: " << openblas_get_num_threads() << "\n";

    std::vector< std::vector<double> > z(r);

    #pragma omp parallel for
    for (int i = 0; i < r; ++i) {
        std::vector<double> xx = x;
        std::vector<double> yy = y;
        C_dgels('N', n, q, 1, xx.data(), n, yy.data(), n);
        z[i] = yy;
    }

    for (int i = 1; i < r; ++i)
        std::cerr << (z[i] == z[0]);
    std::cerr << "\n";

    return 0;
}

[brianborchers]

Does the same code work correctly with other versions of BLAS (or OpenBLAS compiled with different options)?

[hjbreg]

Does the same code work correctly with other versions of BLAS (or OpenBLAS compiled with different options)?

Yes, the code works correctly with netlib-lapack/blas.

[brada4]

Can you get backtrace with e.g. x64dbg?
Is OpenBLAS also compiled in unorthodox way with USE_OPENMP=1 requiring dozen of redist packages from gcc?

[hjbreg]

Sorry, I am not familiar with backtrace debug.
However, I rebuilt single threaded OpenBLAS with NUM_THREADS=24 (default to 4 in previous build) in addition to USE_THREAD=0, and interestingly segmentation fault is gone, but the results are incorrect for OMP_NUM_THREADS>1.
I am confused that why is NUM_THREADS=24 option used when USE_THREAD=0 is set.

[martin-frbg]

See Usage.md, for "historical reasons" NUM_THREADS also sizes an internal buffer. Do you get correct results when you compile OpenBLAS (which version, by the way ?) with USE_OPENMP=1 ?

[hjbreg]

@martin-frbg I think USE_OPENMP=1 can not be used simultaneously with USE_THREAD=0, furthermore only serial version of OpenBLAS (0.3.3) is considered here.
So NUM_THREADS is related to internal buffer, and I wonder whether the internal buffer is thread safety.

[hjbreg]

I will try to build threaded OpenBLAS and use openblas_set_num_threads(1) to get serial OpenBLAS.

[martin-frbg]

Did not see the requirement that OpenBLAS has to be single-threaded. If the error is not reproducible on anything other than Windows it could also be a mingw miscompilation. I will check (later) if valgrind/helgrind on Linux makes any complaints.

[hjbreg]

No segfault but incorrect results with OpenBLAS using USE_THREAD=1 USE_OPENMP=1.
So OpenBLAS will not work with MinGW and OpenMP on Windows.
As the same code is fine on Linux, this may be also a MinGW side problem.

[brada4]

To get backtrace in a debugger (x64dbg is graphical and understands both windows and gcc debug symbols and is free, alternatives would be IDApro, visual studio, ollydbg, each lacking functionality in some way) you run crashing program inside debugger and it catches the crash.

Then you have various options (which open new MDI windows) like backtrace, registers, stack content, disassembly of code around (crashed) EIP and probably in other code points mentioned in traces. Just be courageous and creative around these lines. Nobody expects perfect picture at the first try.

[hjbreg]

@martin-frbg I have updated test results, including Linux, please see the first post for details. I have read the USAGE.md. I still do not understand why single-threaded OpenBLAS may have problem in multiple threads program.

Despite its name, and due to the use of memory buffers in functions like SGEMM, the setting of NUM_THREADS can be relevant even for a single-threaded build of OpenBLAS, if such functions get called by multiple threads of a program that uses OpenBLAS. In some cases, the affected code may simply crash or throw a segmentation fault without displaying the above warning first.

@brada4 Many thanks, I will try later.

[brada4]

#1844 approaches same class of buffers but from different side
Call graph of dgels probably yields more candidate broken functions used in Windows 2-4 samples

[martin-frbg]

@hjbreg could you retry with a build after setting USE_TLS to 0 in Makefile.rule please ? 0.3.3 was meant to revert to the original thread memory allocation code as there were some problems with the new thread-local storage implementation, but unfortunately I had kept the wrong default (unless you build with cmake, that is). Not sure yet if this is in any way related to #1844.

[hjbreg]

Tested previsous buiding options with USE_TLS=0, still not working.
As USE_THREAD=0 NUM_THREADS=1 also does not work on Linux, and the netlib-refblas/lapack works,
then I guess it may be the OpenBLAS side issue. But it is also strange that USE_THREAD=0 NUM_THREADS=16 (or any other value greater than the runtime thread number) works on Linux, but does not work on Windows.
I think we should check globle static variables used in OpenBLAS.