BLAS memory allocation error in Scikit-learn KMeans & kNN & DBSCAN

[OnlyDeniko]

scikit-learn/scikit-learn#20539

Do you have any ideas?

[martin-frbg]

The recommendation given there was on the right track, I wonder why it did not work. Multithreaded OpenBLAS requires a memory buffer per thread, and the maximum number of buffers is set at compile time. So there is an (ideally/normally )invisible limitation caused by what the OpenBLAS that came with either numpy or your operating system was configured for. Does it work when you set OPENBLAS_NUM_THREADS to a smaller value, like 16 or 32 ? (The OpenBLAS that comes with numpy 1.21 is built for 64 threads as recently established here #3318 (comment) but maybe you have some other version imported elsewhere in your combination of programs)

[OnlyDeniko]

I installed numpy: pip install -i https://pypi.anaconda.org/scipy-wheels-nightly/simple numpy
Reproducer works only with OPENBLAS_NUM_THREADS <= 17

[martin-frbg]

Sounds like that package was built with NUM_THREADS=16 although the conda-forge "recipe" for building openblas packages sets this number to 128. Not sure about the who/how/where for the packages though.

[OnlyDeniko]

Just to clarify, I download Scikit-learn from pip channel. Numpy downloads as a dependency and installing with openblas.
I found .so file in site-packages/numpy.libs folder, maybe it can help you libopenblas64_p-r0-6d9684d7.3.17.so

[brada4]

These nightly builds are unstable and are only available as pip packages on PyPI

Please install official stable versions with conda and re-check. They dont even have the record of the release tag 6d9684d7 in the search

[OnlyDeniko]

I have no errors when I download numpy from conda channels (main, conda-forge, intel) because openblas downloads separately.
Error appears when I download numpy from pip. So, I download last stable version of numpy 1.21.1from pip channel and in site-packages/numpy.libs have libopenblasp-r0-2d23e62b.3.17.so

[brada4]

So, problem is gone?
We cannot decode those numbers on real pypi, anaconda pypi, or conda forge. What is certain we did not make nightly build you downloaded from anaconda pypi.

[OnlyDeniko]

No, the problem remains. I do not download any nightly builds, I just type this: pip install numpy

[brada4]

Please check configuration of .so files you got:

ctypes.CDLL("/path/to/your/lib/openblas.so").openblas_get_config()

nproc from your machine? Seems like that was not answered in other thread.

[martin-frbg]

If you do not want to limit the number of OpenBLAS threads, the only solution would seem to be to build OpenBLAS from source with a high enough NUM_THREADS. I cannot match the hash in the library names you mentioned to a specific build (and consequently build options), but I would expect the maximum supported thread count to be at least 64. Maybe what is happening
is that parts of Scikit-learn are themselves making parallel calls into OpenBLAS - limiting the thread count may even provide a performance benefit in that situation

[OnlyDeniko]

Please check configuration of .so files you got:

ctypes.CDLL("/path/to/your/lib/openblas.so").openblas_get_config()

nproc from your machine? Seems like that was not answered in other thread.

nproc=96

>>> import ctypes
>>> ctypes.CDLL("/home/ubuntu/miniconda3/lib/python3.8/site-packages/numpy.libs/libopenblasp-r0-2d23e62b.3.17.so").openblas_get_config()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/ubuntu/miniconda3/lib/python3.8/ctypes/__init__.py", line 381, in __init__
    self._handle = _dlopen(self._name, mode)
OSError: libgfortran-2e0d59d6.so.5.0.0: cannot open shared object file: No such file or directory

$ ldd libopenblasp-r0-2d23e62b.3.17.so
        linux-vdso.so.1 (0x00007ffcdf182000)
        libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f1f42a1b000)
        libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f1f427fc000)
        libgfortran-2e0d59d6.so.5.0.0 => not found
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f1f4240b000)
        /lib64/ld-linux-x86-64.so.2 (0x00007f1f44d16000)

[isuruf]

There are multiple installations you are talking about here.

libopenblas64_p-r0-6d9684d7.3.17.so

This is a nightly build with INTERFACE64=1

libopenblasp-r0-2d23e62b.3.17.so

I'm not sure where you got this from. It doesn't seem to be from PyPI

Can you remove numpy and make sure that there's nothing in /home/ubuntu/miniconda3/lib/python3.8/site-packages/numpy.libs/
and then install using pip install numpy? Send the output of pip install numpy.

[OnlyDeniko]

As you recommended I install numpy using this: pip install -i https://pypi.anaconda.org/scipy-wheels-nightly/simple numpy
And I got in site-packages/numpy.libs this libopenblas64_p-r0-6d9684d7.3.17.so. But it is does not help, so let's forget about it and talk about only about pip install numpy
You can try to download and will get the following result:

$ pip install numpy
Collecting numpy
  Using cached numpy-1.21.1-cp38-cp38-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (15.8 MB)
Installing collected packages: numpy
Successfully installed numpy-1.21.1

[brada4]

Build command for OpenBLAS here: i.e DYNAMIC_ARCH and 128 threads
https://github.com/conda-forge/openblas-feedstock/blob/25dab765a98489e0a3e2ca8c3e7094e21e471425/recipe/build.sh#L68

[OnlyDeniko]

Build command for OpenBLAS here: i.e DYNAMIC_ARCH and 128 threads
https://github.com/conda-forge/openblas-feedstock/blob/25dab765a98489e0a3e2ca8c3e7094e21e471425/recipe/build.sh#L68

I'm not going to manually build openblas. As a user, I want to be sure that with the most ordinary download of a numpy, it will start and work on any machine. When downloading from a conda channels, this happens, but the problem is in the pip channel. Have you been able to reproduce the problem yourself?