numpy test failed with openblas #1673
Comments
|
Could you re-test with current "develop" branch please ? This could be another instance of #1666 |
|
Hi, |
|
Would be a pity if we had to revert all that thread initialization speedup we gained with 0.3.1. BTW what is your cpu type (not that much changed in terms of cpu-specific code lately) ? |
|
@beew would it be possible for you to bisect on one of these remaining failures? one more thing to try would be to see if any of these failures occur with a single threaded version. I recommend trying two cases:
|
|
numpy.core.tests.test_multiarray.TestDot.test_accelerate_framework_sgemv_fix looks like accuracy deviation, not a failure. |
|
Seeing that DYNAMIC_ARCH is involved, there is a (small) chance that this may be from running on one of the older cpus that got moved to the DYNAMIC_OLDER list in 0.3.1 (e.g. if the numpy test was run on an ATOM or OPTERON target), and fallback to the ancient Prescott kernels from libgoto may have exposed an old bug. |
|
@martin-frbg This is because I have export LD_PRELOAD= /usr/lib/libblas.so in my .profile. As mentioned before I switch between different blases and dynamic switching apparently doesn't work for octave, numpy and R unless blas is pre loaded. Merely update-alternatives doesn't work, many online tutorials are wrong, for example ldd or in numpy's case numpy.show_config() merely shows which blas the program is compiled against, not which one is being used at runtime, but actually monitoring the process with lsof you can see which blas is actually used. There is no difference if not set LD_PRELOAD at runtime even with different alternatives (same blas is used). I see different blas is loaded along with performance differences only if I do LD_PRELOAD AND update-alterrantives (for my tests intel-mkl is slightly faster than openblas, both beat a custom built atlas by big margin) so instead of having to manually LD_PRELOAD each time I just put it in .profile. |
|
@sandwichmaker openblas-dev + USE_OPENMP=1+OMP_NUM_THREADS=1 all tests passed. |
|
For some additional info openblas0.3.1 also segfaulted a lot of opencv tests while openblas0.3.0 passed (compiled with make PREFIX=/opt/openblas-mpi NO_AFFINITY=1 DYNAMIC_ARCH=1 USE_OPENMP=1 -j8) Since the opencv tests are somewhat of a mess anyway I don't bother to report that. It seems that the same underlying issues are responsible. |
|
Not reproducible with current develop on my Kaby Lake system (though this has python 3.4.5/numpy 1.9.3). Note 0.3.1 has a known problem with thread allocation that should be fixed in develop. |
|
Problem now reproduced with a source install of current numpy, and bisected to a399d00 which is PR #1625 "Further improvements to memory.c". |
|
Indeed removing all the "!USE_OPENMP" special cases from the new thread memory handling fixes the problem (performance impact not studied). |
|
(I’m at a conference this week)
That’s very interesting, since I when I tested the OpenMP build I only saw the server allocating, and there were other places with the same condition (SMP && !USE_OPENMP).
… On Jul 10, 2018, at 1:47 PM, Martin Kroeker ***@***.***> wrote:
Indeed removing all the "!USE_OPENMP" special cases from the new thread memory handling fixes the problem (performance impact not studied).
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.
|
Yes, numpy has to be installed from source. Sorry for forgetting to mention that. Otherwise I think it links against its own blas if update alternatives is set to openblas. I tried to reproduce this on another machine with numpy installed with pip3 (OS is Ubuntu 18.04, openblas compiled from source and set up update-alternatives to switch between mkl and openblas in same manner) Doing numpy.test() gives this curious error It seems that it is because libfortran got linked twice, once from system openblas once from numpy's(?) This doesn't happen if update-alternatives is set to intel mkl |
|
Looks like it. And it complains specifically because one of the two - numpy I assume - was built with the old fortran77-only version of the GNU compiler. |
|
Is there any benchmark test that shows openblas 0.31 is actually faster than 0.30? I just did the basic random matrix multiplications and I don't see any difference. |
|
Yes, the main speedup was non-OpenMP threaded builds, since the changes
avoided the large amount of locking happening during memory allocation.
OpenMP didn’t have this issue since it wasnt locking (which led me in part
to believe there was only one allocating thread). The gains should still
be there despite the fixes, since the fundamental change (less locking) is
still there.
…On Wed, Jul 11, 2018 at 5:15 PM Martin Kroeker ***@***.***> wrote:
Some numbers were in #1618 <#1618>,
#1625 <#1625> (the data from #1624
<#1624> is less useful as it
combines both the memory.c rewrite and the introduction of AVX512 SGEMM). I
expect some of the gain may already have been lost due to required
corrections, also the main effect should be on pthread builds without
OPENMP.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#1673 (comment)>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/ALSwdlJCw3U0NVGbeaCgO2stVU66KST9ks5uFhZ4gaJpZM4VGbVk>
.
|
updated to openblas0.3.1 and numpy appears to have some problems.
OS is Ubuntu 16.04.4 64 bits. I made sure that I don't have any bit of Ubuntu's packaged openblas in my system. I compiled openblas with
I use update alternatives to switch between intel-mkl and openblas.
tested numpy in python3 with numpy.test() (see outputs below for python and numpy versions)
intel mkl : All tests passed
openblas0.3.0: All test passed
openblas0.3.1 :7 tests failed
The text was updated successfully, but these errors were encountered: