CentOS 6.5 gcc 4.4.7 pthreads dgetrf_parallel segfaults

[debasish83]

Architecture:

processor : 0
vendor_id : GenuineIntel
cpu family : 6
model : 63
model name : Intel(R) Xeon(R) CPU E5-2660 v3 @ 2.60GHz
stepping : 2
cpu MHz : 2599.807
cache size : 25600 KB
physical id : 0
siblings : 20
core id : 0
cpu cores : 10
apicid : 0
initial apicid : 0
fpu : yes
fpu_exception : yes
cpuid level : 15
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm ida arat epb xsaveopt pln pts dts tpr_shadow vnmi flexpriority ept vpid fsgsbase bmi1 avx2 smep bmi2 erms invpcid
bogomips : 5199.61
clflush size : 64
cache_alignment : 64
address sizes : 46 bits physical, 48 bits virtual
power management:

Kernel:
Linux 2.6.32-431.29.2.el6.x86_64

OS: CentOS 6.5 gcc (GCC) 4.4.7 20120313 (Red Hat 4.4.7-4)
GNU Fortran (GCC) 4.4.7 20120313 (Red Hat 4.4.7-4)

If I compile OpenBLAS as follows, pthreads is not linked and dgetrf (LU factorization) runs fine:

make NO_AVX2=1 USE_THREAD=0

If I compile OpenBLAS with default options (cpu = 40), pthreads library is linked and then dgetrf gives me following stacktrace:

project math
[info] Set current project to breeze (in build file:/data0/debasish/breeze/)
runMain breeze.optimize.proximal.QuadraticMinimizer 4096 1 0.01 0.99
[warn] Credentials file /home/debasish/.ivy2/.credentials does not exist
[warn] compile:run::javaOptions will be ignored, compile:run::fork is set to false
[info] Running breeze.optimize.proximal.QuadraticMinimizer 4096 1 0.01 0.99
Generating randomized QPs with rank 4096 equalities 1
Jun 12, 2016 5:11:08 PM com.github.fommil.jni.JniLoader liberalLoad
INFO: successfully loaded /tmp/jniloader5524104160912461112netlib-native_system-linux-x86_64.so
Test QuadraticMinimizer, CG , BFGS and OWLQN with 4096 variables and 1 equality constraints
Jun 12, 2016 5:11:11 PM com.github.fommil.jni.JniLoader load
INFO: already loaded netlib-native_system-linux-x86_64.so
An irrecoverable stack overflow has occurred.
Please check if any of your loaded .so files has enabled executable stack (see man page execstack(8))

A fatal error has been detected by the Java Runtime Environment:

SIGSEGV (0xb) at pc=0x00007fbeca5e0167, pid=81202, tid=140460089050880

JRE version: Java(TM) SE Runtime Environment (7.0_67-b01) (build 1.7.0_67-b01)

Java VM: Java HotSpot(TM) 64-Bit Server VM (24.65-b04 mixed mode linux-amd64 compressed oops)

Problematic frame:

C [libblas.so.3+0x3d4167] dgetrf_parallel+0x6c7

Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again

An error report file with more information is saved as:

/data0/debasish/breeze/hs_err_pid81202.log

With multiple threads, I am getting around 2X improvement in dpotrs and dpotrf and the runtime gain is significant to ignore and compile without pthreads...

dgetrf and dgetrs for LU solver is puking. Is this a known issue or a bug in the code ? If there is bug I can debug and possibly add a fix. Please confirm.

[martin-frbg]

Which version of OpenBLAS ? Does "with default options" still include the NO_AVX2=1 (probably yes, given your old compiler/assembler environment) ?
Perhaps try if setting USE_OPENMP=1 to improve thread safety helps.

[debasish83]

Yes I have used NO_AVX2=1 since we have CentOS 6.5 with gcc 4.4.7 on our dev nodes...We are upgrading to CentOS 7.2 but it will take time...If I compile with USE_OPENMP=1, then the segfaults goes away...Is it because I am calling through JNI, the pthread version that's default is not working and I need to use openmp ? Let me know what you think is the reason and I can close the issue

[debasish83]

This compilation worked fine:
make NO_AVX2=1 USE_OPENMP=1 PREFIX=/data0/debasish/opt/OpenBLAS/ install
And then to improve runtime, I ran with:
export OMP_NUM_THREADS=8 or OMP_NUM_THREADS=4 which gave runtimes at par with Mac VecLib (I am not sure VecLib run with how many cores)

[debasish83]

@martin-frbg @xianyi Can you please confirm if the pthread failure was a bug ? If OPEN_MP is more robust than pthread option why do we have both ? If it is a bug, I can debug it for users who are using the pthread option and not open-mp...

[brada4]

What bug?

Did you read the documentaion?
https://github.com/xianyi/OpenBLAS/blob/develop/GotoBLAS_06WeirdPerformance.txt

Evolving further: Are your JNI calls to multi-threaded BLAS synchronized?
https://github.com/xianyi/OpenBLAS/wiki/faq#multi-threaded

[brada4]

Can you repeat same with openjdk? Oracle jdk kind of forbids disassembling

[brada4]

Can you repeat same with openjdk in a gdb debugger and provide
gdb> t a a bt
Output at the moment of crash?

[debasish83]

I will test and update...

[debasish83]

I am going to close it since we used single core OpenBLAS for production and Spark does the multi partition processing with each partition is assigned to one core which then calls single core OpenBLAS