Some tests fail for NVIDIA HPC SDK 20.7, 20.9, 20.11

[wyphan]

HI,

I just tried compiling reference LAPACK 3.9.0 using the newly released NVIDIA HPC SDK 20.7 on an AMD Zen2 processor (Ryzen 5 3600X). I noticed that some of the tests failed:

			-->   LAPACK TESTING SUMMARY  <--
		Processing LAPACK Testing output found in the TESTING directory
SUMMARY             	nb test run 	numerical error   	other error  
================   	===========	=================	================  
REAL             	1107279		285	(0.026%)	0	(0.000%)	
DOUBLE PRECISION	1221707		280	(0.023%)	0	(0.000%)	
COMPLEX          	641118		23	(0.004%)	0	(0.000%)	
COMPLEX16         	684278		140	(0.020%)	0	(0.000%)	

--> ALL PRECISIONS	3654382		728	(0.020%)	0	(0.000%)	

Testing REAL              Singular-Value-Decomposition-ssvd.out
  SBD drivers:     56 out of  14820 tests failed to pass the threshold
  SBD drivers:     56 out of  14820 tests failed to pass the threshold
  SBD drivers:     56 out of  14820 tests failed to pass the threshold
  SBD drivers:     56 out of  14820 tests failed to pass the threshold
  SBD drivers:     56 out of  14820 tests failed to pass the threshold
 passed: 51300
failing to pass the threshold: 280

Testing REAL              Linear-Equation-routines-stest.out
  SLS drivers:      4 out of 105840 tests failed to pass the threshold
 passed: 299334
failing to pass the threshold: 4

Testing REAL              RFP-linear-equation-routines-stest_rfp.out
   STFSM auxiliary routine:     1 out of  7776 tests failed to pass the threshold
 passed: 5352
failing to pass the threshold: 1

Testing DOUBLE PRECISION Singular-Value-Decomposition-dsvd.out
  DBD drivers:     56 out of  14820 tests failed to pass the threshold
  DBD drivers:     56 out of  14820 tests failed to pass the threshold
  DBD drivers:     56 out of  14820 tests failed to pass the threshold
  DBD drivers:     56 out of  14820 tests failed to pass the threshold
  DBD drivers:     56 out of  14820 tests failed to pass the threshold
 passed: 51300
failing to pass the threshold: 280

Testing COMPLEX           Linear-Equation-routines-ctest.out
  CPB:     11 out of   3458 tests failed to pass the threshold
  CPB drivers:      4 out of   4750 tests failed to pass the threshold
  CLS drivers:      8 out of 105840 tests failed to pass the threshold
 passed: 304541
failing to pass the threshold: 23

Testing COMPLEX16          Singular-Value-Decomposition-zsvd.out
  ZBD drivers:     28 out of  14340 tests failed to pass the threshold
  ZBD drivers:     28 out of  14340 tests failed to pass the threshold
  ZBD drivers:     28 out of  14340 tests failed to pass the threshold
  ZBD drivers:     28 out of  14340 tests failed to pass the threshold
  ZBD drivers:     28 out of  14340 tests failed to pass the threshold
 passed: 20425
failing to pass the threshold: 140

Attached is the full testing log:
testing_results.txt

Edit: added processor name

[wyphan]

Also, here is the make.inc that I used to compile. I roughly followed the steps listed in this page to build a shared library version of LAPACK by modifying Makefile and SRC/Makefile, but I think these modifications should be unrelated to the testing failures.

[martin-frbg]

If nvfortran is in any way related to recent flang you could check if adding -Kieee to FFLAGS helps (And with the AMD AOCC flavor of flang, I found it necessary to add -fno-unroll-loops so this could be another option to try and narrow it down)

[wyphan]

@martin-frbg I think it is more related to the PGI compiler than AOCC flang (Actually, the pgfortran alias is still there and now points to nvfortran), but I'll give it a try once I get back to my Zen2 workstation.

Edit: the -Kieee flag does the job! Now it's down to only 5 numerical errors:

			-->   LAPACK TESTING SUMMARY  <--
		Processing LAPACK Testing output found in the TESTING directory
SUMMARY             	nb test run 	numerical error   	other error  
================   	===========	=================	================  
REAL             	1300419		1	(0.000%)	0	(0.000%)	
DOUBLE PRECISION	1302223		4	(0.000%)	4	(0.000%)	
COMPLEX          	768366		0	(0.000%)	0	(0.000%)	
COMPLEX16         	769178		0	(0.000%)	0	(0.000%)	

--> ALL PRECISIONS	4140186		5	(0.000%)	4	(0.000%)

Testing REAL              RFP-linear-equation-routines-stest_rfp.out
   STFSM auxiliary routine:     1 out of  7776 tests failed to pass the threshold
 passed: 5352
failing to pass the threshold: 1

Testing DOUBLE PRECISION Nonsymmetric-Eigenvalue-ded.out
  DDRVES: DGEES1 returned INFO=     6.
  DDRVES: DGEES1 returned INFO=     6.
  DES:    2 out of  3264 tests failed to pass the threshold
  DGET24: DGEESX1 returned INFO=     6.
  DGET24: DGEESX1 returned INFO=     6.
  DSX:    2 out of  3494 tests failed to pass the threshold
 passed: 6198
failing to pass the threshold: 4
Info Error: 4

[wyphan]

Update: Building with NVIDIA HPC SDK version 20.9 and 20.11 also results in some errors. As suggested by @martin-frbg (at least for building OpenBLAS with PGI compilers / NVIDIA HPC SDK), building reference LAPACK also requires the -Kieee compiler flag. Attached is the make.inc file (renamed to make.inc.nv.txt) that I use for reference LAPACK, and the three full build logs (compressed as gzip files), each with NVIDIA HPC SDK 20.7, 20.9, and 20.11, respectively.

The command that I use to build is

$ make clean
$ make -j 12 blas_testing lapack_testing > build-nv20.11.log 2>&1

make.inc.nv.txt

build-nv20.7.log.gz
build-nv20.9.log.gz
build-nv20.11.log.gz