[SYCL][CUDA] llvm-test-suite : AtomicRef/sub.cpp test fails for DPC++ with new Pass Manager turned on

[mlychkov]

Describe the bug
When DPC++ was tested with new Pass Manager turned on by default AtomicRef/sub.cpp test from llvm-test-suite failed with the following error:

[2022-03-01T16:25:36.560Z] ******************** TEST 'SYCL :: AtomicRef/sub.cpp' FAILED ********************
[2022-03-01T16:25:36.560Z] Script:
[2022-03-01T16:25:36.560Z] --
...
[2022-03-01T16:25:36.562Z] --
[2022-03-01T16:25:36.562Z] Exit Code: -9
[2022-03-01T16:25:36.562Z] Timeout: Reached timeout of 600 seconds
[2022-03-01T16:25:36.562Z] 
[2022-03-01T16:25:36.562Z] Command Output (stdout):
[2022-03-01T16:25:36.562Z] --
...
[2022-03-01T16:25:37.337Z] $ ":" "RUN: at line 147"
[2022-03-01T16:25:37.337Z] note: command had no output on stdout or stderr
[2022-03-01T16:25:37.337Z] $ "/localdisk2/iusers/nstester/Codeplay_jenkins-test2/workspace/SYCL_CI/intel/Lin/LLVM_Test_Suite_CUDA/llvm.obj/bin/clang++" "-fsycl" "-fsycl-unnamed-lambda" "-fsycl-targets=nvptx64-nvidia-cuda" "/localdisk2/iusers/nstester/Codeplay_jenkins-test2/workspace/SYCL_CI/intel/Lin/LLVM_Test_Suite_CUDA/llvm-test-suite/SYCL/AtomicRef/sub.cpp" "-o" "/localdisk2/iusers/nstester/Codeplay_jenkins-test2/workspace/SYCL_CI/intel/Lin/LLVM_Test_Suite_CUDA/build/SYCL/AtomicRef/Output/sub.cpp.tmp.out" "-Xsycl-target-backend=nvptx64-nvidia-cuda" "--cuda-gpu-arch=sm_70" "-DACQ_REL" "-DSYSTEM" "-DATOMIC64"
[2022-03-01T16:25:37.337Z] # command stderr:
...
[2022-03-01T16:25:37.349Z]                                      ^
[2022-03-01T16:25:37.349Z] 22 warnings generated.
[2022-03-01T16:25:37.349Z] 
[2022-03-01T16:25:37.349Z] error: command failed with exit status: -9
[2022-03-01T16:25:37.349Z] error: command reached timeout: True

PR with patch to enable new PM: #4860
Link on a failed test log: http://llvm-ci-test2.intel.com:8080/blue/organizations/jenkins/SYCL_CI%2Fintel%2FLin%2FLLVM_Test_Suite_CUDA/detail/LLVM_Test_Suite_CUDA/355/pipeline

To Reproduce
Please describe the steps to reproduce the behavior:

1. Compile DPC++ compiler with new PM turned on by default (see PR Enable new pass manager back by default #4860).
2. Execute llvm-test-suite
3. All tests should pass, but AtomicRef tests may fail

Environment (please complete the following information):

• OS: Linux
• Target device and vendor: Nvidia GPU
• DPC++ version: 273bdcf

Additional context
AtomicRef tests were added with the patch intel/llvm-test-suite#534.
They are huge and it is a known issue that there may be a failure due to test timeout in CI.

[bader]

@mlychkov, I can't access the log from public network.

error: command reached timeout: True

It's not a bug in DPC++, but rather with the test framework. It doesn't give enough time for test to run itself.

@t4c1, I see that intel/llvm-test-suite@88ee9d1 significantly increased the time to run llvm-test-suite. I'll provide data for Intel GEN9 on Level Zero.

Statistics from the run before we updated atomic tests (from https://github.com/intel/llvm/runs/5296817617?check_suite_focus=true):

Slowest Tests:
--------------------------------------------------------------------------
66.98s: SYCL :: Regression/optimization_level_debug_info.cpp
41.91s: SYCL :: ESIMD/api/simd_view_select_2d.cpp
40.34s: SYCL :: HostInteropTask/host-task-dependency3.cpp
34.77s: SYCL :: Regression/commandlist/gpu.cpp
34.67s: SYCL :: GroupAlgorithm/SYCL2020/sort.cpp
32.98s: SYCL :: Basic/multisource.cpp
31.11s: SYCL :: GroupAlgorithm/exclusive_scan_sycl2020.cpp
31.10s: SYCL :: GroupAlgorithm/inclusive_scan_sycl2020.cpp
30.74s: SYCL :: Basic/vector_byte.cpp
30.63s: SYCL :: KernelAndProgram/multiple-kernel-linking.cpp
29.79s: SYCL :: ESIMD/api/simd_copy_to_from.cpp
29.02s: SYCL :: Reduction/reduction_reducer_op_eq.cpp
28.82s: SYCL :: Basic/vector_operators.cpp
26.39s: SYCL :: ESIMD/regression/complex-lib-lin.cpp
24.90s: SYCL :: ESIMD/api/slm_gather_scatter_heavy.cpp
23.87s: SYCL :: SubGroup/load_store.cpp
23.60s: SYCL :: SubGroup/scan_spirv13.cpp
22.96s: SYCL :: ESIMD/api/bin_and_cmp_ops_heavy.cpp
22.96s: SYCL :: Basic/stream/stream.cpp
22.43s: SYCL :: Regression/2020-spec-constants-debug-info.cpp

Tests Times:
--------------------------------------------------------------------------
[    Range    ] :: [               Percentage               ] :: [ Count ]
--------------------------------------------------------------------------
[65.0s,70.0s) :: [                                        ] :: [  1/765]
[60.0s,65.0s) :: [                                        ] :: [  0/765]
[55.0s,60.0s) :: [                                        ] :: [  0/765]
[50.0s,55.0s) :: [                                        ] :: [  0/765]
[45.0s,50.0s) :: [                                        ] :: [  0/765]
[40.0s,45.0s) :: [                                        ] :: [  2/765]
[35.0s,40.0s) :: [                                        ] :: [  2/765]
[30.0s,35.0s) :: [                                        ] :: [  8/765]
[25.0s,30.0s) :: [                                        ] :: [  2/765]
[20.0s,25.0s) :: [                                        ] :: [ 15/765]
[15.0s,20.0s) :: [*                                       ] :: [ 30/765]
[10.0s,15.0s) :: [*****                                   ] :: [108/765]
[ 5.0s,10.0s) :: [**********************                  ] :: [421/765]
[ 0.0s, 5.0s) :: [*********                               ] :: [176/765]
--------------------------------------------------------------------------

The same statistics now (https://github.com/intel/llvm/runs/5402651128?check_suite_focus=true):

Slowest Tests:
--------------------------------------------------------------------------
312.26s: SYCL :: AtomicRef/sub.cpp
308.57s: SYCL :: AtomicRef/add.cpp
250.49s: SYCL :: AtomicRef/sub_generic.cpp
247.70s: SYCL :: AtomicRef/add_generic.cpp
223.39s: SYCL :: AtomicRef/exchange.cpp
219.51s: SYCL :: AtomicRef/compare_exchange.cpp
216.09s: SYCL :: AtomicRef/min.cpp
213.85s: SYCL :: AtomicRef/max.cpp
206.20s: SYCL :: AtomicRef/exchange_generic.cpp
205.22s: SYCL :: AtomicRef/or.cpp
205.05s: SYCL :: AtomicRef/xor.cpp
200.90s: SYCL :: AtomicRef/compare_exchange_generic.cpp
200.62s: SYCL :: AtomicRef/and.cpp
200.23s: SYCL :: AtomicRef/max_generic.cpp
198.56s: SYCL :: AtomicRef/min_generic.cpp
193.90s: SYCL :: AtomicRef/or_generic.cpp
192.27s: SYCL :: AtomicRef/xor_generic.cpp
191.36s: SYCL :: AtomicRef/and_generic.cpp
185.90s: SYCL :: AtomicRef/sub_native_fp.cpp
182.98s: SYCL :: AtomicRef/add_native_fp.cpp

Tests Times:
--------------------------------------------------------------------------
[   Range   ] :: [               Percentage               ] :: [ Count ]
--------------------------------------------------------------------------
[300s,320s) :: [                                        ] :: [  2/803]
[280s,300s) :: [                                        ] :: [  0/803]
[260s,280s) :: [                                        ] :: [  0/803]
[240s,260s) :: [                                        ] :: [  2/803]
[220s,240s) :: [                                        ] :: [  3/803]
[200s,220s) :: [                                        ] :: [  8/803]
[180s,200s) :: [                                        ] :: [  9/803]
[160s,180s) :: [                                        ] :: [  2/803]
[140s,160s) :: [                                        ] :: [  0/803]
[120s,140s) :: [                                        ] :: [  0/803]
[100s,120s) :: [                                        ] :: [  0/803]
[ 80s,100s) :: [                                        ] :: [  0/803]
[ 60s, 80s) :: [                                        ] :: [  1/803]
[ 40s, 60s) :: [                                        ] :: [  2/803]
[ 20s, 40s) :: [*                                       ] :: [ 27/803]
[  0s, 20s) :: [*************************************   ] :: [747/803]
--------------------------------------------------------------------------

I suggest we either:

1. optimize the tests. I think multiple compilations is the main reason of poor performance.
2. or reduce the scope of llvm-test-suite. We can move extensive coverage to SYCL-CTS and leave mininal set of checks in llvm-test, but I would prefer the tests to run in < 10s, otherwise they are too expensive for pre-commit testing.

Tagging @vladimirlaz and @alexbatashev for awareness.

[bader]

Update for my previous comment. I found the @vladimirlaz committed intel/llvm-test-suite#879 four days ago, which should reduce the execution time of AtomicRef tests.

[t4c1]

The issue with atomic_ref tests timing out has been resolved by intel/llvm-test-suite#879.