check-all misses 3200+ compiler-rt tests in runtimes build

[rorth]

As has been reported for a long time (probably since the LLVM 14 release cycle when the runtimes build was first made the reltest.sh default), building with -DLLVM_ENABLE_RUNTIMES=compiler-rt (which is the default these days) misses a large number of compiler-rt tests that are run with -DLLVM_ENABLE_PROJECTS=compiler-rt. The issue came up again in Issue #71971 and has apparently never been noticed during the development of the runtimes build (which doesn't speak particularly highly for the diligence of the testing at that point) and completely ignored despite the release tester reports since, massively reducing compiler-rt test coverage and missing all sorts of issues in new patches that would be noticed immedidately otherwise.

To provide detail, I've now run two x86_64-pc-linux-gnu builds, one projects build and one runtimes build. Here are the differences in overall test results from just that change (which should be a no-op for test results):

-Total Discovered Tests: 110511
-  Skipped          :     84 (0.08%)
-  Unsupported      :   3723 (3.37%)
-  Passed           : 106411 (96.29%)
-  Expectedly Failed:    292 (0.26%)
+Total Discovered Tests: 107217
+  Skipped          :     48 (0.04%)
+  Unsupported      :   3591 (3.35%)
+  Passed           : 103287 (96.33%)
+  Expectedly Failed:    290 (0.27%)
   Failed           :      1 (0.00%)

As one can see, more than 3200 tests are no longer run in a runtimes build. I've then extracted the names of the tests run from both build's ninja check-all output. Here are the test categrories that are missing in the runtimes build:

    836 AddressSanitizer-Unit
      3 Builtins
    215 Builtins-i386-linux
    215 Builtins-x86_64-linux
     70 GwpAsan-Unittest
     12 Interception-Unit
     58 LLVMFuzzer-Unittest
    194 SanitizerCommon-Unit
    275 ScudoStandalone-Unit
    275 ScudoStandalone-Unit-GwpAsanTorture
     79 ThreadSanitizer-Unit
     15 XRay-x86_64-linux		15 out of 29 tests turn UNSUPPORTED

In most cases, all tests from a category are missing, while for the xray tests about half of the tests changed from PASS to UNSUPPORTED.

I then looked for the missing tests: in the case of builtins, the lit.* files that controlled the tests used to live in

-test/builtins/lit.site.cfg.py
-test/builtins/Unit/I386LinuxConfig/lit.site.cfg.py
-test/builtins/Unit/X86_64LinuxConfig/lit.site.cfg.py

while they are now missing completely: they are neither present in runtimes-bins nor in builtins-bins where they should be after splitting the object directories for builtins from the rest of compiler-rt. Without the lit.* files, it's no wonder the builtins tests aren't run at all.

For the other missing tests, the lit.* files are present, but I don't yet see why the tests aren't run.

[vitalybuka]

Interesting. Can you please provide full cmake command?
I recently, about month ago, switched our bots from DLLVM_ENABLE_PROJECTS to DLLVM_ENABLE_RUNTIMES and compared full verbose output. There were small lld related difference, which I fixed, but nothing big.

[rorth]

I just saw it when looking at a LLVM 17 release build with reltest.sh, which always does a runtimes build. When looking for Builtins-x86_64-linux :: divtc3_test.c, I came up blank.

The main builds I'm referring to above were 2-stage builds. The exact invocations are:

• projects build: cmake -G Ninja -DCMAKE_C_COMPILER=/usr/bin/gcc -DCMAKE_CXX_COMPILER=/usr/bin/g++ -DCMAKE_C_FLAGS= -DCMAKE_CXX_FLAGS= -DCMAKE_BUILD_TYPE=Release '-DLLVM_ENABLE_PROJECTS=clang;clang-tools-extra;polly;flang;compiler-rt' '-DLLVM_LIT_ARGS=-v -j 64' -DLLVM_PARALLEL_COMPILE_JOBS=64 -DLLVM_PARALLEL_LINK_JOBS=32 -DLLVM_ENABLE_ASSERTIONS=ON -DCLANG_ENABLE_BOOTSTRAP=On '-DCLANG_BOOTSTRAP_PASSTHROUGH=;BUILD_SHARED_LIBS;CLANG_DEFAULT_LINKER;CLANG_ENABLE_GLD;CMAKE_BUILD_TYPE;CMAKE_VERBOSE_MAKEFILE;COMPILER_RT_DEBUG;ENABLE_X86_RELAX_RELOCATIONS;LIBXML2_LIBRARY;LLVM_ENABLE_ASSERTIONS;LLVM_ENABLE_LLD;LLVM_ENABLE_PER_TARGET_RUNTIME_DIR;LLVM_ENABLE_PROJECTS;LLVM_ENABLE_RUNTIMES;LLVM_HOST_TRIPLE;LLVM_LIT_ARGS;LLVM_PARALLEL_COMPILE_JOBS;LLVM_PARALLEL_LINK_JOBS;LLVM_TARGETS_TO_BUILD;' -DLLVM_ENABLE_PER_TARGET_RUNTIME_DIR=Off /vol/llvm/src/llvm-project/local/llvm
• runtimes build: cmake -G Ninja -DCMAKE_C_COMPILER=/usr/bin/gcc -DCMAKE_CXX_COMPILER=/usr/bin/g++ -DCMAKE_C_FLAGS= -DCMAKE_CXX_FLAGS= -DCMAKE_BUILD_TYPE=Release '-DLLVM_ENABLE_PROJECTS=clang;clang-tools-extra;polly;flang' '-DLLVM_LIT_ARGS=-v -j 64' -DLLVM_PARALLEL_COMPILE_JOBS=64 -DLLVM_PARALLEL_LINK_JOBS=32 -DLLVM_ENABLE_ASSERTIONS=ON -DCLANG_ENABLE_BOOTSTRAP=On '-DCLANG_BOOTSTRAP_PASSTHROUGH=;BUILD_SHARED_LIBS;CLANG_DEFAULT_LINKER;CLANG_ENABLE_GLD;CMAKE_BUILD_TYPE;CMAKE_VERBOSE_MAKEFILE;COMPILER_RT_DEBUG;ENABLE_X86_RELAX_RELOCATIONS;LIBXML2_LIBRARY;LLVM_ENABLE_ASSERTIONS;LLVM_ENABLE_LLD;LLVM_ENABLE_PER_TARGET_RUNTIME_DIR;LLVM_ENABLE_PROJECTS;LLVM_ENABLE_RUNTIMES;LLVM_HOST_TRIPLE;LLVM_LIT_ARGS;LLVM_PARALLEL_COMPILE_JOBS;LLVM_PARALLEL_LINK_JOBS;LLVM_TARGETS_TO_BUILD;' -DLLVM_ENABLE_RUNTIMES=compiler-rt /vol/llvm/src/llvm-project/local/llvm

[vitalybuka]

I can reproduce this. DLLVM_ENABLE_RUNTIMES check-all is missing all/most unittests and all builtin tests.

However check-compiler-rt is missing only builtin tests.

On our sanitizer bots we run check-compiler-rt, and builtins are not in the scope of the group, so I probably was not concerned switching to DLLVM_ENABLE_RUNTIMES.

[petrhosek]

I have managed to reproduce this and I think I understand the issue, I'll try to come up with a solution.

[petrhosek]

I have confirmed that most of these will be addressed by #72955, the only ones that's not covered by that pull request are builtins tests which need a different solution.

[rorth]

When building main on sparc64-unknown-linux-gnu as a runtimes build in preparation for LLVM 18.1.0 rc1, I noticed that the builtins tests are still not run with check-all. I couldn't even find any target to run them at all.

[rorth]

Unfortunately, the lack of check-builtins support is still an issue after 9 months. IIUC this is (or isn't) handled in the terrible maze that is llvm/runtimes/CMakeLists.txt: while the runtimes support in there does add check-runtimes* targets, the (sort of) equivalent builtins support does not.

[rorth]

When doing a 2-stage build on Solaris (amd64 and sparcv9) yesterday, this warning reminded me of upcoming doom:

CMake Warning at CMakeLists.txt:161 (message):
  Using LLVM_ENABLE_PROJECTS=compiler-rt is deprecated now, and will become a
  fatal error in the LLVM 21 release.  Please use
  -DLLVM_ENABLE_RUNTIMES=compiler-rt or see the instructions at
  https://compiler-rt.llvm.org/ for building the runtimes.

Prompted by this, ran two 2-stage builds on x86_64-pc-linux-gnu, one with LLVM_ENABLE_PROJECTS=compiler-rt, the other with LLVM_ENABLE_RUNTIMES=compiler-rt and compared the sets of tests run in each case:

• As I'd feared, the builtins tests are still not run at all in a runtimes build.
• Besides, quite a number of xray tests change from PASS to UNSUPPORTED:

-PASS: XRay-x86_64-linux :: TestCases/Posix/always-never-instrument.cpp
+UNSUPPORTED: XRay-x86_64-linux :: TestCases/Posix/always-never-instrument.cpp
-PASS: XRay-x86_64-linux :: TestCases/Posix/basic-filtering.cpp
+UNSUPPORTED: XRay-x86_64-linux :: TestCases/Posix/basic-filtering.cpp
-PASS: XRay-x86_64-linux :: TestCases/Posix/basic-mode-dso.cpp
+UNSUPPORTED: XRay-x86_64-linux :: TestCases/Posix/basic-mode-dso.cpp
-PASS: XRay-x86_64-linux :: TestCases/Posix/clang-no-xray-instrument.cpp
+UNSUPPORTED: XRay-x86_64-linux :: TestCases/Posix/clang-no-xray-instrument.cpp
-PASS: XRay-x86_64-linux :: TestCases/Posix/common-trampoline-alignment.cpp
+UNSUPPORTED: XRay-x86_64-linux :: TestCases/Posix/common-trampoline-alignment.c
pp
-PASS: XRay-x86_64-linux :: TestCases/Posix/c-test.cpp
+UNSUPPORTED: XRay-x86_64-linux :: TestCases/Posix/c-test.cpp
-PASS: XRay-x86_64-linux :: TestCases/Posix/custom-event-handler-alignment.cpp
+UNSUPPORTED: XRay-x86_64-linux :: TestCases/Posix/custom-event-handler-alignment.cpp
-PASS: XRay-x86_64-linux :: TestCases/Posix/custom-event-logging.cpp
+UNSUPPORTED: XRay-x86_64-linux :: TestCases/Posix/custom-event-logging.cpp
-PASS: XRay-x86_64-linux :: TestCases/Posix/default-options.cpp
+UNSUPPORTED: XRay-x86_64-linux :: TestCases/Posix/default-options.cpp
-PASS: XRay-x86_64-linux :: TestCases/Posix/fdr-mode.cpp
+UNSUPPORTED: XRay-x86_64-linux :: TestCases/Posix/fdr-mode.cpp
-PASS: XRay-x86_64-linux :: TestCases/Posix/fdr-mode-inmemory.cpp
+UNSUPPORTED: XRay-x86_64-linux :: TestCases/Posix/fdr-mode-inmemory.cpp
-PASS: XRay-x86_64-linux :: TestCases/Posix/fdr-mode-multiple.cpp
+UNSUPPORTED: XRay-x86_64-linux :: TestCases/Posix/fdr-mode-multiple.cpp
-PASS: XRay-x86_64-linux :: TestCases/Posix/fdr-thread-order.cpp
+UNSUPPORTED: XRay-x86_64-linux :: TestCases/Posix/fdr-thread-order.cpp
-PASS: XRay-x86_64-linux :: TestCases/Posix/fork_basic_logging.cpp
+UNSUPPORTED: XRay-x86_64-linux :: TestCases/Posix/fork_basic_logging.cpp
-PASS: XRay-x86_64-linux :: TestCases/Posix/profiling-multi-threaded.cpp
+UNSUPPORTED: XRay-x86_64-linux :: TestCases/Posix/profiling-multi-threaded.cpp
-PASS: XRay-x86_64-linux :: TestCases/Posix/profiling-single-threaded.cpp
+UNSUPPORTED: XRay-x86_64-linux :: TestCases/Posix/profiling-single-threaded.cpp
-PASS: XRay-x86_64-linux :: TestCases/Posix/quiet-start.cpp
+UNSUPPORTED: XRay-x86_64-linux :: TestCases/Posix/quiet-start.cpp

I haven't even started looking at those.

This combination of missing tests clearly indicates that the runtimes build of compiler-rt is not ready for prime time, and the fact that this issue has been ignored for a year and a half bodes ill for the actual maintenance of the runtimes build.

I understand all too well that maintaining two different ways of building compiler-rt in parallel is a total nightmare, but the replacement needs to maintain existing functionality to be acceptable. I've looked into fixing this myself, but quickly got lost in the maze the ist CMake.