compiler-rt build missing library file

[liam-clink]

Build for the full LLVM project (minus libc) completes, but fails tests due to a missing library file when running check-all. What could be causing this?

Error message:

********************
FAIL: ThreadSanitizer-x86_64 :: real_deadlock_detector_stress_test.cpp (63376 of 77101)
******************** TEST 'ThreadSanitizer-x86_64 :: real_deadlock_detector_stress_test.cpp' FAILED ********************
Script:
--
: 'RUN: at line 1';      /home/liam/Downloads/llvm-project-15.0.3.src/build/./bin/clang  --driver-mode=g++ -fsanitize=thread -Wall  -m64  -msse4.2   -gline-tables-only -I/home/liam/Downloads/llvm-project-15.0.3.src/compiler-rt/test/tsan/../ -std=c++11 -I/home/liam/Downloads/llvm-project-15.0.3.src/compiler-rt/test/tsan/../ -nostdinc++ -I/home/liam/Downloads/llvm-project-15.0.3.src/build/runtimes/runtimes-bins/compiler-rt/lib/tsan/libcxx_tsan_x86_64/include/c++/v1 -O1 /home/liam/Downloads/llvm-project-15.0.3.src/compiler-rt/test/tsan/real_deadlock_detector_stress_test.cpp /home/liam/Downloads/llvm-project-15.0.3.src/build/runtimes/runtimes-bins/compiler-rt/lib/tsan/libcxx_tsan_x86_64/lib/libc++.a -o /home/liam/Downloads/llvm-project-15.0.3.src/build/runtimes/runtimes-bins/compiler-rt/test/tsan/X86_64Config/Output/real_deadlock_detector_stress_test.cpp.tmp &&  /home/liam/Downloads/llvm-project-15.0.3.src/build/runtimes/runtimes-bins/compiler-rt/test/tsan/X86_64Config/Output/real_deadlock_detector_stress_test.cpp.tmp 2>&1 | FileCheck /home/liam/Downloads/llvm-project-15.0.3.src/compiler-rt/test/tsan/real_deadlock_detector_stress_test.cpp
--
Exit Code: 1

Command Output (stderr):
--
clang-15: error: no such file or directory: '/home/liam/Downloads/llvm-project-15.0.3.src/build/runtimes/runtimes-bins/compiler-rt/lib/tsan/libcxx_tsan_x86_64/lib/libc++.a'

--

Build command:

cmake -S ../llvm -B . -G Ninja -DCMAKE_BUILD_TYPE=Release -DLLVM_TARGETS_TO_BUILD=X86 -DLLVM_ENABLE_PROJECTS="bolt;llvm;clang;lld;lldb;clang-tools-extra;cross-project-tests;libclc;polly;pstl" -DLLVM_ENABLE_RUNTIMES="openmp;libcxx;libcxxabi;libunwind;compiler-rt" -DCMAKE_C_COMPILER=clang -DCMAKE_CXX_COMPILER=clang++ -DLIBCXX_USE_COMPILER_RT=YES -DLLVM_USE_LINKER=lld -DLIBCXXABI_USE_LLVM_UNWINDER=YES -DLLVM_INSTALL_UTILS=ON -DCMAKE_INSTALL_PREFIX=$HOME/llvm15_temp

[asb]

Same issue for the following build command:

    cmake -G Ninja -DCMAKE_BUILD_TYPE="Release" \
      -DLLVM_ENABLE_PROJECTS="clang;clang-tools-extra;lld" \
      -DLLVM_ENABLE_RUNTIMES="compiler-rt;libcxx;libcxxabi" \
      -DLLVM_ENABLE_ASSERTIONS=ON \
      -DLLVM_CCACHE_BUILD=ON \
      -DCMAKE_C_COMPILER=clang -DCMAKE_CXX_COMPILER=clang++ \
      -DLLVM_ENABLE_LLD=True \
      -DLLVM_TARGETS_TO_BUILD="all" \
      ../../llvm

Failed tests (from ninja check-all):

Failed Tests (23):
  ThreadSanitizer-x86_64 :: Linux/dlopen_static_tls.cpp
  ThreadSanitizer-x86_64 :: atexit5.cpp
  ThreadSanitizer-x86_64 :: compare_exchange.cpp
  ThreadSanitizer-x86_64 :: custom_mutex0.cpp
  ThreadSanitizer-x86_64 :: custom_mutex1.cpp
  ThreadSanitizer-x86_64 :: custom_mutex2.cpp
  ThreadSanitizer-x86_64 :: custom_mutex3.cpp
  ThreadSanitizer-x86_64 :: custom_mutex4.cpp
  ThreadSanitizer-x86_64 :: custom_mutex5.cpp
  ThreadSanitizer-x86_64 :: deadlock_detector_stress_test.cpp
  ThreadSanitizer-x86_64 :: dl_iterate_phdr.cpp
  ThreadSanitizer-x86_64 :: dlclose.cpp
  ThreadSanitizer-x86_64 :: ignore_lib0.cpp
  ThreadSanitizer-x86_64 :: ignore_lib1.cpp
  ThreadSanitizer-x86_64 :: ignore_lib2.cpp
  ThreadSanitizer-x86_64 :: ignore_lib3.cpp
  ThreadSanitizer-x86_64 :: ignore_lib4.cpp
  ThreadSanitizer-x86_64 :: ignore_lib5.cpp
  ThreadSanitizer-x86_64 :: ignore_lib6.cpp
  ThreadSanitizer-x86_64 :: ignored-interceptors-mmap.cpp
  ThreadSanitizer-x86_64 :: libcxx/std_shared_ptr.cpp
  ThreadSanitizer-x86_64 :: load_shared_lib.cpp
  ThreadSanitizer-x86_64 :: real_deadlock_detector_stress_test.cpp

Almost all are due to missing libc++.a, but some are due to failed includes (e.g. <new>, <string>).

[pauljohnston2009]

same issue when running

./test-release.sh -release 17.0.2 -final -triple x86_64-linux-gnu -use-ninja

did you find any solution to this?

[pauljohnston2009]

I have found a few things, but still no real solution.

The missing files seem to be generated from "add_custom_libcxx"

llvm-project/compiler-rt/lib/tsan/CMakeLists.txt

Line 30 in 3c356ee

add_custom_libcxx(libcxx_tsan_${arch} ${LIBCXX_PREFIX}

This appears to be similar to the fuzzer tests, which are working fine (and generate the needed files)

https://github.com/llvm/llvm-project/blob/3c356eef31ab309466c198de3915037b068d8861/compiler-rt/lib/fuzzer/CMakeLists.txt#L161C5-L161C22

ie.

$ ls final/Phase3/Release/llvmCore-17.0.3-final.obj/runtimes/runtimes-bins/compiler-rt/lib/tsan/libcxx_tsan_x86_64/
src  tmp
$ ls final/Phase3/Release/llvmCore-17.0.3-final.obj/runtimes/runtimes-bins/compiler-rt/lib/fuzzer/libcxx_fuzzer_x86_64
bin  build.ninja  CMakeCache.txt  CMakeFiles  cmake_install.cmake  include  lib  libcxx  libcxxabi  lit.tests  llvm-lit  src  tmp

I can get the files to be generated (and the tests to pass) by doing

// wait for tests to fail
cd final/Phase3/Release/llvmCore-17.0.3-final.obj/
cmake .
ninja check-tsan 

note: The check-tsan target does not exist before running cmake ., and I have to let the tests fail first before cmake . will generate the check-tsan target.

From my understanding, cmake . should not change anything since it should use the same configuration from the cmake cache.

I also noticed that the tests compile fine without the extra args added by

llvm-project/compiler-rt/test/tsan/lit.cfg.py

Line 76 in 3c356ee

config.substitutions.append(("%link_libcxx_tsan", libcxx_a))

The -nostdinc++ and explicit link of "final/Phase3/Release/llvmCore-17.0.3-final.obj/runtimes/runtimes-bins/compiler-rt/lib/tsan/libcxx_tsan_x86_64/lib/libc++.a" is preventing it from using the libcxx from Phase3.

What is the reasoning for the custom libcxx? Is it fine to use the one from the Phase3 install directory?

Hopefully this is of some help.

[tstellar]

@dvyukov Do you have ideas about this? I'm seeing similar errors in the official release builds: https://github.com/llvm/llvm-project/actions/runs/6861861275/job/18658422688

[tstellar]

@petrhosek I know it's been a while since you worked on this, but do you have any ideas here?

[dvyukov]

No, I don't have any ideas. ninja check-tsan used to pass for me.

[pauljohnston2009]

I had a look at those logs, not sure if its related, but I am also getting the file differences in hwasan too. I tried doing a phase 4 to see if it was something like #11572 (comment)

# Comparing Phase 3 and Phase 4 files
file hwasan_tag_mismatch_riscv64.S.o differs between phase 3 and phase 4
file hwasan_setjmp_riscv64.S.o differs between phase 3 and phase 4
file hwasan_tag_mismatch_aarch64.S.o differs between phase 3 and phase 4
file hwasan_setjmp_aarch64.S.o differs between phase 3 and phase 4
file hwasan_tag_mismatch_riscv64.S.o differs between phase 3 and phase 4
file hwasan_setjmp_riscv64.S.o differs between phase 3 and phase 4
file hwasan_tag_mismatch_aarch64.S.o differs between phase 3 and phase 4
file hwasan_setjmp_aarch64.S.o differs between phase 3 and phase 4
file hwasan_tag_mismatch_riscv64.S.o differs between phase 3 and phase 4
file hwasan_setjmp_riscv64.S.o differs between phase 3 and phase 4
file hwasan_tag_mismatch_aarch64.S.o differs between phase 3 and phase 4
file hwasan_setjmp_aarch64.S.o differs between phase 3 and phase 4
file hwasan_tag_mismatch_riscv64.S.o differs between phase 3 and phase 4
file hwasan_setjmp_riscv64.S.o differs between phase 3 and phase 4
file hwasan_tag_mismatch_aarch64.S.o differs between phase 3 and phase 4
file hwasan_setjmp_aarch64.S.o differs between phase 3 and phase 4

Should I open another ticket for this? Do you think its safe to use the output even with the file differences and test failures?

[petrhosek]

@tstellar I'll try to reproduce this locally and will report back.

[tstellar]

@petrhosek I have a fix here: #72955