Bolt creating corrupted instrumented binary for a C++ binary that uses Boost library (arm64)

[jcgomezv]

I am trying to instrument a binary created by C++ that uses boost library arm64 architecture bolt tool crashes in _ZNK5boost16exception_detail10clone_implINS0_10bad_alloc_EE7rethrowEv, if I skip this bolts get to create binary but the resulting binary crashes with corrupted stack very early in the start:

Program received signal SIGBUS, Bus error.
0xddde490194029342 in ?? ()
(gdb) where
#0 0xddde490194029342 in ?? ()
#1 0x0000aaaaaf8cc730 in __libc_csu_init ()
#2 0x0000fffff4c71d4c in __libc_start_main () from /lib64/libc.so.6
#3 0x0000aaaaacb198c4 in _start ()
Backtrace stopped: previous frame identical to this frame (corrupt stack?)

[llvmbot]

@llvm/issue-subscribers-bolt

Author: None (jcgomezv)

I am trying to instrument a binary created by C++ that uses boost library arm64 architecture bolt tool crashes in _ZNK5boost16exception_detail10clone_implINS0_10bad_alloc_EE7rethrowEv, if I skip this bolts get to create binary but the resulting binary crashes with corrupted stack very early in the start:

Program received signal SIGBUS, Bus error.
0xddde490194029342 in ?? ()
(gdb) where
#0 0xddde490194029342 in ?? ()
#1 0x0000aaaaaf8cc730 in __libc_csu_init ()
#2 0x0000fffff4c71d4c in __libc_start_main () from /lib64/libc.so.6
#3 0x0000aaaaacb198c4 in _start ()
Backtrace stopped: previous frame identical to this frame (corrupt stack?)

[jcgomezv]

@ilinpv @peterwaller-arm : any idea how to work around this one?

[jcgomezv]

@paschalis-mpeis ?

[jcgomezv]

Command I am running to generate instrumented binary:
llvm-bolt csdd.orig -update-debug-sections -instrument -o csdd-inst-clear-v0 --skip-funcs="_start/1,__libc_start_main/1,__libc_csu_init/1,csdMain/1,startService/1,_ZNK5boost16exception_detail10clone_implINS0_10bad_alloc_EE7rethrowEv/1"

Original file details: (compiled with PIE)
file csdd.orig
csdd.orig: ELF 64-bit LSB shared object, ARM aarch64, version 1 (GNU/Linux), dynamically linked (uses shared libs), for GNU/Linux 3.7.0, not stripped

[jcgomezv]

g++ version: aarch64-unknown-linux-gnu-g++ (GCC) 10.5.0

[jcgomezv]

This is where I crash when I instrument not skipping any functions:
Binary Function "_init" after disassembly {
Number : 1
State : disassembled
Address : 0x117e08
Size : 0x14
MaxSize : 0x14
Offset : 0x117e08
Section : .init
Orc Section : .local.text._init
LSDA : 0x0
IsSimple : 1
IsMultiEntry: 0
IsSplit : 0
BB Count : 0
}
.LBB00:
00000000: stp x29, x30, [sp, #-0x10]!
00000004: mov x29, sp
00000008: bl "call_weak_fn/1" # Offset: 8
0000000c: ldp x29, x30, [sp], #0x10
00000010: ret # Offset: 16
DWARF CFI Instructions:

End of Function "_init"

Binary Function "_ZNK5boost16exception_detail10clone_implINS0_10bad_alloc_EE7rethrowEv/1(*2)" after disassembly {
All names : _ZNK5boost16exception_detail10clone_implINS0_10bad_alloc_EE7rethrowEv/1
_ZNK5boost16exception_detail10clone_implINS0_10bad_alloc_EE7rethrowEv/crtstuff.c/1
Number : 2
State : disassembled
Address : 0x11d9a0
Size : 0xac
MaxSize : 0xac
Offset : 0x11d9a0
Section : .text
Orc Section : .local.text._ZNK5boost16exception_detail10clone_implINS0_10bad_alloc_EE7rethrowEv/1
LSDA : 0x0
IsSimple : 1
IsMultiEntry: 0
IsSplit : 0
BB Count : 0
}
.LBB01:
00000000: stp x29, x30, [sp, #-0x20]!
00000004: mov x29, sp
00000008: stp x19, x20, [sp, #0x10]
0000000c: orr x20, xzr, x0

[ilinpv]

@jcgomezv would you mind to provide Bolt crash stack trace and Bolt log adding "-v 2" ?

[jcgomezv]

llvm-bolt csdd -update-debug-sections -instrument  -o csdd-inst-clear --print-disasm -v 2 > csdd-disasmllvm-bolt: /home/gomezjc/OpenSource/llvm-project/llvm/include/llvm/MC/MCInst.h:82: int64_t llvm::MCOperand::getImm() const: Assertion `isImm() && "This is not an immediate"' failed.
Stack dump without symbol names (ensure you have llvm-symbolizer in your PATH or set the environment var `LLVM_SYMBOLIZER_PATH` to point to it):
0  llvm-bolt       0x0000000000e58c10
1  llvm-bolt       0x0000000000e56660
2  linux-vdso.so.1 0x0000ffffaa01c834 __kernel_rt_sigreturn + 0
3  libc.so.6       0x0000ffffa9b3b834 gsignal + 180
4  libc.so.6       0x0000ffffa9b3d140 abort + 352
5  libc.so.6       0x0000ffffa9b34780
6  libc.so.6       0x0000ffffa9b347fc
7  llvm-bolt       0x00000000007ee240
8  llvm-bolt       0x000000000080f0b0
9  llvm-bolt       0x00000000015f42bc
10 llvm-bolt       0x000000000162fde0
11 llvm-bolt       0x0000000000f0aad0
12 llvm-bolt       0x0000000000f389e4
13 llvm-bolt       0x000000000040b4c8
14 libc.so.6       0x0000ffffa9b28da4 __libc_start_main + 228
15 llvm-bolt       0x00000000004865f0
PLEASE submit a bug report to https://github.com/llvm/llvm-project/issues/ and include the crash backtrace.
Stack dump:
0.      Program arguments: llvm-bolt csdd -update-debug-sections -instrument -o csdd-inst-clear --print-disasm -v 2
zsh: abort (core dumped)  llvm-bolt csdd -update-debug-sections -instrument -o csdd-inst-clear  -v 2 > 

[jcgomezv]

Apologies for the long delay was busy with the other project!

[jcgomezv]

Here is the actual output:

BOLT-INFO: setting size of function dlsym@PLT to 16 (was 0)
BOLT-INFO: setting size of function getrusage@PLT to 16 (was 0)
BOLT-INFO: setting size of function strtoll@PLT to 16 (was 0)
BOLT-INFO: setting size of function _ZStrsIcSt11char_traitsIcESaIcEERSt13basic_istreamIT_T0_ES7_RNSt7__cxx1112basic_stringIS4_S5_T1_EE@PLT to 16 (was 0)
BOLT-INFO: setting size of function times@PLT to 16 (was 0)
BOLT-INFO: setting size of function _ZNSi10_M_extractImEERSiRT_@PLT to 16 (was 0)
BOLT-INFO: setting size of function pthread_condattr_setpshared@PLT to 16 (was 0)
BOLT-INFO: setting size of function _ZN6google8protobuf2io17CodedOutputStream13WriteVarint64Em@PLT to 16 (was 0)
BOLT-INFO: setting size of function _ZN6google8protobuf55protobuf_AddDesc_google_2fprotobuf_2fdescriptor_2eprotoEv@PLT to 16 (was 0)
BOLT-INFO: setting size of function _ZN6google8protobuf12FieldOptions16default_instanceEv@PLT to 16 (was 0)
BOLT-INFO: setting size of function _ZN6google8protobuf8internal12ExtensionSet17RegisterExtensionEPKNS0_11MessageLiteEihbb@PLT to 16 (was 0)
BOLT-INFO: setting size of function _ZN6google8protobuf8internal12ExtensionSet21RegisterEnumExtensionEPKNS0_11MessageLiteEihbbPFbiE@PLT to 16 (was 0)
BOLT-INFO: setting size of function printf@PLT to 16 (was 0)
BOLT-INFO: setting size of function putchar@PLT to 16 (was 0)
BOLT-INFO: setting size of function isprint@PLT to 16 (was 0)
BOLT-INFO: setting size of function _start to 72 (was 0)
BOLT-INFO: setting size of function crc32_iscsi_refl_pmull to 48 (was 0)
BOLT-INFO: setting size of function _fini to 16 (was 0)
BOLT-INFO: 603 out of 603 CUs will be updated
Binary Function "_init" after disassembly {
  Number      : 1
  State       : disassembled
  Address     : 0x117da0
  Size        : 0x14
  MaxSize     : 0x14
  Offset      : 0x117da0
  Section     : .init
  Orc Section : .local.text._init
  LSDA        : 0x0
  IsSimple    : 1
  IsMultiEntry: 0
  IsSplit     : 0
  BB Count    : 0
}
.LBB00:
    00000000:   stp     x29, x30, [sp, #-0x10]!
    00000004:   mov     x29, sp
    00000008:   bl      "call_weak_fn/1" # Offset: 8
    0000000c:   ldp     x29, x30, [sp], #0x10
    00000010:   ret # Offset: 16
DWARF CFI Instructions:
    <empty>
End of Function "_init"


Binary Function "_ZNK5boost16exception_detail10clone_implINS0_10bad_alloc_EE7rethrowEv/1(*2)" after disassembly {
  All names   : _ZNK5boost16exception_detail10clone_implINS0_10bad_alloc_EE7rethrowEv/1
                _ZNK5boost16exception_detail10clone_implINS0_10bad_alloc_EE7rethrowEv/crtstuff.c/1
  Number      : 2
  State       : disassembled
  Address     : 0x11d940
  Size        : 0xac
  MaxSize     : 0xac
  Offset      : 0x11d940
  Section     : .text
  Orc Section : .local.text._ZNK5boost16exception_detail10clone_implINS0_10bad_alloc_EE7rethrowEv/1
  LSDA        : 0x0
  IsSimple    : 1
  IsMultiEntry: 0
  IsSplit     : 0
  BB Count    : 0
}
.LBB01:
    00000000:   stp     x29, x30, [sp, #-0x20]!
    00000004:   mov     x29, sp
    00000008:   stp     x19, x20, [sp, #0x10]
    0000000c:   orr     x20, xzr, x0

[jcgomezv]

(gdb) where
#0  0x0000fffff7b1b834 in raise () from /lib64/libc.so.6
#1  0x0000fffff7b1d140 in abort () from /lib64/libc.so.6
#2  0x0000fffff7b14780 in __assert_fail_base () from /lib64/libc.so.6
#3  0x0000fffff7b147fc in __assert_fail () from /lib64/libc.so.6
#4  0x00000000007ee240 in llvm::AArch64InstPrinter::printShifter(llvm::MCInst const*, unsigned int, llvm::MCSubtargetInfo const&, llvm::raw_ostream&) ()
#5  0x000000000080f0b0 in llvm::AArch64InstPrinter::printInst(llvm::MCInst const*, unsigned long, llvm::StringRef, llvm::MCSubtargetInfo const&, llvm::raw_ostream&) ()
#6  0x00000000015f42bc in llvm::bolt::BinaryContext::printInstruction(llvm::raw_ostream&, llvm::MCInst const&, unsigned long, llvm::bolt::BinaryFunction const*, bool, bool, bool, llvm::StringRef) const ()
#7  0x000000000162fde0 in llvm::bolt::BinaryFunction::print(llvm::raw_ostream&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) [clone .localalias] ()
#8  0x0000000000f0aad0 in llvm::bolt::RewriteInstance::disassembleFunctions() [clone .localalias] ()
#9  0x0000000000f389e4 in llvm::bolt::RewriteInstance::run() ()
#10 0x000000000040b4c8 in main ()
(gdb) bt app

[jcgomezv]

This command creates a binary but the binary is not viable:
llvm-bolt csdd -update-debug-sections -instrument -o csdd-inst-clear --skip-funcs="_ZNK5boost16exception_detail10clone_implINS0_10bad_alloc_EE7rethrowEv/1"

[jcgomezv]

I will get the trace for that crash, I thinkt it crashes right at init

[jcgomezv]

The binary created before crashes with corrupted stack like this:

(gdb) bt
#0 0x000000000000000e in ?? ()
#1 0x0000aaaacead8a70 in __libc_csu_init ()
#2 0x0000ffffaf7fed4c in __libc_start_main () from /lib64/libc.so.6
#3 0x0000aaaacbd28e68 in _start ()
Backtrace stopped: previous frame identical to this frame (corrupt stack?)