miscompiled TLS wrapper, likely coroutine related, with sanitizer

[avikivity]

Prior to 533b7c1, compiling some coroutine that references a TLS variable generates this wrapper:

0000000000029d80 <_ZTWN2db13schema_tablesL14the_merge_lockE>:
   29d80: 50                           	pushq	%rax
   29d81: e8 4a b3 10 00               	callq	0x1350d0 <__tls_init>
   29d86: 64 48 8b 04 25 00 00 00 00   	movq	%fs:0x0, %rax
   29d8f: 48 8d 80 00 00 00 00         	leaq	(%rax), %rax
		0000000000029d92:  R_X86_64_TPOFF32	_ZN2db13schema_tablesL14the_merge_lockE
   29d96: 59                           	popq	%rcx
   29d97: c3                           	retq
   29d98: 0f 1f 84 00 00 00 00 00      	nopl	(%rax,%rax)

It correctly calls __tls_init.

With 533b7c1 and later, up to 18.1.1, the following TLS wrapper is generated:

0000000000029d80 <_ZTWN2db13schema_tablesL14the_merge_lockE>:
   29d80: 50                           	pushq	%rax
   29d81: e8 00 00 00 00               	callq	0x29d86 <_ZTWN2db13schema_tablesL14the_merge_lockE+0x6>
		0000000000029d82:  R_X86_64_PLT32	_ZTH15data_type_for_vIN7seastar13basic_sstringIcjLj15ELb1EEEE-0x4
   29d86: 64 48 8b 04 25 00 00 00 00   	movq	%fs:0x0, %rax
   29d8f: 48 8d 80 00 00 00 00         	leaq	(%rax), %rax
		0000000000029d92:  R_X86_64_TPOFF32	_ZN2db13schema_tablesL14the_merge_lockE
   29d96: 59                           	popq	%rcx
   29d97: c3                           	retq
   29d98: 0f 1f 84 00 00 00 00 00      	nopl	(%rax,%rax)

The call to __tls_init was replaced by a call to some random function. When the coroutine is then called, it does not initialize the object.

I will follow up with a full reproducer.

[avikivity]

Copying @dianqk, the author of the patch.

[tchaikov]

another data point is that if the source is compiled with -O0, the issue goes away. i guess the reason is that -O0 disables the globalopt pass.

[avikivity]

Reproducer:

wget https://scratch.scylladb.com/schema_tables.ii


clang++ -MD -MT build/debug/db/schema_tables.o -MF build/debug/db/schema_tables.o.d -std=c++20 -I/home/avi/scylla/seastar/include -I/home/avi/scylla/build/debug/seastar/gen/include -U_FORTIFY_SOURCE -Werror=unused-result -fstack-clash-protection -fsanitize=address -fsanitize=undefined -fno-sanitize=vptr -DSEASTAR_API_LEVEL=7 -DSEASTAR_BUILD_SHARED_LIBS -DSEASTAR_SSTRING -DSEASTAR_LOGGER_COMPILE_TIME_FMT -DSEASTAR_SCHEDULING_GROUPS_COUNT=16 -DSEASTAR_DEBUG -DSEASTAR_DEFAULT_ALLOCATOR -DSEASTAR_SHUFFLE_TASK_QUEUE -DSEASTAR_DEBUG_SHARED_PTR -DSEASTAR_DEBUG_PROMISE -DSEASTAR_LOGGER_TYPE_STDOUT -DSEASTAR_TYPE_ERASE_MORE -DFMT_SHARED -I/usr/include/p11-kit-1 -ffile-prefix-map=/home/avi/scylla=. -march=westmere -DDEBUG -DSANITIZE -DDEBUG_LSA_SANITIZER -DSCYLLA_ENABLE_ERROR_INJECTION -Og -DSCYLLA_BUILD_MODE=debug -g -gz -iquote. -iquote build/debug/gen -std=gnu++20  -ffile-prefix-map=/home/avi/scylla=. -march=westmere -DBOOST_ALL_DYN_LINK    -fvisibility=hidden -isystem abseil -Wall -Werror -Wextra -Wimplicit-fallthrough -Wno-mismatched-tags -Wno-c++11-narrowing -Wno-overloaded-virtual -Wno-unused-parameter -Wno-unsupported-friend -Wno-missing-field-initializers -Wno-deprecated-copy -Wno-psabi -Wno-enum-constexpr-conversion -Wno-error=deprecated-declarations -DXXH_PRIVATE_API -DSEASTAR_TESTING_MAIN  -S  schema_tables.ii -o schema_tables.s

grep -A10 '_ZTWN2db13schema_tablesL14the_merge_lockE.*:' schema_tables.s

[dianqk]

Thanks. I can reproduce it. The bisected outcome might not be the issue itself, but I'll investigate this first.

[avikivity]

Simplified command line:

clang++    -fsanitize=address       -Og    -std=gnu++20       -Wno-c++11-narrowing -Wno-unsupported-friend  -S  schema_tables.ii -o schema_tables.s && grep -A10 '_ZTWN2db13schema_tablesL14the_merge_lockE.*:' schema_tables.s

I verified that -fsanitize=address is required.

[avikivity]

Yes it's probably coroutines not emitting llvm.used. I remember bugs in this area.

[llvmbot]

@llvm/issue-subscribers-coroutines

Author: Avi Kivity (avikivity)

Prior to 533b7c1, compiling some coroutine that references a TLS variable generates this wrapper:

0000000000029d80 <_ZTWN2db13schema_tablesL14the_merge_lockE>:
   29d80: 50                           	pushq	%rax
   29d81: e8 4a b3 10 00               	callq	0x1350d0 <__tls_init>
   29d86: 64 48 8b 04 25 00 00 00 00   	movq	%fs:0x0, %rax
   29d8f: 48 8d 80 00 00 00 00         	leaq	(%rax), %rax
		0000000000029d92:  R_X86_64_TPOFF32	_ZN2db13schema_tablesL14the_merge_lockE
   29d96: 59                           	popq	%rcx
   29d97: c3                           	retq
   29d98: 0f 1f 84 00 00 00 00 00      	nopl	(%rax,%rax)

It correctly calls __tls_init.

With 533b7c1 and later, up to 18.1.1, the following TLS wrapper is generated:

0000000000029d80 <_ZTWN2db13schema_tablesL14the_merge_lockE>:
   29d80: 50                           	pushq	%rax
   29d81: e8 00 00 00 00               	callq	0x29d86 <_ZTWN2db13schema_tablesL14the_merge_lockE+0x6>
		0000000000029d82:  R_X86_64_PLT32	_ZTH15data_type_for_vIN7seastar13basic_sstringIcjLj15ELb1EEEE-0x4
   29d86: 64 48 8b 04 25 00 00 00 00   	movq	%fs:0x0, %rax
   29d8f: 48 8d 80 00 00 00 00         	leaq	(%rax), %rax
		0000000000029d92:  R_X86_64_TPOFF32	_ZN2db13schema_tablesL14the_merge_lockE
   29d96: 59                           	popq	%rcx
   29d97: c3                           	retq
   29d98: 0f 1f 84 00 00 00 00 00      	nopl	(%rax,%rax)

The call to __tls_init was replaced by a call to some random function. When the coroutine is then called, it does not initialize the object.

I will follow up with a full reproducer.

[dianqk]

Simplified command line:

clang++    -fsanitize=address       -Og    -std=gnu++20       -Wno-c++11-narrowing -Wno-unsupported-friend  -S  schema_tables.ii -o schema_tables.s && grep -A10 '_ZTWN2db13schema_tablesL14the_merge_lockE.*:' schema_tables.s

I verified that -fsanitize=address is required.

It should also need -fvisibility=hidden.

[dianqk]

I haven't found any obvious issues, unless I missed something.

_ZTH15data_type_for_vIN7seastar13basic_sstringIcjLj15ELb1EEEE is an alias for __tls_init:

@_ZTHN2db13schema_tablesL14the_merge_lockE = internal alias void (), ptr @__tls_init
@_ZTH15data_type_for_vIN7seastar13basic_sstringIcjLj15ELb1EEEE = linkonce_odr alias void (), ptr @__tls_init
@_ZTH15data_type_for_vIiE = linkonce_odr alias void (), ptr @__tls_init

Perhaps we need a runtime issue reproduction?

[avikivity]

Aha. The runtime reproducer is quite heavy. If you want, I can provide a pre-built docker image and instructions to expose the problem with gdb.