Skip to content

clang-12 strangely optimize a loop into recursive call to main function #54528

Closed
@wierton

Description

@wierton

the code:

int loop_1 = 100;
int loop_2 = 7;
int flag = 0;

int test(void) {
  int i = 0;
  int counter = 0;

  while (counter < 1) {  // note: replace counter < 1 by true won't trigger segfault
    if (flag & 1) {
    }
    flag++;
  }
  return 1;
}

int main() {
  if (test() != 1) abort();
  return 0;
}

the above code has an infinite loop in function test, however when compile it with clang-12 -O1, the produced executable will immediately segfault.

I disassembled the a.out and find that:

0000000000401110 <main>:
  401110:	50                   	push   %rax
  401111:	e8 fa ff ff ff       	callq  401110 <main>
  401116:	66 2e 0f 1f 84 00 00 	nopw   %cs:0x0(%rax,%rax,1)
  40111d:	00 00 00 

clang-12 optimize the above code into a strange recursive call of main function and finally cause stack overflow.

I know that unterminated loop is undefined behavior, theoretically compiler can do anything, but such optimization is still counterintuitive.

Not only this, while the loop condition counter < 1 is changed into 1, this two versions should be identical, but clang-12 reasonably optimize it into infinite loop.

A much more strange case is that, when enable -O2 for clang-12, clang-12 will directly optimize out the whole main function, this cause gdb failed to break on main function:

$ clang-12 -O2 a.c -o a.out -ggdb3
$ objdump -d a.out | grep 'main>'
$ gdb -q ./a.out
Reading symbols from ./a.out...
(gdb) b main
Breakpoint 1 at 0x401110
(gdb) r
Starting program: ~/a.out 

Breakpoint 1, 0x0000000000401110 in __libc_csu_init ()
(gdb)

Metadata

Metadata

Assignees

No one assigned

    Labels

    llvm:optimizationsquestionA question, not bug report. Check out https://llvm.org/docs/GettingInvolved.html instead!

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions