[C++20] [Modules] Undefined reference to std::allocator<char>::deallocate in Clang 19.0.0git when a function exported from a module returns a type template instantiation which uses std::allocator<char>

[ashley-hawkins]

Sorry if the title is not very clear. Basically if in my main.cc I'm importing a module, and that module exports a function which returns std::string, and I call that function from main.cc, it will fail to compile on latest clang with a linker error that

/opt/compiler-explorer/gcc-snapshot/lib/gcc/x86_64-linux-gnu/14.0.1/../../../../x86_64-linux-gnu/bin/ld: CMakeFiles/repro.dir/main.cc.o: in function `std::allocator_traits<std::allocator<char> >::deallocate(std::allocator<char>&, char*, unsigned long)':
/opt/compiler-explorer/gcc-snapshot/lib/gcc/x86_64-linux-gnu/14.0.1/../../../../include/c++/14.0.1/bits/alloc_traits.h:513: undefined reference to `std::allocator<char>::deallocate(char*, unsigned long)'
clang++: error: linker command failed with exit code 1 (use -v to see invocation)

I tried to make a minimal reproducer but I'm not sure whether or not this is actually minimal, because I don't know if it could be reproduced without std::allocator or if it's specifically a problem with std::allocator. In module.cc you can change the return type of repro::repro to std::string to see that it also happens with std::string. It also happens with std::vector<char>, and std::deque<char>, and I guess any other standard container would have the same problem with an element type of char. I wasn't able to reproduce without using modules, which is why I labelled it as a modules problem.

Compiler Explorer reproducer for 19.0.0git: https://godbolt.org/z/5GbczMEa7
Compiler Explorer example of inability to reproduce for 18.1.0: https://godbolt.org/z/EEajTa1nj (same code as in above I just changed the compiler to x86-64 Clang 18.1.0)

As of writing this:

Problematic version of Clang tested on Compiler Explorer:

clang version 19.0.0git (https://github.com/llvm/llvm-project.git c43932e)

Problematic version of Clang that I originally found the problem on, from apt.llvm.org:

Debian clang version 19.0.0 (++20240327112225+c9d12664f2f9-1~exp1~20240327112239.613) c9d1266

Working version of Clang tested on Compiler Explorer:

clang version 18.1.0rc (https://github.com/llvm/llvm-project.git 461274b)

[llvmbot]

@llvm/issue-subscribers-clang-modules

Author: None (ashley-hawkins)

Sorry if the title is not very clear. Basically if in my main.cc I'm importing a module, and that module exports a function which returns std::string, and I call that function from main.cc, it will fail to compile on latest clang with a linker error that ``` /opt/compiler-explorer/gcc-snapshot/lib/gcc/x86_64-linux-gnu/14.0.1/../../../../x86_64-linux-gnu/bin/ld: CMakeFiles/repro.dir/main.cc.o: in function `std::allocator_traits<std::allocator<char> >::deallocate(std::allocator<char>&, char*, unsigned long)': /opt/compiler-explorer/gcc-snapshot/lib/gcc/x86_64-linux-gnu/14.0.1/../../../../include/c++/14.0.1/bits/alloc_traits.h:513: undefined reference to `std::allocator<char>::deallocate(char*, unsigned long)' clang++: error: linker command failed with exit code 1 (use -v to see invocation) ```

I tried to make a minimal reproducer but I'm not sure whether or not this is actually minimal, because I don't know if it could be reproduced without std::allocator or if it's specifically a problem with std::allocator. In module.cc you can change the return type of repro::repro to std::string to see that it also happens with std::string. It also happens with std::vector<char>, and std::deque<char>, and I guess any other standard container would have the same problem with an element type of char. I wasn't able to reproduce without using modules, which is why I labelled it as a modules problem.

Compiler Explorer reproducer for 19.0.0git: https://godbolt.org/z/5GbczMEa7
Compiler Explorer example of inability to reproduce for 18.1.0: https://godbolt.org/z/EEajTa1nj (same code as in above I just changed the compiler to x86-64 Clang 18.1.0)

As of writing this:

Problematic version of Clang tested on Compiler Explorer:
> clang version 19.0.0git (https://github.com/llvm/llvm-project.git c43932e)

Problematic version of Clang that I originally found the problem on, from apt.llvm.org:
> Debian clang version 19.0.0 (++20240327112225+c9d12664f2f9-1~exp1~20240327112239.613) c9d1266

Working version of Clang tested on Compiler Explorer:
> clang version 18.1.0rc (https://github.com/llvm/llvm-project.git 461274b)

[ChuanqiXu9]

I suspect this is not a compiler's issue but a standard library issue.

See my workaround and the comments in https://github.com/alibaba/async_simple/blob/d6f201192bb332fc9dabcfdf3fa29e85c83c7f45/third_party_module/stdmodules/string.cppm#L28-L42.

And it works for https://godbolt.org/z/7Y3r7sM66.

Also the problem disappear too if we're using libc++: https://godbolt.org/z/oPrY3Y8Es

So I think this is a libstdc++ issue. I'll close this as not planned. Feel free to comment or reopen this.

[ashley-hawkins]

I did a git bisect, and observed that the issue only appears in 1ecbab5 and later.

Is this change just revealing a problem with libstdc++? If it's the case, I will see if it needs to be reported to libstdc++'s bug tracker.

[davidstone]

Historically, clang has worked around issues with libstdc++ so that clang users are not forced to use libc++ (that might not be possible for ABI reasons).

That being said, I'm not sure exactly what the issue is here. I see extern template allocator<char>; in allocator.h and I see template allocator<char> in allocator-inst.cc. I don't know how it could not work not but it used to work before -- I would expect both or neither. This makes me suspect this is a clang issue?

If I'm understanding the linked commit (1ecbab5) correctly, then I'm also surprised by that behavior. It seems like that commit turns something that literally says "always inline this" into "unless this only under these conditions". I would expect always_inline to be stronger than inline, not stronger within a module but weaker between modules. There is a reference to ABI requirements in that commit, but I don't see the issue here -- someone who marks a function always_inline isn't going for ABI stability as they change that function. Does this mean that people who want the 'old' behavior would have to say both inline and always_inline?

[ChuanqiXu9]

That being said, I'm not sure exactly what the issue is here. I see extern template allocator; in allocator.h and I see template allocator in allocator-inst.cc. I don't know how it could not work not but it used to work before -- I would expect both or neither. This makes me suspect this is a clang issue?

This should not be relevant to clang. Since I can't see the symbols in libstdc++.so, which should be compiled from GCC. So I believe this may be a GCC defect. But,

Historically, clang has worked around issues with libstdc++ so that clang users are not forced to use libc++ (that might not be possible for ABI reasons).

Yeah, I've received multiple complaints about this too. So probably I need to do some workaround in clang even if I think it is a GCC issue.

If I'm understanding the linked commit (1ecbab5) correctly, then I'm also surprised by that behavior. It seems like that commit turns something that literally says "always inline this" into "unless this only under these conditions". I would expect always_inline to be stronger than inline, not stronger within a module but weaker between modules. There is a reference to ABI requirements in that commit, but I don't see the issue here -- someone who marks a function always_inline isn't going for ABI stability as they change that function. Does this mean that people who want the 'old' behavior would have to say both inline and always_inline?

My thought was, the inline key words in C++ standard means linkage, and the always_inline attribute is an extension to pass the information to the optimizer. They looks similar but not the same thing from my side. But probably I have to add back the original behavior due to the reason above.

[iains]

That being said, I'm not sure exactly what the issue is here. I see extern template allocator; in allocator.h and I see template allocator in allocator-inst.cc. I don't know how it could not work not but it used to work before -- I would expect both or neither. This makes me suspect this is a clang issue?

This should not be relevant to clang. Since I can't see the symbols in libstdc++.so, which should be compiled from GCC. So I believe this may be a GCC defect. But,

Please could you file a Bugzilla for GCC/libstc++ (https://gcc.gnu.org/bugzilla/) in that case; there is an intent that the compilers and libraries can interoperate.

[jwakely]

Since I can't see the symbols in libstdc++.so, which should be compiled from GCC.

The functions that generate those symbols only exist for __cplusplus >= 202002L because they only need to exist for C++20 constexpr (de)allocations, which must be (de)allocated by std::allocator<T>::allocate and not some_base_class::allocate. So for C++20 there are always_inline forwarding functions that just call the base class allocate and deallocate. They should not need to be exported from the .so because nobody ever depends on them: they're always inlined.

It seems that modules changes this picture, and extern symbols are now needed for these always_inline functions.

Compiling the explicit instantiation definitions as C++20 instead of C++98 causes the explicit instantiation definitions to include those functions, so they're in the .so

That change might be OK for GCC 14 still, but backporting to GCC 13 would be complicated.