libc++ 18 iostream compatiblity #79933

Not planned

Labels

Since libc++13, iostream classes(for examplebasic_ifstream<char>) explicit instantiations exist in libc++ shared library. When building user code with libc++ <= 17, instantiations are in user code by default unless targeting _LIBCPP_ABI_VERSION >= 2, so we can build with libc++ 17 and run with libc++ < 13(for example android ndk r23, ubuntu 20.04). [In libc++ 18](

llvm-project/libcxx/include/fstream

Line 1765 in 12563ea

#if _LIBCPP_AVAILABILITY_HAS_ADDITIONAL_IOSTREAM_EXPLICIT_INSTANTIATIONS_1

), _LIBCPP_AVAILABILITY_HAS_ADDITIONAL_IOSTREAM_EXPLICIT_INSTANTIATIONS_1 is always 1 for linux, so instantiations are not in user code, and produced binary depends on new symbols in libc++ shared library. This can reduce user binary size, but breaks libc++ compatibility, users must redistribute libc++ with their apps or SDKs. But what if an app depends on multiple 3rdparty SDKs built with different libc++ versions? Let the SDK users find out the highest version from these SDKs(how?) and delete others to avoid link error?

My suggestion is

#if defined(_LIBCPP_BUILDING_LIBRARY) || _LIBCPP_ABI_VERSION >= 2
# define _LIBCPP_AVAILABILITY_HAS_ADDITIONAL_IOSTREAM_EXPLICIT_INSTANTIATIONS_1 1
#else
# define _LIBCPP_AVAILABILITY_HAS_ADDITIONAL_IOSTREAM_EXPLICIT_INSTANTIATIONS_1 0
#endif

just like what we do for _LIBCPP_ABI_BAD_FUNCTION_CALL_KEY_FUNCTION

added

mentioned this

[libc++][NFC] Refactor _LIBCPP_AVAILABILITY_HAS_* macros to always be defined #71002

ldionne

Member

If you want to support back-deploying on Linux platforms, you should add availability markup that defines when various symbols were introduced to such platform. We have all the "infrastructure" to support it in place already, it just happens that only Apple seems to take advantage of it: https://github.com/llvm/llvm-project/blob/main/libcxx/include/__availability#L19

ldionne

Member

This can reduce user binary size, but breaks libc++ compatibility, users must redistribute libc++ with their apps or SDKs. But what if an app depends on multiple 3rdparty SDKs built with different libc++ versions? Let the SDK users find out the highest version from these SDKs(how?) and delete others to avoid link error?

I don't understand this. If a SDK on Linux is shipping libc++, it needs to be shipping both the headers and the .so (or .a). In that case, the vendor owning the SDK should set up availability markup as described above for their SDK / platform.

wang-bin

Author

If you want to support back-deploying on Linux platforms, you should add availability markup that defines when various symbols were introduced to such platform. We have all the "infrastructure" to support it in place already, it just happens that only Apple seems to take advantage of it: https://github.com/llvm/llvm-project/blob/main/libcxx/include/__availability#L19

linux is different from apple, there are many distros, for example debian, ubuntu, arch etc. none of them supports availability markup

I don't understand this. If a SDK on Linux is shipping libc++, it needs to be shipping both the headers and the .so (or .a). In that case, the vendor owning the SDK should set up availability markup as described above for their SDK / platform.

No, only SDK header and .so. c++ headers and libs come from toolchain. If c++ headers and .so must be shipped, what about other dependencies, for example libc?

philnik777

Contributor

If you want to support back-deploying on Linux platforms, you should add availability markup that defines when various symbols were introduced to such platform. We have all the "infrastructure" to support it in place already, it just happens that only Apple seems to take advantage of it: https://github.com/llvm/llvm-project/blob/main/libcxx/include/__availability#L19

linux is different from apple, there are many distros, for example debian, ubuntu, arch etc. none of them supports availability markup

Yes, there are lots of distros. We are very aware of that, and we are aware that they don't have availability markups. If you want backwards compatibility support, you have to maintain the markups. Otherwise we have the same policy as libstdc++: you don't get backwards compatibility.

ldionne

Member

linux is different from apple, there are many distros, for example debian, ubuntu, arch etc. none of them supports availability markup

The reason why none of them support availability markup is likely that none of these distributions support back-deploying. Like I said, we have the infrastructure in place to support it if desired, I made sure to write it in a vendor-agnostic way when I refactored that area of the codebase a few years ago.

I don't understand this. If a SDK on Linux is shipping libc++, it needs to be shipping both the headers and the .so (or .a). In that case, the vendor owning the SDK should set up availability markup as described above for their SDK / platform.

No, only SDK header and .so. c++ headers and libs come from toolchain. If c++ headers and .so must be shipped, what about other dependencies, for example libc?

I don't understand the vending model you're implying here. Why would you ship the .so in the SDK but the headers in the toolchain? I don't understand how that makes sense. Note that this is what we used to do at Apple and we moved away from it so we now ship everything in the SDK instead. The other option is to ship everything in the toolchain, but I don't understand why it makes sense to ship the .so in the SDK but the headers in the toolchain.

wang-bin

Author

linux is different from apple, there are many distros, for example debian, ubuntu, arch etc. none of them supports availability markup

The reason why none of them support availability markup is likely that none of these distributions support back-deploying. Like I said, we have the infrastructure in place to support it if desired, I made sure to write it in a vendor-agnostic way when I refactored that area of the codebase a few years ago.

Linux is just an example. The issue is about libc++ comaptibility. I want to build my library using the latest libc++, and run with old libc++. I think that's why _LIBCPP_ABI_BAD_FUNCTION_CALL_KEY_FUNCTION exists. BTW, I solve the problem by explicitly instantiating these classes in my code, but I have to add instantiation code in every shared library and executable, which is not convenient. Before libc++18, I only need to add some build flags, then the result binary is compatible with all libc++ versions.

I don't understand this. If a SDK on Linux is shipping libc++, it needs to be shipping both the headers and the .so (or .a). In that case, the vendor owning the SDK should set up availability markup as described above for their SDK / platform.

No, only SDK header and .so. c++ headers and libs come from toolchain. If c++ headers and .so must be shipped, what about other dependencies, for example libc?

I don't understand the vending model you're implying here. Why would you ship the .so in the SDK but the headers in the toolchain? I don't understand how that makes sense. Note that this is what we used to do at Apple and we moved away from it so we now ship everything in the SDK instead. The other option is to ship everything in the toolchain, but I don't understand why it makes sense to ship the .so in the SDK but the headers in the toolchain.

Because a 3rdparty SDK(for example Qt) doesn't ship libc++/libstdc++, c++ is provided by toolchain. Apple provides everything in the SDK because it's the system SDK, not a 3rdparty SDK. Shipping libc++ in a 3rdparty SDK is not my idea, it's from #71002 (comment)

philnik777

Contributor

linux is different from apple, there are many distros, for example debian, ubuntu, arch etc. none of them supports availability markup

The reason why none of them support availability markup is likely that none of these distributions support back-deploying. Like I said, we have the infrastructure in place to support it if desired, I made sure to write it in a vendor-agnostic way when I refactored that area of the codebase a few years ago.

Linux is just an example. The issue is about libc++ comaptibility. I want to build my library using the latest libc++, and run with old libc++. I think that's why _LIBCPP_ABI_BAD_FUNCTION_CALL_KEY_FUNCTION exists. BTW, I solve the problem by explicitly instantiating these classes in my code, but I have to add instantiation code in every shared library and executable, which is not convenient. Before libc++18, I only need to add some build flags, then the result binary is compatible with all libc++ versions.

We understand what you want to do, and we say that the solution is to add availability markups to libc++. Without markups we're unable to support that, since we'd be unable to extend the dylib. _LIBCPP_ABI_BAD_FUNCTION_CALL_KEY_FUNCTION should also be changed to an availability feature, but that's a bit more complicated. It will most likely happen in the next few releases.

ldionne

Member

Closing since I don't think there's more to add here. The path to "fix" this is clear, we have everything in place to support that, but what this issue is requesting (removing the iostreams instantiations by default) is not something we will pursue.