[libc++] Per-header header modules

[vgvassilev]

Commit 571178a introduces per header file module with a main motivation to enable detection of cyclic includes. This means that now, instead of having a single binary artifact std.pcm or similar we get hundreds of pcm files for the same content. That's problematic for us due to several reasons:

• Makes modules redistribution harder -- we carry build artifacts across nodes and it makes it difficult to enumerate all stl-related modules including private ones.
• Efficiency -- we care about binary size and in-memory footprint. Going for a many pcm files has an overhead. Consider we have several textual files (such as cassert). The textual content becomes duplicated in every pcm file and in all dependent pcm files too.

If the motivation is only to be able to diagnose cyclic includes we can do that as part of the build process of libc++. For example, echo "#include <libcxx/header>" | clang -xc++ -fsyntax-only - will be a sufficient check ensuring header consistency. Alternatively, we could have a "debug" modulemap with that structure.

If there is more motivation to the one I understood, I'd like to suggest having a separate modulemap, called std.modulemap which keeps the headers into a single pcm file and clients like us can use it instead. Note that the module std [system] { header "__std_clang_module" export * } does not work for us because it triggers building of all gazillion pcms.

@ian-twilightcoder, @ldionne, depending on the direction we get in this issue, I can work on creating the std.modulemap and opening a pull request.

cc: @hahnjo, @dpiparo, @ChuanqiXu9.

[llvmbot]

@llvm/issue-subscribers-clang-modules

Author: Vassil Vassilev (vgvassilev)

Commit 571178a introduces per header file module with a main motivation to enable detection of cyclic includes. This means that now, instead of having a single binary artifact `std.pcm` or similar we get hundreds of pcm files for the same content. That's problematic for us due to several reasons: * Makes modules redistribution harder -- we carry build artifacts across nodes and it makes it difficult to enumerate all stl-related modules including private ones. * Efficiency -- we care about binary size and in-memory footprint. Going for a many pcm files has an overhead. Consider we have several textual files (such as cassert). The textual content becomes duplicated in every `pcm` file and in all dependent `pcm` files too.

If the motivation is only to be able to diagnose cyclic includes we can do that as part of the build process of libc++. For example, echo "#include <libcxx/header>" | clang -xc++ -fsyntax-only - will be a sufficient check ensuring header consistency. Alternatively, we could have a "debug" modulemap with that structure.

If there is more motivation to the one I understood, I'd like to suggest having a separate modulemap, called std.modulemap which keeps the headers into a single pcm file and clients like us can use it instead. Note that the module std [system] { header "__std_clang_module" export * } does not work for us because it triggers building of all gazillion pcms.

@ian-twilightcoder, @ldionne, depending on the direction we get in this issue, I can work on creating the std.modulemap and opening a pull request.

cc: @hahnjo, @dpiparo, @ChuanqiXu9.

[EricWF]

I agree this is broken. I've had success wrapping all the submodules up in a single top level std module. This prevents the build race conditions I run into with LLVM, and I suspect would resolve your issues as well?

[vgvassilev]

I agree this is broken. I've had success wrapping all the submodules up in a single top level std module. This prevents the build race conditions I run into with LLVM, and I suspect would resolve your issues as well?

Do you have already a patch?

[ldionne]

My preference here would be to maintain a single modulemap and to simply gradually merge back modules without creating cycles. I suspect it is possible to drive towards a folder organization + modulemap that are consistent, don't introduce cycles and allow having decently sized submodules. If that's feasible, I think that would be the most viable alternative.

[vgvassilev]

@ldionne, I believe can detect cycles using different approaches (standalone per-header compilation or using the modulemap submodule infrastructure). Do I understand correctly your comment that we will have again a single module for all of stl?

[ian-twilightcoder]

It's not about include cycles. Each C standard library header has to be in its own top level module to support module layering between libc++, clang, and libc*. Take inttypes.h and stdint.h. Each libc++ header does an #include_next of the libc header. The problem arises when inttypes.h includes stdint.h. That creates two module cycles between the cxx and c modules: cxx:inttypes.h -> c:inttypes.h -> cxx:stdint.h and c:inttypes.h -> cxx:stdint.h -> c:stdint.h. The only way to break those two cycles is to make top level modules for all 4 headers: cxx_inttypes:inttypes.h -> c_inttypes:inttypes.h -> cxx_stdint:stdint.h -> c_stdint:stdint.h.

Each library has no way of knowing what the others' versions of the C standard library headers will include, so all of the C standard library headers in all libraries must be top level modules. In C++, the C standard library headers are quite intertwined with the C++ standard library headers by standard, and so those too need to be top level modules.

libc++ cannot go back to a single module for all of its headers. However, we can greatly reduce the number of modules by grouping the private implementation headers. I had a PR that did that programmatically https://reviews.llvm.org/D155141. That takes the module count from ~1000 to ~200. However, it was poorly received. I tried do something similar by hand, but it was utterly miserable to maintain because the layering is greatly perturbed every time an implementation header is added, or changes what it includes. Trying to break an 8+ module cycle is very difficult, the bad vertex is difficult to identify, and rearranging the modules to eliminate the bad vertex is also very tricky. Louis thinks it all can be done by hand, but if that's true then it requires someone with much more knowledge about how libc++ is implemented, organized, and layered than me.

* if your libc does not have a module map then you can't use clang modules. As far as I know, Apple is the only libc that has module maps. I'll leave it to @Bigcheese to explain the gory details of why a modular header can't include a non-modular header if anyone wants to know.

[vgvassilev]

@ian-twilightcoder, thank you for your post and background.

The only way to break those two cycles is to make top level modules for all 4 headers: cxx_inttypes:inttypes.h -> c_inttypes:inttypes.h -> cxx_stdint:stdint.h -> c_stdint:stdint.h.

Most of the times one can get away with that using modules re-exports. It might be a little more tricky to make it work across libraries. However, I am wondering how the single module approach worked until recently.

libc++ cannot go back to a single module for all of its headers.

That was the case before, right? I am wondering what has changed. FWIW, we are able to make libstdc++ having a single module (since version 4.9) and I am pretty sure it was more entangled than libc++ was or will ever be.

By the way, if I edit the libc++ modulemap file and wrap everything into a top-level module I get working single module. I am really confused. I am missing something...

[ldionne]

This is what I'd like us to do: #98214
I think we can alleviate 80% of the existing pain with minimal engineering, minimal changes to the code bases and minimal added complexity to the library. I might be wrong, but if I am wrong, then there's something fundamentally wrong with Clang modules because we are not doing anything that fancy in libc++.

Only the parts of the library that can be re-included from C headers should be tricky to implement and will probably require its own top-level modules, but I can't imagine that everything else also requires its own module.

[ian-twilightcoder]

I think you'll find that __algorithm, __chrono, __functional, __memory, __ranges, __tuple, __type_traits, __utility (and others) are too intertwined for a simple approach like that, but maybe you can do some more sophisticated cleanup so that they layer better and aren't a continuous source of module cycles as their headers change.

[ldionne]

That's the idea. I expect that only a few things in __algorithm, __chrono & friends might cause cycles, and those can (and should) probably extracted into some other module. Maybe they belong in some kind of __core module or whatever. I think we'd find out as we go.

@ian-twilightcoder If we introduce cycles, would we find out in our CI testing? In particular, would we find out about re-introducing cycles with the C stdlib headers in the SDK if we happened to break that, or would we only find out downstream a while later?

[ian-twilightcoder]

Yeah, the cycles will show up in the modules CI tests. As long as you keep the c stdlib headers as their own top level modules, you should remain acyclic with the underlying libc.

[vgvassilev]

Yeah, the cycles will show up in the modules CI tests. As long as you keep the c stdlib headers as their own top level modules, you should remain acyclic with the underlying libc.

But that was my point initially. This mode is mostly for making sure things do not regress in layering but that’s not necessarily the right way to ship a release. You can have that mode in the CI without duplicating too much modulemap infrastructure.