[C++20] [Modules] Support module level lookup (#122887)

Close #90154

This patch is also an optimization to the lookup process to utilize the
information provided by `export` keyword.

Previously, in the lookup process, the `export` keyword only takes part
in the check part, it doesn't get involved in the lookup process. That
said, previously, in a name lookup for 'name', we would load all of
declarations with the name 'name' and check if these declarations are
valid or not. It works well. But it is inefficient since it may load
declarations that may not be wanted.

Note that this patch actually did a trick in the lookup process instead
of bring module information to DeclarationName or considering module
information when deciding if two declarations are the same. So it may
not be a surprise to me if there are missing cases. But it is not a
regression. It should be already the case. Issue reports are welcomed.

In this patch, I tried to split the big lookup table into a lookup table
as before and a module local lookup table, which takes a combination of
the ID of the DeclContext and hash value of the primary module name as
the key. And refactored `DeclContext::lookup()` method to take the
module information. So that a lookup in a DeclContext won't load
declarations that are local to **other** modules.

And also I think it is already beneficial to split the big lookup table
since it may reduce the conflicts during lookups in the hash table.

BTW, this patch introduced a **regression** for a reachability rule in
C++20 but it was false-negative. See
'clang/test/CXX/module/module.interface/p7.cpp' for details.

This patch is not expected to introduce any other
regressions for non-c++20-modules users since the module local lookup
table should be empty for them.

---

On the API side, this patch unfortunately add a maybe-confusing argument
`Module *NamedModule` to
`ExternalASTSource::FindExternalVisibleDeclsByName()`. People may think
we can get the information from the first argument `const DeclContext
*DC`. But sadly there are declarations (e.g., namespace) can appear in
multiple different modules as a single declaration. So we have to add
additional information to indicate this.

Author: ChuanqiXu9

clang/docs/ReleaseNotes.rst

@@ -316,6 +316,8 @@ C++23 Feature Support
 C++20 Feature Support
 ^^^^^^^^^^^^^^^^^^^^^
 
+- Implemented module level lookup for C++20 modules. (#GH90154)
+
 
 Resolutions to C++ Defect Reports
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

clang/include/clang/AST/DeclBase.h

@@ -836,6 +836,10 @@ class alignas(8) Decl {
     return isFromASTFile() ? getImportedOwningModule() : getLocalOwningModule();
   }
 
+  /// Get the top level owning named module that owns this declaration if any.
+  /// \returns nullptr if the declaration is not owned by a named module.
+  Module *getTopLevelOwningNamedModule() const;
+
   /// Get the module that owns this declaration for linkage purposes.
   /// There only ever is such a standard C++ module.
   Module *getOwningModuleForLinkage() const;

clang/include/clang/Serialization/ASTBitCodes.h

@@ -738,6 +738,8 @@ enum ASTRecordTypes {
   CXX_ADDED_TEMPLATE_SPECIALIZATION = 74,
 
   CXX_ADDED_TEMPLATE_PARTIAL_SPECIALIZATION = 75,
+
+  UPDATE_MODULE_LOCAL_VISIBLE = 76,
 };
 
 /// Record types used within a source manager block.
@@ -1334,6 +1336,10 @@ enum DeclCode {
   /// into a DeclContext via DeclContext::lookup.
   DECL_CONTEXT_VISIBLE,
 
+  /// A record containing the set of declarations that are
+  /// only visible from DeclContext in the same module.
+  DECL_CONTEXT_MODULE_LOCAL_VISIBLE,
+
   /// A LabelDecl record.
   DECL_LABEL,
 

clang/include/clang/Serialization/ASTReader.h

@@ -353,6 +353,7 @@ class ASTIdentifierLookupTrait;
 
 /// The on-disk hash table(s) used for DeclContext name lookup.
 struct DeclContextLookupTable;
+struct ModuleLocalLookupTable;
 
 /// The on-disk hash table(s) used for specialization decls.
 struct LazySpecializationInfoLookupTable;
@@ -523,9 +524,14 @@ class ASTReader
   /// in the chain.
   DeclUpdateOffsetsMap DeclUpdateOffsets;
 
+  struct LookupBlockOffsets {
+    uint64_t LexicalOffset;
+    uint64_t VisibleOffset;
+    uint64_t ModuleLocalOffset;
+  };
+
   using DelayedNamespaceOffsetMapTy =
-      llvm::DenseMap<GlobalDeclID, std::pair</*LexicalOffset*/ uint64_t,
-                                             /*VisibleOffset*/ uint64_t>>;
+      llvm::DenseMap<GlobalDeclID, LookupBlockOffsets>;
 
   /// Mapping from global declaration IDs to the lexical and visible block
   /// offset for delayed namespace in reduced BMI.
@@ -631,6 +637,9 @@ class ASTReader
   /// Map from a DeclContext to its lookup tables.
   llvm::DenseMap<const DeclContext *,
                  serialization::reader::DeclContextLookupTable> Lookups;
+  llvm::DenseMap<const DeclContext *,
+                 serialization::reader::ModuleLocalLookupTable>
+      ModuleLocalLookups;
 
   using SpecLookupTableTy =
       llvm::DenseMap<const Decl *,
@@ -659,6 +668,8 @@ class ASTReader
   /// Updates to the visible declarations of declaration contexts that
   /// haven't been loaded yet.
   llvm::DenseMap<GlobalDeclID, DeclContextVisibleUpdates> PendingVisibleUpdates;
+  llvm::DenseMap<GlobalDeclID, DeclContextVisibleUpdates>
+      PendingModuleLocalVisibleUpdates;
 
   using SpecializationsUpdate = SmallVector<UpdateData, 1>;
   using SpecializationsUpdateMap =
@@ -696,7 +707,8 @@ class ASTReader
   /// Read the record that describes the visible contents of a DC.
   bool ReadVisibleDeclContextStorage(ModuleFile &M,
                                      llvm::BitstreamCursor &Cursor,
-                                     uint64_t Offset, GlobalDeclID ID);
+                                     uint64_t Offset, GlobalDeclID ID,
+                                     bool IsModuleLocal);
 
   bool ReadSpecializations(ModuleFile &M, llvm::BitstreamCursor &Cursor,
                            uint64_t Offset, Decl *D, bool IsPartial);
@@ -1132,6 +1144,10 @@ class ASTReader
   /// Number of visible decl contexts read/total.
   unsigned NumVisibleDeclContextsRead = 0, TotalVisibleDeclContexts = 0;
 
+  /// Number of module local visible decl contexts read/total.
+  unsigned NumModuleLocalVisibleDeclContexts = 0,
+           TotalModuleLocalVisibleDeclContexts = 0;
+
   /// Total size of modules, in bits, currently loaded
   uint64_t TotalModulesSizeInBits = 0;
 
@@ -1444,6 +1460,9 @@ class ASTReader
   const serialization::reader::DeclContextLookupTable *
   getLoadedLookupTables(DeclContext *Primary) const;
 
+  const serialization::reader::ModuleLocalLookupTable *
+  getModuleLocalLookupTables(DeclContext *Primary) const;
+
   /// Get the loaded specializations lookup tables for \p D,
   /// if any.
   serialization::reader::LazySpecializationInfoLookupTable *
@@ -2608,6 +2627,10 @@ inline bool shouldSkipCheckingODR(const Decl *D) {
          (D->isFromGlobalModule() || D->isFromHeaderUnit());
 }
 
+/// Calculate a hash value for the primary module name of the given module.
+/// \returns std::nullopt if M is not a C++ standard module.
+std::optional<unsigned> getPrimaryModuleHash(const Module *M);
+
 } // namespace clang
 
 #endif // LLVM_CLANG_SERIALIZATION_ASTREADER_H

clang/include/clang/Serialization/ASTWriter.h

@@ -492,6 +492,10 @@ class ASTWriter : public ASTDeserializationListener,
   /// file.
   unsigned NumVisibleDeclContexts = 0;
 
+  /// The number of module local visible declcontexts written to the AST
+  /// file.
+  unsigned NumModuleLocalDeclContexts = 0;
+
   /// A mapping from each known submodule to its ID number, which will
   /// be a positive integer.
   llvm::DenseMap<const Module *, unsigned> SubmoduleIDs;
@@ -587,11 +591,15 @@ class ASTWriter : public ASTDeserializationListener,
   uint64_t WriteSpecializationInfoLookupTable(
       const NamedDecl *D, llvm::SmallVectorImpl<const Decl *> &Specializations,
       bool IsPartial);
-  void GenerateNameLookupTable(ASTContext &Context, const DeclContext *DC,
-                               llvm::SmallVectorImpl<char> &LookupTable);
+  void
+  GenerateNameLookupTable(ASTContext &Context, const DeclContext *DC,
+                          llvm::SmallVectorImpl<char> &LookupTable,
+                          llvm::SmallVectorImpl<char> &ModuleLocalLookupTable);
   uint64_t WriteDeclContextLexicalBlock(ASTContext &Context,
                                         const DeclContext *DC);
-  uint64_t WriteDeclContextVisibleBlock(ASTContext &Context, DeclContext *DC);
+  void WriteDeclContextVisibleBlock(ASTContext &Context, DeclContext *DC,
+                                    uint64_t &VisibleBlockOffset,
+                                    uint64_t &ModuleLocalBlockOffset);
   void WriteTypeDeclOffsets();
   void WriteFileDeclIDsMap();
   void WriteComments(ASTContext &Context);
@@ -624,7 +632,9 @@ class ASTWriter : public ASTDeserializationListener,
   unsigned DeclParmVarAbbrev = 0;
   unsigned DeclContextLexicalAbbrev = 0;
   unsigned DeclContextVisibleLookupAbbrev = 0;
+  unsigned DeclModuleLocalVisibleLookupAbbrev = 0;
   unsigned UpdateVisibleAbbrev = 0;
+  unsigned ModuleLocalUpdateVisibleAbbrev = 0;
   unsigned DeclRecordAbbrev = 0;
   unsigned DeclTypedefAbbrev = 0;
   unsigned DeclVarAbbrev = 0;

clang/lib/AST/DeclBase.cpp

@@ -130,6 +130,14 @@ void Decl::setOwningModuleID(unsigned ID) {
   *IDAddress |= (uint64_t)ID << 48;
 }
 
+Module *Decl::getTopLevelOwningNamedModule() const {
+  if (getOwningModule() &&
+      getOwningModule()->getTopLevelModule()->isNamedModule())
+    return getOwningModule()->getTopLevelModule();
+
+  return nullptr;
+}
+
 Module *Decl::getOwningModuleSlow() const {
   assert(isFromASTFile() && "Not from AST file?");
   return getASTContext().getExternalSource()->getModule(getOwningModuleID());