Recent CMake changes prefer pybind11 installed in /usr/local/include to development files [macOS]

[jbarlow83]

Issue description

Checkout HEAD - commit 1732046

mkdir build
cd build
cmake ..
make [check]

This is a regression compared to v2.5.0, where commands above work without issue.

cmake: 3.18.1
clang: Apple clang version 11.0.0 (clang-1100.0.33.17)
make: GNU Make 3.81

Result:

[  2%] Building CXX object tests/CMakeFiles/pybind11_cross_module_tests.dir/pybind11_cross_module_tests.cpp.o
~/src/pybind11/tests/pybind11_cross_module_tests.cpp:10:
~/src/pybind11/tests/pybind11_tests.h:56:1: error: unknown type name 'PYBIND11_NAMESPACE_BEGIN'
PYBIND11_NAMESPACE_BEGIN(pybind11)
^
~/src/pybind11/tests/pybind11_tests.h:56:35: error: expected ';' after top level declarator
PYBIND11_NAMESPACE_BEGIN(pybind11)
                                  ^
                                  ;
~/src/pybind11/tests/pybind11_tests.h:64:1: error: unknown type name 'PYBIND11_NAMESPACE_END'
PYBIND11_NAMESPACE_END(detail)
^
~/src/pybind11/tests/pybind11_tests.h:64:31: error: expected ';' after top level declarator
PYBIND11_NAMESPACE_END(detail)
                              ^
                              ;
~/src/pybind11/tests/pybind11_cross_module_tests.cpp:11:
~/src/pybind11/tests/local_bindings.h:12:19: error: no template named 'LocalBase'
using LocalType = LocalBase<0>;
                  ^
~/src/pybind11/tests/local_bindings.h:14:22: error: no template named 'LocalBase'
using NonLocalType = LocalBase<1>;
                     ^
~/src/pybind11/tests/local_bindings.h:16:19: error: no template named 'LocalBase'
using NonLocal2 = LocalBase<2>;
                  ^
~/src/pybind11/tests/local_bindings.h:18:23: error: no template named 'LocalBase'
using LocalExternal = LocalBase<3>;
                      ^
~/src/pybind11/tests/local_bindings.h:20:26: error: no template named 'LocalBase'
using MixedLocalGlobal = LocalBase<4>;
                         ^
~/src/pybind11/tests/local_bindings.h:22:26: error: no template named 'LocalBase'
using MixedGlobalLocal = LocalBase<5>;
                         ^
~/src/pybind11/tests/local_bindings.h:25:23: error: no template named 'LocalBase'
using ExternalType1 = LocalBase<6>;
                      ^
~/src/pybind11/tests/local_bindings.h:26:23: error: no template named 'LocalBase'
using ExternalType2 = LocalBase<7>;
                      ^
~/src/pybind11/tests/local_bindings.h:28:30: error: use of undeclared identifier 'LocalType'
using LocalVec = std::vector<LocalType>;
                             ^
~/src/pybind11/tests/local_bindings.h:29:31: error: use of undeclared identifier 'NonLocal2'
using LocalVec2 = std::vector<NonLocal2>;
                              ^
~/src/pybind11/tests/local_bindings.h:30:50: error: use of undeclared identifier 'LocalType'
using LocalMap = std::unordered_map<std::string, LocalType>;
                                                 ^
~/src/pybind11/tests/local_bindings.h:31:33: error: use of undeclared identifier 'NonLocalType'
using NonLocalVec = std::vector<NonLocalType>;
                                ^
~/src/pybind11/tests/local_bindings.h:32:34: error: use of undeclared identifier 'NonLocal2'
using NonLocalVec2 = std::vector<NonLocal2>;
                                 ^
~/src/pybind11/tests/local_bindings.h:33:53: error: use of undeclared identifier 'NonLocalType'
using NonLocalMap = std::unordered_map<std::string, NonLocalType>;
                                                    ^
~/src/pybind11/tests/local_bindings.h:36:22: error: use of undeclared identifier 'LocalVec'
PYBIND11_MAKE_OPAQUE(LocalVec);
                     ^
fatal error: too many errors emitted, stopping now [-ferror-limit=]
20 errors generated.
make[2]: *** [tests/CMakeFiles/pybind11_cross_module_tests.dir/pybind11_cross_module_tests.cpp.o] Error 1
make[1]: *** [tests/CMakeFiles/pybind11_cross_module_tests.dir/all] Error 2
make: *** [all] Error 2

The same errors occurs in each of these cases:

• attempt this with homebrew python (3.8) in the system environment
• attempt the same through a virtual environment
• pip install . is used to install pybind11
• pip install -e . is used to install pybind11 in edit mode

[bstaletic]

The CI is green though. Can you check if including pybind11/common.h in ~/src/pybind11/tests/pybind11_tests.h changes anything?

[YannickJadoul]

I can't really reproduce, it seems. Also on commit 1732046, with Python 3.8 (Python 3.8.2 (v3.8.2:7b3ab5921f, Feb 24 2020, 17:52:18), not sure where it originated), macOS Catilina (10.15.4), clang 11.0.0 (Apple clang version 11.0.0 (clang-1100.0.33.12)).

[jbarlow83]

The issue was my homebrew pybind11 package installed in /usr/local/include took precedence.

Whatever v2.5.0 did, it ensured that the project pybind11 includes took priority over the system install.

[henryiii]

Two possible reasons. If we used to run export(pybind11), then that is now disabled by default in CMake (since 3.15) unless CMAKE_EXPORT_PACKAGE_REGISTRY is set. That was primarily because it can be surprising to have build directories "magically" be picked up by other unrelated build directories; this makes the export explicit on the user's side, too.

But since I don't think I remember seeing a build directory export, I assume you are doing something to pick it up? What are you doing that causes it to be picked up (before)? Do you have a repo with a CMakeLists somewhere I can look at? The old tools were bad at changing the CMAKE_MODULE_PATH, which find_packages and add_subdirectories should not do, but don't see why that would change an initial discovery.

Edit: Wait, is this for a different project, or for building pybind11's own tests? It would be better if I actually read the issue, apparently.

[henryiii]

Nvm, I know what changed. Let me think about it.

In the meantime, the correct thing to do with brew is to brew unlink pybind11 when you don't want it, and brew link pybind11 when you do.

[henryiii]

I was able to reproduce, Boost is the culprit, and it only affects building pybind11's tests. You have to have boost and pybind11 installed in the same location to trigger the incorrect include ordering.

[henryiii]

By the way, use:

mkdir build
cd build
cmake ..
VERBOSE=1 make [check]

Or, better yet:

cmake -S . -B build
cmake --build build -v

To see the incorrect include ordering.