When wheels are deployed, pybind11 is unnecessarily installed

[paulmelnikow]

Hi! Thanks so much for this very useful example. I've been studying it as I've been trying to understand the best way to set up wheels for a project.

I see that pybind11 is in both install_requires and setup_requires. When installing a source distribution through pip, pip will first install pybind11, so the headers are available and the build works.

If it's removed from install_requires, I need to install pybind11 and my project together. Otherwise I get errors like this:

    bindings.cc:1:10: fatal error: 'pybind11/pybind11.h' file not found
    #include <pybind11/pybind11.h>
             ^~~~~~~~~~~~~~~~~~~~~
    1 error generated.
    error: command 'clang' failed with exit status 1

However there's a downside. If I build a wheel, when it's later installed the install_requires take effect, and pybind11 is installed on the deployment machine. As far as I can tell this is completely unnecessary as the built extensions do not have a runtime dependency on pybind11.

What I would like is these two behaviors:

• When a downstream user is installing the source distribution, pybind11 is installed.
• When a wheel is built, the wheel does not declare an install dependency on pybind11.

This probably is not a pybind issue; it's more of a setuptools issue as it relates to pybind11, though this seems like a good place to develop the recipe.

One idea is to remove pybind11 from install_requires and add some code that runs on build_ext to programmatically install pybind11 (when a valid version isn't already available).

[paulmelnikow]

I found pybind/pybind11#1067 which is related to this issue.

pep517 and pyproject.toml seems like it's perfect for this: https://martin-thoma.com/pyproject-toml/

There's one problem, which is that pybind11.get_include(True|False) returns the wrong value for a pep517 temporary install.

[paulmelnikow]

Until pybind/pybind11#1067 is fixed, this is what I'm doing:

# Adapted from https://github.com/pybind/python_example/blob/master/setup.py
class get_pybind_include(object):
    """Helper class to determine the pybind11 include path
    The purpose of this class is to postpone importing pybind11
    until it is actually installed, so that the ``get_include()``
    method can be invoked. """

    def __init__(self, user=False, pep517=False):
        self.user = user
        self.pep517 = pep517

    def __str__(self):
        import os
        import pybind11

        interpreter_include_path = pybind11.get_include(self.user)

        if self.pep517:
            # When pybind11 is installed permanently in site packages, the headers
            # will be in the interpreter include path above. PEP 517 provides an
            # experimental feature for build system dependencies. When installing
            # a package from a source distribvution, first its build dependencies
            # are installed in a temporary location. pybind11 does not return the
            # correct path for this condition, so we glom together a second path,
            # and ultimately specify them _both_ in the include search path.
            # https://github.com/pybind/pybind11/issues/1067
            return os.path.abspath(
                os.path.join(
                    os.path.dirname(pybind11.__file__),
                    "..",
                    "..",
                    "..",
                    "..",
                    "include",
                    os.path.basename(interpreter_include_path),
                )
            )
        else:
            return interpreter_include_path


# `tiny_obj_loader.cc` contains implementation of tiny_obj_loader.
m = setuptools.Extension(
    "tinyobjloader",
    extra_compile_args=["-std=c++11"],
    sources=["bindings.cc", "tiny_obj_loader.cc"],
    include_dirs=[
        # Support `build_ext` finding tinyobjloader (without first running
        # `sdist`).
        "..",
        # Support `build_ext` finding pybind 11 (provided it's permanently
        # installed).
        get_pybind_include(),
        get_pybind_include(user=True),
        # Support building from a source distribution finding pybind11 from
        # a PEP 517 temporary install.
        get_pybind_include(pep517=True),
    ],
    language="c++",
)

It has the characteristics I'm looking for:

• From the project repo, python setup.py bdist_wheel works as long as pybind11 is installed.
• When installing from a source distribution, pybind11 is installed temporarily for building.
• When installing a wheel, pybind11 is not installed.