Cannot install from setup.py [get_include() returns 'include']

[DomHudson]

Issue description

The current python example setup.py layout fails to correctly compile the extension. This feels like a recent change as I have used it's template previously with no trouble.

Any help would be very much appreciated! Thanks.

Reproducible example code

git clone git@github.com:pybind/python_example.git
cd python_example
virtualenv -p python3 env
. env/bin/activate
pip install .

The expected behaviour is that the package compiles the extension and installs normally.

Actual behaviour is that the pybind11/pybind11.h header is not found. If I install and run the get_include function of the pybind11 library manually, it returns the string "include".

python -c "import pybind11; print(pybind11.get_include());"

include

System

Ubuntu 16.04.4 LTS
Python 3.5.2

---

Processing /home/dom/Code/python_example
Collecting pybind11>=2.2 (from python-example==0.0.1)
  Using cached pybind11-2.2.2-py2.py3-none-any.whl
Building wheels for collected packages: python-example
  Running setup.py bdist_wheel for python-example ... error
  Complete output from command /home/dom/Code/python_example/env/bin/python3 -u -c "import setuptools, tokenize;__file__='/tmp/pip-req-build-n_8b1m9l/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" bdist_wheel -d /tmp/pip-wheel-rgj1ns18 --python-tag cp35:
  running bdist_wheel
  running build
  running build_ext
  creating tmp
  x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -I/usr/include/python3.5m -I/home/dom/Code/python_example/env/include/python3.5m -c /tmp/tmpyndbordo.cpp -o tmp/tmpyndbordo.o -std=c++14
  cc1plus: warning: command line option ‘-Wstrict-prototypes’ is valid for C/ObjC but not for C++
  x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -I/usr/include/python3.5m -I/home/dom/Code/python_example/env/include/python3.5m -c /tmp/tmpk7a0py5z.cpp -o tmp/tmpk7a0py5z.o -fvisibility=hidden
  cc1plus: warning: command line option ‘-Wstrict-prototypes’ is valid for C/ObjC but not for C++
  building 'python_example' extension
  Traceback (most recent call last):
    File "<string>", line 1, in <module>
    File "/tmp/pip-req-build-n_8b1m9l/setup.py", line 103, in <module>
      zip_safe=False,
    File "/home/dom/Code/python_example/env/lib/python3.5/site-packages/setuptools/__init__.py", line 129, in setup
      return distutils.core.setup(**attrs)
    File "/usr/lib/python3.5/distutils/core.py", line 148, in setup
      dist.run_commands()
    File "/usr/lib/python3.5/distutils/dist.py", line 955, in run_commands
      self.run_command(cmd)
    File "/usr/lib/python3.5/distutils/dist.py", line 974, in run_command
      cmd_obj.run()
    File "/home/dom/Code/python_example/env/lib/python3.5/site-packages/wheel/bdist_wheel.py", line 204, in run
      self.run_command('build')
    File "/usr/lib/python3.5/distutils/cmd.py", line 313, in run_command
      self.distribution.run_command(command)
    File "/usr/lib/python3.5/distutils/dist.py", line 974, in run_command
      cmd_obj.run()
    File "/usr/lib/python3.5/distutils/command/build.py", line 135, in run
      self.run_command(cmd_name)
    File "/usr/lib/python3.5/distutils/cmd.py", line 313, in run_command
      self.distribution.run_command(command)
    File "/usr/lib/python3.5/distutils/dist.py", line 974, in run_command
      cmd_obj.run()
    File "/home/dom/Code/python_example/env/lib/python3.5/site-packages/setuptools/command/build_ext.py", line 78, in run
      _build_ext.run(self)
    File "/usr/lib/python3.5/distutils/command/build_ext.py", line 338, in run
      self.build_extensions()
    File "/tmp/pip-req-build-n_8b1m9l/setup.py", line 90, in build_extensions
      build_ext.build_extensions(self)
    File "/usr/lib/python3.5/distutils/command/build_ext.py", line 447, in build_extensions
      self._build_extensions_serial()
    File "/usr/lib/python3.5/distutils/command/build_ext.py", line 472, in _build_extensions_serial
      self.build_extension(ext)
    File "/home/dom/Code/python_example/env/lib/python3.5/site-packages/setuptools/command/build_ext.py", line 199, in build_extension
      _build_ext.build_extension(self, ext)
    File "/usr/lib/python3.5/distutils/command/build_ext.py", line 532, in build_extension
      depends=ext.depends)
    File "/usr/lib/python3.5/distutils/ccompiler.py", line 566, in compile
      depends, extra_postargs)
    File "/usr/lib/python3.5/distutils/ccompiler.py", line 341, in _setup_compile
      pp_opts = gen_preprocess_options(macros, incdirs)
    File "/usr/lib/python3.5/distutils/ccompiler.py", line 1075, in gen_preprocess_options
      pp_opts.append("-I%s" % dir)
    File "/tmp/pip-req-build-n_8b1m9l/setup.py", line 20, in __str__
      import pybind11
  ImportError: No module named 'pybind11'
  
  ----------------------------------------
  Failed building wheel for python-example
  Running setup.py clean for python-example
Failed to build python-example
Installing collected packages: pybind11, python-example
  Running setup.py install for python-example ... error
    Complete output from command /home/dom/Code/python_example/env/bin/python3 -u -c "import setuptools, tokenize;__file__='/tmp/pip-req-build-n_8b1m9l/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-record-kn9p2t6m/install-record.txt --single-version-externally-managed --compile --install-headers /home/dom/Code/python_example/env/include/site/python3.5/python-example:
    running install
    running build
    running build_ext
    x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -I/usr/include/python3.5m -I/home/dom/Code/python_example/env/include/python3.5m -c /tmp/tmpi3hv316z.cpp -o tmp/tmpi3hv316z.o -std=c++14
    cc1plus: warning: command line option ‘-Wstrict-prototypes’ is valid for C/ObjC but not for C++
    x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -I/usr/include/python3.5m -I/home/dom/Code/python_example/env/include/python3.5m -c /tmp/tmpe302g89s.cpp -o tmp/tmpe302g89s.o -fvisibility=hidden
    cc1plus: warning: command line option ‘-Wstrict-prototypes’ is valid for C/ObjC but not for C++
    building 'python_example' extension
    creating build
    creating build/temp.linux-x86_64-3.5
    creating build/temp.linux-x86_64-3.5/src
    x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -Iinclude -Iinclude -I/usr/include/python3.5m -I/home/dom/Code/python_example/env/include/python3.5m -c src/main.cpp -o build/temp.linux-x86_64-3.5/src/main.o -DVERSION_INFO="0.0.1" -std=c++14 -fvisibility=hidden
    cc1plus: warning: command line option ‘-Wstrict-prototypes’ is valid for C/ObjC but not for C++
    src/main.cpp:1:31: fatal error: pybind11/pybind11.h: No such file or directory
    compilation terminated.
    error: command 'x86_64-linux-gnu-gcc' failed with exit status 1
    
    ----------------------------------------
Command "/home/dom/Code/python_example/env/bin/python3 -u -c "import setuptools, tokenize;__file__='/tmp/pip-req-build-n_8b1m9l/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-record-kn9p2t6m/install-record.txt --single-version-externally-managed --compile --install-headers /home/dom/Code/python_example/env/include/site/python3.5/python-example" failed with error code 1 in /tmp/pip-req-build-n_8b1m9l/

[DomHudson]

This actually appears to be a pip problem. When creating the virtualenv it was installing pip 10.0.0b1 which doesn't have the necessary locations package that pybind11 looks for. Instead the package is located at pip._internal.locations.

https://github.com/pypa/pip/blob/release/10.0.0/src/pip/_internal/locations.py)`.

Forcing an install of the previous version fixes it.

pip install pip==9.0.3

[DomHudson]

I'm reopening this as I believe it should still be fixed to account for pip>=10.0.

[SylvainCorlay]

I opened an issue upstream in pip: pypa/pip#5191

[eladj]

I'm adding this link:
https://mail.python.org/pipermail/distutils-sig/2017-October/031642.html
According to this, in pip 10 a direct call to the internal API of pip is not supported (and never was recommended).
So we need another method to get the include directory.

[DomHudson]

Thank you @eladj .

Closing this as this now-merged PR fixes the issue as discussed here.