pybind11

pybind11 copied to clipboard

Required prerequisites

• [X] Make sure you've read the documentation. Your issue may be addressed there.
• [X] Search the issue tracker and Discussions to verify that this hasn't already been reported. +1 or comment there if it has.
• [ ] Consider asking first in the Gitter chat room or in a Discussion.

What version (or hash if on master) of pybind11 are you using?

ed466da571fbc1d711351eb818e4bf82adb99eca

Problem description

When compiling a project which uses Pybind11 (oxDNA, update_pybind branch), compilation fails when using Python 3.11 on Mac OS in a base Conda environment because it finds the incorrect include directory. Compilation works with with 3.11 on Linux, or on Mac if I create a fresh environment with 3.11 (or any other version) and compile there.

If set(PYBIND11_FINDPYTTHON True) is not included in our CMakeLists.txt, Cmake generates the following error:

CMake Error in oxpy/CMakeLists.txt:
  Imported target "pybind11::module" includes non-existent path

    "~/miniconda3/python.app/Contents/include/python3.11"

  in its INTERFACE_INCLUDE_DIRECTORIES.  Possible reasons include:

  * The path was deleted, renamed, or moved to another location.

  * An install or uninstall procedure did not complete successfully.

  * The installation package was faulty and references files it does not
  provide.

The actual include directory is ~/miniconda3/include/python3.11/

If, however, set(PYBIND11_FINDPYTTHON True) is included in our CMakeLists.txt as suggested in the documentation, no errors are emitted by Cmake, but the PYTHON_INCLUDE_DIRS variable is empty and the compilation fails with

In file included from ~/software/oxDNA/oxpy/pybind11/include/pybind11/detail/../attr.h:13:
~/software/oxDNA/oxpy/pybind11/include/pybind11/detail/common.h:266:10: fatal error: 'Python.h' file not found

So while this could also be a problem with Cmake or Conda, I suspect something is misconfigured in the Pybind11 CmakeList.txt for this specific case.

Reproducible example code

No response

Is this a regression? Put the last known working version here if it is.

This worked in Pybind11 2.2.4, but we have not tested intervening versions.

Same issue.

Here's a workaround method: https://bitbucket.org/mituq/muq2/issues/23/imported-target-pybind11-module-includes

This can sometimes happen when CMake doesn't find the right version of python. You can get past this issue by manually setting the path to the Python executable you want to use with pybind11 with the PYTHON_EXECUTABLE environment variable. For example, here is a CMake command setting the installation directory and specifying the python executable on a Mac: cmake -DCMAKE_INSTALL_PREFIX=~/Installations -DMUQ_USE_PYTHON=ON -DPYTHON_EXECUTABLE=~/opt/anaconda3/bin/python ../

This lets it find the interpreter, but it still fails to find the include directories

-- Found PythonInterp: /Users/erikpoppleton/miniconda3/bin/python (Required is at least version "3.8") 
CMake Error at oxpy/pybind11/tools/FindPythonLibsNew.cmake:147 (message):
  Python config failure:

Call Stack (most recent call first):
  oxpy/pybind11/tools/pybind11Tools.cmake:50 (find_package)
  oxpy/pybind11/tools/pybind11Common.cmake:180 (include)
  oxpy/pybind11/CMakeLists.txt:208 (include)

-- Configuring incomplete, errors occurred!

It does, however work if I specify the include directory as well:

cmake -DPython=1 -DPYTHON_EXECUTABLE=$HOME/miniconda3/bin/python -DPYTHON_INCLUDE_DIRS=$HOME/miniconda3/include/python3.11 ..

I am still wondering if this is something which can be fixed, or if it is a fundamental limitation of the way Cmake finds Python.

on ubuntu and in a virtual environment (not conda), it was solved by sudo apt install python[version]-dev*. I think the equivalent of it for macos is python-devel. *Header files and a static library for Python[version].

@ErikPoppleton @redthing1 @JinlongLi2016

The problem I originally posted about is not that the files are not installed (as appears to be your case), but that Cmake sets the path to them incorrectly.

yes that's right.