[BUG]: pip installed pybind11 not picked up by CMake

[NAThompson]

Required prerequisites

• [X] Make sure you've read the documentation. Your issue may be addressed there.
• [X] Search the issue tracker and Discussions to verify that this hasn't already been reported. +1 or comment there if it has.
• [x] Consider asking first in the Gitter chat room or in a Discussion.

Problem description

If I install pybind11 via pip, then CMake cannot find it.

Reproducible example code

deleteme$ python3.9 -m venv .
deleteme$ . bin/activate
deleteme$ python3 -m pip install pybind11
Collecting pybind11
  Using cached pybind11-2.8.1-py2.py3-none-any.whl (208 kB)
Installing collected packages: pybind11
Successfully installed pybind11-2.8.1
deleteme$ cat CMakeLists.txt
cmake_minimum_required(VERSION 3.21.1)
project(Reproduce)
find_package(pybind11 REQUIRED)
deleteme$ mkdir build && cd build
build$ cmake --version
cmake version 3.21.1
build$ cmake ../
CMake Error at CMakeLists.txt:5 (find_package):
  By not providing "Findpybind11.cmake" in CMAKE_MODULE_PATH this project has
  asked CMake to find a package configuration file provided by "pybind11",
  but CMake did not find one.

  Could not find a package configuration file provided by "pybind11" with any
  of the following names:

    pybind11Config.cmake
    pybind11-config.cmake

  Add the installation prefix of "pybind11" to CMAKE_PREFIX_PATH or set
  "pybind11_DIR" to a directory containing one of the above files.  If
  "pybind11" provides a separate development package or SDK, be sure it has
  been installed.


-- Configuring incomplete, errors occurred!

[NAThompson]

I have analyzed this is a bit more depth.

If I have a virtualenv deleteme, then I can fix this issue by copying the pybind11/share/cmake into lib:

(deleteme) ➜  deleteme cp -R lib/python3.9/site-packages/pybind11/share/cmake/ lib/
(deleteme) ➜  deleteme ls lib/cmake
pybind11

Moreover, by trying out a few other places within the ${VIRTUAL_ENV} directory, it appears that this is the only place that CMake will look in a virtualenv.

I also have the following workarounds: Since the ${VIRTUAL_ENV}/bin is on the path, we can use:

$ cmake -Dpybind11_DIR=`pybind11-config --cmakedir` ../

and since VIRTUAL_ENV is in ENV, we have

$ cmake -DCMAKE_PREFIX_PATH=${VIRTUAL_ENV}/lib/python3.9/site-packages/ ../

[henryiii]

Try pip install pybind11[global], or use pybind11-config --cmakedir.

[NAThompson]

@henryiii : Yes, both of those suggestions work, but the first is not available for users on HPC systems (which is why I'm putting pybind11 in a virtualenv in the first place), and the second requires documentation and users to actually read that documentation.

I therefore feel like the correct behavior is to put the pybind11/share/cmake files in a place that CMake looks by default. That means $VIRTUAL_ENV/lib/cmake.

(FWIW I tried creating a PR to make this happen but failed . . .)

[henryiii]

The pybind11[global] option does exactly that. ~~Instead of~~ In addition to going into $VIRTUAL_ENV/lib/python3.9/site-packages/pybind11/share/cmake, it goes into $VIRTUAL_ENV/share/cmake.

[henryiii]

I'm also planning to work on discovery in the context of Scikit-build, see https://iscinumpy.gitlab.io/post/scikit-build-proposal/ and https://github.com/scikit-build/scikit-build/wiki#possible-roadmap

[NAThompson]

The pybind11[global] option does exactly that.

aaand I'm a moron. Sorry for the noise. Validated that the syntax Requires-Dist: pybind11[global] (>= 2.7.1) works in wheels as well.

While we're on the topic, any reason by pybind11[global] isn't the default?

[AdamWRichardson]

Sorry for bringing this issue up again but I've found the exact same problem on Windows (I know). Using pybind11[global] solved the issue for me on linux but doesn't solve it on Windows and cmake still gives the exact error message above. Is there any Windows specific thing that I'm missing?

[craigscott-crascit]

Using pybind11[global] does work on Windows for a project I'm working with. Make sure you specify a version constraint of at least pybind11 2.9.2 (going from memory, so I might have that version number wrong). I think earlier versions have a problem in this area and maybe that's why it isn't working in your case.

[AdamWRichardson]

Thanks @craigscott-crascit, I have the version pinned at >=2.10 so should be using a new enough version.

[borontion]

I have been using the variable Python3_SITELIB from Python3's cmake as hint for pybind:

find_package(Python3 REQUIRED Interpreter Development.Module)
find_package(pybind11 CONFIG REQUIRED HINTS "${Python3_SITELIB}")