[BUG]: CMAKE pybind11 tries to find Python even if it has already been found with find_package() (and finds the wrong version) (Windows 10)

[romainaugier]

Required prerequisites

• [X] Make sure you've read the documentation. Your issue may be addressed there.
• [X] Search the issue tracker and Discussions to verify that this hasn't already been reported. +1 or comment there if it has.
• [X] Consider asking first in the Gitter chat room or in a Discussion.

What version (or hash if on master) of pybind11 are you using?

65370f330ea9349bd9e996028b4cb7adfde37660

Problem description

Hello,

I am using pybind in a project and I am having an error when I try to configure the project using CMake.

I use find_package with the required version of python before adding the pybind directory (as recommanded in the documentation here and I am getting this error when configuring the CMake project :

CMake Error at C:/Program Files/CMake/share/cmake-3.29/Modules/FindPackageHandleStandardArgs.cmake:230 (message):                                                                                             Could NOT find PythonInterp: Found unsuitable version "3.10.5", but                                                                                                                                         required is at least "3.12" (found C:/Python/3105/python.exe)                                                                                                                                             Call Stack (most recent call first):                                                                                                                                                                          C:/Program Files/CMake/share/cmake-3.29/Modules/FindPackageHandleStandardArgs.cmake:598 (_FPHSA_FAILURE_MESSAGE)                                                                                            C:/Program Files/CMake/share/cmake-3.29/Modules/FindPythonInterp.cmake:182 (FIND_PACKAGE_HANDLE_STANDARD_ARGS)                                                                                              pybind11/tools/FindPythonLibsNew.cmake:114 (find_package)                                                                                                                                                   pybind11/tools/pybind11Tools.cmake:50 (find_package)                                                                                                                                                        pybind11/tools/pybind11Common.cmake:192 (include)                                                                                                                                                           pybind11/CMakeLists.txt:237 (include)

And just before the error, Python 3.12 is found :

-- Found Python3: C:/Python/3123/python.exe (found suitable version "3.12.3", minimum required is "3.12") found components: Interpreter Development Development.Module Development.Embed

Reproducible example code

// Cmd
cmake -S . -B build -DPYTHON_VERSION="3.12"

// CMakeLists.txt
...
# Python
find_package(Python3 ${PYTHON_VERSION} COMPONENTS Interpreter Development REQUIRED)

if(NOT ${Python_FOUND})
    message(FATAL_ERROR "Python can't be found, exiting CMake")
endif()

set(PYBIND11_PYTHON_VERSION ${PYTHON_VERSION})
set(PYBIND11_FINDPYTHON OFF)

# Pybind
add_subdirectory(pybind11 CONFIG)
...

Is this a regression? Put the last known working version here if it is.

Not a regression

[romainaugier]

After more research I found that setting

set(PYBIND11_NOPYTHON ON)

get rid of the error I had and compiles fine. However I'd still like to know if there is a way to force Pybind11 to use the required version.

Thanks,

[huweiATgithub]

I also encountered this error.

Steps to Reproduce

• find python using find_package(Python COMPONENTS Interpreter Development)
• don't set option PYBIND11_FINDPYTHON
• use pybind11 through add_subdirectory

Why

In Pybind11's CMakeLists.txt, it defines PYBIND11_FINDPYTHON which defaulted to OFF when using pybind through add_subdirectory(pybind11): https://github.com/pybind/pybind11/blob/750257797ca5eda6f1561f6c3f7147bcdd92cf52/CMakeLists.txt#L135-L136

Then, when searching Python, it will always use the classic mode if PYBIND11_FINDPYTHON is not set to ON: https://github.com/pybind/pybind11/blob/750257797ca5eda6f1561f6c3f7147bcdd92cf52/tools/pybind11Common.cmake#L174-L194

Inconsistent with Documentation

The documentation in: https://github.com/pybind/pybind11/blob/750257797ca5eda6f1561f6c3f7147bcdd92cf52/tools/pybind11Config.cmake.in#L70-L75 and in, https://github.com/pybind/pybind11/blob/750257797ca5eda6f1561f6c3f7147bcdd92cf52/docs/faq.rst?plain=1#L250-L293 does not match the actual behavior.

Workaround

set PYBIND11_FINDPYTHON to on before adding pybind11.

Possible resolution

Would it be good to activate the following branch when Python_FOUND or Python3_FOUND is present? https://github.com/pybind/pybind11/blob/750257797ca5eda6f1561f6c3f7147bcdd92cf52/tools/pybind11Common.cmake#L178-L184

[QuantuMope]

I can confirm that this is an issue in Ubuntu 22.04 and Ubuntu 24.04 as well.

Getting very weird behavior. For example, let's assume I am on Ubuntu 24.04 which uses python 3.12. I've also installed python 3.9. My goal is to create a pybind module for both python versions.

Cmake version is 3.28.3. Pybind11 verison is 2.11.1.

Getting this to work for the system python is as simple as running

find_package(Python3 REQUIRED COMPONENTS Interpreter Development)
find_package(pybind11 REQUIRED)

pybind11_add_module(kincpp
        ...)

Resulting in kincpp.cpython-312-x86_64-linux-gnu.so.

Now, if I want to create a module for python 3.9, I change my cmake to be as so

find_package(Python3 3.9 EXACT REQUIRED COMPONENTS Interpreter Development)
find_package(pybind11 REQUIRED)

pybind11_add_module(kincpp
        ...)

When building this the terminal outputs

-- Found Python3: /usr/bin/python3.9 (found suitable exact version "3.9.21") found components: Interpreter Development Development.Module Development.Embed 

and the file is properly named as kincpp.cpython-39-x86_64-linux-gnu.so.

But when I import this using python3.9 I get the following ImportError:

(venv) asjchoi@QuantuMope:~/Desktop/Code/kincpp/kincpp (main)$ python3.9
Python 3.9.21 (main, Dec  4 2024, 08:53:34) 
[GCC 13.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import kincpp
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: Python version mismatch: module was compiled for Python 3.12, but the interpreter version is incompatible: 3.9.21 (main, Dec  4 2024, 08:53:34) 
[GCC 13.2.0].

Here's the kicker. If I rename kincpp.cpython-39-x86_64-linux-gnu.so --> kincpp.cpython-312-x86_64-linux-gnu.so, it works for python3.12.

A workaround for this is to do the following hack:

find_package(pybind11 REQUIRED)  # find pybind11 first and then the desired python version
find_package(Python3 3.9 EXACT REQUIRED COMPONENTS Interpreter Development)

pybind11_add_module(kincpp
        ...)

This results in kincpp.cpython-312-x86_64-linux-gnu.so that when renamed to 39, works for python3.9.

In other words, the pybind11 find command is overwriting the previously found python3 package. If I do the hack, I can correctly set the python version to 3.9 but it seems like internally, pybind11 still incorrectly sets the file name to 312.

No combination of usage of the PYBIND11_FINDPYTHON and PYBIND11_PYTHON_VERSION commands help. In fact, to me these seem to have absolutely no affect.