[BUG] cmake detects wrong PYTHON_LIBRARIES even if PYTHON_EXECUTABLE is set

[mbasaglia]

Issue description

When building on a system with multiple python installations, pybind11's cmake script set PYTHON_LIBRARIES to the wrong values.

I was suggested in the gitter chat to set PYTHON_EXECUTABLE. With that it shows the right interpreter, but the libraries are still pulled from the wrong path.

Reproducible example code

It happened to me when trying to link to the MSYS2 python in an Appveyor job, but I've head people had the same issue in other systems as well.

[YannickJadoul]

@mbasaglia Can you give an actual reproduction? Something we can run/look at? Do you have the logs of this AppVeyor job, at least?

[henryiii]

I'm not sure how well MSYS2 is supported by CMake. Can you try the new FindPython (will probably need to update CMake on AppVeyor, say with chocolaty)? This is mostly a function of the old FindPythonInterp/FindPythonLibs (or the new FindPython if you use it).

[mbasaglia]

@mbasaglia Can you give an actual reproduction? Something we can run/look at? Do you have the logs of this AppVeyor job, at least?

this build: https://ci.appveyor.com/project/mattia.basaglia/glaxnimate/builds/36210916?fullLog=true

++ cmake.exe .. -DQt5_DIR=/mingw64/lib/cmake/Qt5 -DZLIB_LIBRARY=/mingw64/lib/libz.a -DCMAKE_PREFIX_PATH=/mingw64/lib/ -DZLIB_INCLUDE_DIR=/mingw64/include -DPYTHON_EXECUTABLE=/mingw64/bin/python3 -G 'MSYS Makefiles' -DCMAKE_INSTALL_PREFIX=
...
-- Found PythonInterp: C:/msys64/mingw64/bin/python3 (found version "3.8.6") 
-- Found PythonLibs: C:/Python39-x64/libs/python39.lib
...
(this is a message with the python include dir)
-- C:/Python39-x64/include

I'm not sure how well MSYS2 is supported by CMake. Can you try the new FindPython (will probably need to update CMake on AppVeyor, say with chocolaty)? This is mostly a function of the old FindPythonInterp/FindPythonLibs (or the new FindPython if you use it).

I'm using the Cmake that comes with MSYS2, looks like it's cmake 3.18.4

[henryiii]

Can you try -DPYBIND11_FINDPYTHON=ON? That will use CMake's new FindPython module instead of the long-deprecated PythonInterp/PythonLibs. Are you sure you have installed the dev package for Python? (at least, in some Unix package managers they are separate)

[mbasaglia]

Can you try -DPYBIND11_FINDPYTHON=ON? That will use CMake's new FindPython module instead of the long-deprecated PythonInterp/PythonLibs. Are you sure you have installed the dev package for Python? (at least, in some Unix package managers they are separate)

with that it seems to completely ignore PYTHON_EXECUTABLE

++ cmake.exe .. -DQt5_DIR=/mingw64/lib/cmake/Qt5 -DZLIB_LIBRARY=/mingw64/lib/libz.a -DCMAKE_PREFIX_PATH=/mingw64/lib/ -DZLIB_INCLUDE_DIR=/mingw64/include -DPYTHON_PREFIX=/mingw64/ -DPYTHON_LIBRARIES=/mingw64/bin/libpython3.8.dll -DPYTHON_EXECUTABLE=/mingw64/bin/python3 -DPYBIND11_FINDPYTHON=ON -G 'MSYS Makefiles' -DCMAKE_INSTALL_PREFIX=
...
-- Python3 enabled
-- pybind11 v2.6.1 dev1
-- Found Python: C:/Python39-x64/python.exe (found version "3.9.0") found components: Interpreter Development Development.Module Development.Embed 
...
-- C:/Python39-x64/include

Also note that if I move that C:/Python39-x64 directory to something else the build process works and it links to the right libraries so it isn't an issue of not having the libraries in the MSYS2 environment. (While this work-around is fine for CI, it would not when the issue happens on someone's physical computer)

[henryiii]

Sorry, with the new FindPython you use Python_EXECUTABLE instead (PYTHON -> Python).

[YannickJadoul]

Sorry, with the new FindPython you use PythonEXECUTABLE instead (PYTHON -> Python).

Python_EXECUTABLE, then, right? (just to be clear to @mbasaglia)

[henryiii]

Yes, just corrected. 😳

[mbasaglia]

Same result, the interpreter is set correctly but libraries aren't

++ cmake.exe .. -DQt5_DIR=/mingw64/lib/cmake/Qt5 -DZLIB_LIBRARY=/mingw64/lib/libz.a -DCMAKE_PREFIX_PATH=/mingw64/lib/ -DZLIB_INCLUDE_DIR=/mingw64/include -DPYTHON_PREFIX=/mingw64/ -DPYTHON_LIBRARIES=/mingw64/bin/libpython3.8.dll -DPython_EXECUTABLE=/mingw64/bin/python3 -DPYBIND11_FINDPYTHON=ON -G 'MSYS Makefiles' -DCMAKE_INSTALL_PREFIX=
...
-- Python3 enabled
-- pybind11 v2.6.1 dev1
-- Found Python: C:/msys64/mingw64/bin/python3 (found version "3.8.6") found components: Interpreter Development Development.Module Development.Embed 
-- Performing Test HAS_FLTO
-- Performing Test HAS_FLTO - Success
-- PYTHON_INCLUDE_DIRS C:/Python39-x64/include
-- PYTHON_LIBRARIES C:/Python39-x64/libs/python39.lib

[mbasaglia]

I tried even setting PYTHON_LIBRARIES myself but it gets overwritten

[henryiii]

You have to check Python_LIBRARIES and Python_INCLUDE_DIRS, are those correct? Are you still running find_package(PythonLibs somewhere? PYTHON_INCLUDE_DIRS and PYTHON_LIBRARIES should not even be set when using the new search. The new search found the right one, so the question is did it find the right libraries too?

[mbasaglia]

I am calling find_package(PythonLibs) to conditionally calling add_subdirectory for pybind.

Python_INCLUDE_DIRS and Python_LIBRARIES were both empty when I tried

-DPython_EXECUTABLE=/mingw64/bin/python3 -DPYBIND11_FINDPYTHON=ON

[henryiii]

You should not use both FindPython and FindPythonInterp/FindPythonLibs. They unified it into a single find module partially due to the issue you are seeing - the two modules are not always in sync on what they find (and other reasons, but this is one). I don't know how those could be empty, though, if you find the developer component... Docs are here: https://cmake.org/cmake/help/v3.18/module/FindPython.html

There are lots of caveats to using FindPythonInterp/FindPythonLibs; I think one is that FindPythonInterp must be called before FindPythonLibs.

[fabien-ors]

I experience the same issue using CMake 3.21.4, python 3.9.2, pybind11 2.9.2 on Ubuntu 18.04. The simple following CMakeLists.txt

cmake_minimum_required(VERSION 3.20)
project(test_find_pybind11)
find_package(Python3)
find_package(pybind11)

produces the following :

-- ...
-- Found Python3: /home/fors/Programmes/Python3.9.2/bin/python3.9 (found version "3.9.2") found components: Interpreter
-- Found PythonInterp: /usr/bin/python3.6 (found version "3.6.9") 
-- Found PythonLibs: /usr/lib/x86_64-linux-gnu/libpython3.6m.so
-- ...

Note: CMake tells that the "Manually-specified" PYBIND11_FINDPYTHON variable is not used by the project.

If I fetch pybind11 from github:

cmake_minimum_required(VERSION 3.20)
project(test_find_pybind11)
find_package(Python3)
#find_package(pybind11)
include(FetchContent)
FetchContent_Declare(
  pybind11
  GIT_REPOSITORY https://github.com/pybind/pybind11.git
  GIT_TAG v2.9.2
)
FetchContent_MakeAvailable(pybind11)

There is no more issue (PythonInterp/PythonLibs are undefined)

[Michaelzhouisnotwhite]

Same result, the interpreter is set correctly but libraries aren't

++ cmake.exe .. -DQt5_DIR=/mingw64/lib/cmake/Qt5 -DZLIB_LIBRARY=/mingw64/lib/libz.a -DCMAKE_PREFIX_PATH=/mingw64/lib/ -DZLIB_INCLUDE_DIR=/mingw64/include -DPYTHON_PREFIX=/mingw64/ -DPYTHON_LIBRARIES=/mingw64/bin/libpython3.8.dll -DPython_EXECUTABLE=/mingw64/bin/python3 -DPYBIND11_FINDPYTHON=ON -G 'MSYS Makefiles' -DCMAKE_INSTALL_PREFIX=
...
-- Python3 enabled
-- pybind11 v2.6.1 dev1
-- Found Python: C:/msys64/mingw64/bin/python3 (found version "3.8.6") found components: Interpreter Development Development.Module Development.Embed 
-- Performing Test HAS_FLTO
-- Performing Test HAS_FLTO - Success
-- PYTHON_INCLUDE_DIRS C:/Python39-x64/include
-- PYTHON_LIBRARIES C:/Python39-x64/libs/python39.lib

maybe the library ends with python3.10.dll.a. You miss ".a". I use -DPython_LIBRARY:STRING=C:\tools\msys64\mingw64/lib/libpython3.10.dll.a which the library becomes correct.