[python3] Hints from python3/vcpkg-cmake-wrapper.cmake are being ignored by FindPython

[RealLitb]

Describe the bug We have the problem that the FindPython module of cmake links against the wrong python library (it links against the non-debug library). This causes a linker error because certain debug symbols are not present in the libpython3.11.so but are present in the correct libpython3.11d.so which it will not link against.

If we execute cmake configure twice, it will use the correct library. If we execute it only once, it won't. We narrowed it down to this code of Modules/FindPackage/Support.cmake:

      if ("LIBRARY" IN_LIST _${_PYTHON_PREFIX}_FIND_DEVELOPMENT_${id}_ARTIFACTS)
        unset (_${_PYTHON_PREFIX}_LIBRARY_RELEASE CACHE)
        unset (_${_PYTHON_PREFIX}_LIBRARY_DEBUG CACHE)
      endif()

These lines are executed, if a cached signature of the value of ${_${_PYTHON_PREFIX}_LIBRARY_RELEASE} differ between configure runs. When we run cmake the first time, there is no signature yet and it appears this causes the file to clear the hints that the VCPKG overriding find_package sets for it!

The Support.cmake file then continues with a built-in search for the library, which only finds the release library.

[RealLitb]

As a workaround we execute the find_package twice, but maybe this workaround could instead be employed in the cmake overrider of find_package?

find_package(Python3 3.11 COMPONENTS Interpreter Development REQUIRED)
find_package(Python3 3.11 COMPONENTS Interpreter Development REQUIRED)

[JonLiu1993]

@RealLitb, thanks for posting this question, could you please provide steps to reproduce?

[RealLitb]

@RealLitb, thanks for posting this question, could you please provide steps to reproduce?

Yes. Clone a fresh vcpkg and run the bootstrap. Then create the following CMake:

cmake_minimum_required(VERSION 3.27)
project(Test)

find_package(Python3 REQUIRED COMPONENTS Interpreter Development)

add_executable(Test)
target_sources(Test PRIVATE main.cc)
target_link_libraries(Test PRIVATE Python3::Python)

With a main.cc that only has int main() { }. Use the following Manifest

{
  "dependencies": [ "python3" ]
}

Then do the following configure step, where ../git/vcpkg is where you cloned VCPKG.

$ cmake -DCMAKE_TOOLCHAIN_FILE=../git/vcpkg/scripts/buildsystems/vcpkg.cmake -DVCPKG_TARGET_TRIPLET=x64-linux-dynamic -DCMAKE_SYSTEM_PROCESSOR=x86_64 -DCMAKE_BUILD_TYPE=Debug .

Take note that you request a Debug build.

Now build with ninja -v and notice the incorrect linked library:

[2/2] : && /usr/bin/c++ -g CMakeFiles/Test.dir/main.cc.o -o Test -Wl,-rpath,/home/ZESLAN/jschaub/vcpkgtest/vcpkg_installed/x64-linux-dynamic/lib vcpkg_installed/x64-linux-dynamic/lib/libpython3.11.so && :

Instead, it should have linked to vcpkg_installed/x64-linux-dynamic/debug/lib/libpython3.11d.so.

Now repeat the exact same configure step and run ninja -v again. Notice that it relinks and outputs

[1/1] : && /usr/bin/c++ -g CMakeFiles/Test.dir/main.cc.o -o Test -Wl,-rpath,/home/ZESLAN/jschaub/vcpkgtest/vcpkg_installed/x64-linux-dynamic/debug/lib vcpkg_installed/x64-linux-dynamic/debug/lib/libpython3.11d.so && :

On the reconfigure, it takes the correct library. The reason for this is, I think, the behavior mentioned in the initial post above.

[RealLitb]

I just found a simple workaround, albeit (like status quo aswell) is very ugly. It is based on the fact that "unset(... CACHE)" does not unset a non-cache variable of the same name (also non-cache variables shadow cache entries of the same name.. they are two separate things). So after the CMake file in "Modules/FindPython/Support.cmake" executes unset (_${_PYTHON_PREFIX}_LIBRARY_RELEASE CACHE), the variable of that name still exists and will be used by the module.

Therefore the following little tweak will make cmake configure the correct library even on the first cmake run: Insert "NO_CACHE" in the find_library calls in vcpkg-cmake-wrapper.cmake (https://github.com/microsoft/vcpkg/blob/master/ports/python3/vcpkg-cmake-wrapper.cmake).

[JonLiu1993]

root@test:~# /root/vcpkg/downloads/tools/cmake-3.27.1-linux/cmake-3.27.1-linux-x86_64/bin/cmake -DCMAKE_TOOLCHAIN_FILE=/root/vcpkg/scripts/buildsystems/vcpkg.cmake -DVCPKG_TARGET_TRIPLET=x64-linux -DCMAKE_BUILD_TYPE=Debug .
-- The C compiler identification is GNU 11.4.0
-- The CXX compiler identification is GNU 11.4.0
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /usr/bin/cc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /usr/bin/c++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Found Python3: /usr/bin/python3.10 (found version "3.10.12") found components: Interpreter Development Development.Module Development.Embed
-- Configuring done (0.8s)
-- Generating done (0.0s)
-- Build files have been written to: /root

[RealLitb]

root@test:~# /root/vcpkg/downloads/tools/cmake-3.27.1-linux/cmake-3.27.1-linux-x86_64/bin/cmake -DCMAKE_TOOLCHAIN_FILE=/root/vcpkg/scripts/buildsystems/vcpkg.cmake -DVCPKG_TARGET_TRIPLET=x64-linux -DCMAKE_BUILD_TYPE=Debug .
-- The C compiler identification is GNU 11.4.0
-- The CXX compiler identification is GNU 11.4.0
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /usr/bin/cc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /usr/bin/c++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Found Python3: /usr/bin/python3.10 (found version "3.10.12") found components: Interpreter Development Development.Module Development.Embed
-- Configuring done (0.8s)
-- Generating done (0.0s)
-- Build files have been written to: /root

Why have you posted the configure output?

[JonLiu1993]

root@test:~# /root/vcpkg/downloads/tools/cmake-3.27.1-linux/cmake-3.27.1-linux-x86_64/bin/cmake -DCMAKE_TOOLCHAIN_FILE=/root/vcpkg/scripts/buildsystems/vcpkg.cmake -DVCPKG_TARGET_TRIPLET=x64-linux -DCMAKE_BUILD_TYPE=Debug .
-- The C compiler identification is GNU 11.4.0
-- The CXX compiler identification is GNU 11.4.0
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /usr/bin/cc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /usr/bin/c++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Found Python3: /usr/bin/python3.10 (found version "3.10.12") found components: Interpreter Development Development.Module Development.Embed
-- Configuring done (0.8s)
-- Generating done (0.0s)
-- Build files have been written to: /root

Why have you posted the configure output?

This is the result I reproduced based on the steps you provided.

[RealLitb]

root@test:~# /root/vcpkg/downloads/tools/cmake-3.27.1-linux/cmake-3.27.1-linux-x86_64/bin/cmake -DCMAKE_TOOLCHAIN_FILE=/root/vcpkg/scripts/buildsystems/vcpkg.cmake -DVCPKG_TARGET_TRIPLET=x64-linux -DCMAKE_BUILD_TYPE=Debug .
-- The C compiler identification is GNU 11.4.0
-- The CXX compiler identification is GNU 11.4.0
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /usr/bin/cc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /usr/bin/c++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Found Python3: /usr/bin/python3.10 (found version "3.10.12") found components: Interpreter Development Development.Module Development.Embed
-- Configuring done (0.8s)
-- Generating done (0.0s)
-- Build files have been written to: /root

Why have you posted the configure output?

This is the result I reproduced based on the steps you provided.

Excellent. If you now run ninja -v (or make) you will find that it does not link the correct python library.

[Osyotr]

-- Found Python3: /usr/bin/python3.10 (found version "3.10.12") found components: Interpreter Development Development.Module Development.Embed

This is not python from vcpkg...

[RealLitb]

I modified my example to show a case that actually fails to link, rather than "merely" showing a case where VCPKG picks the wrong library (non-debug) in Debug configuration (because of the empty main function, it did not yield to any problem).

Use this main from the pybind11 tutorial:

#include <pybind11/embed.h> // everything needed for embedding
namespace py = pybind11;

int main() {
    py::scoped_interpreter guard{}; // start the interpreter and keep it alive
    py::print("Hello, World!"); // use the Python API
}

Modify the CMakeLists.txt this way:

cmake_minimum_required(VERSION 3.27)
project(Test)
find_package(pybind11 REQUIRED CONFIG)
find_package(Python3 REQUIRED COMPONENTS Interpreter Development)

add_executable(Test)
target_sources(Test PRIVATE main.cc)
target_link_libraries(Test PRIVATE pybind11::embed)

And add "pybind11" to the vcpkg.json. Then configure as before (if you don't remove the CMake cache, remember to pass --fresh to cmake!). The output on my system is

-- Running vcpkg install
Detecting compiler hash for triplet x64-linux...
Detecting compiler hash for triplet x64-linux-dynamic...
All requested packages are currently installed.
Total install time: 284 ns
... usage infos stripped ...
-- Running vcpkg install - done
-- The C compiler identification is GNU 12.2.0
-- The CXX compiler identification is GNU 12.2.0
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /usr/bin/cc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /usr/bin/c++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Found Python: /home/ZESLAN/jschaub/vcpkgtest/vcpkg_installed/x64-linux-dynamic/tools/python3/python3.11 (found suitable version "3.11.5", minimum required is "3.6") found components: Interpreter Development Development.Module Development.Embed
-- Performing Test HAS_FLTO
-- Performing Test HAS_FLTO - Failed
-- Found pybind11: /home/ZESLAN/jschaub/vcpkgtest/vcpkg_installed/x64-linux-dynamic/include (found version "2.11.1")
-- Found Python3: /home/ZESLAN/jschaub/vcpkgtest/vcpkg_installed/x64-linux-dynamic/tools/python3/python3.11 (found version "3.11.5") found components: Interpreter Development Development.Module Development.Embed
-- Configuring done (1.8s)
-- Generating done (0.0s)
-- Build files have been written to: /home/ZESLAN/jschaub/vcpkgtest

The result of "ninja -v" is:

[1/2] /usr/bin/c++ -DPy_DEBUG -isystem /home/ZESLAN/jschaub/vcpkgtest/vcpkg_installed/x64-linux-dynamic/include -isystem /home/ZESLAN/jschaub/vcpkgtest/vcpkg_installed/x64-linux-dynamic/include/python3.11 -g -MD -MT CMakeFiles/Test.dir/main.cc.o -MF CMakeFiles/Test.dir/main.cc.o.d -o CMakeFiles/Test.dir/main.cc.o -c /home/ZESLAN/jschaub/vcpkgtest/main.cc
[2/2] : && /usr/bin/c++ -g  CMakeFiles/Test.dir/main.cc.o -o Test  -Wl,-rpath,/home/ZESLAN/jschaub/vcpkgtest/vcpkg_installed/x64-linux-dynamic/lib  vcpkg_installed/x64-linux-dynamic/lib/libpython3.11.so && :
FAILED: Test 
: && /usr/bin/c++ -g  CMakeFiles/Test.dir/main.cc.o -o Test  -Wl,-rpath,/home/ZESLAN/jschaub/vcpkgtest/vcpkg_installed/x64-linux-dynamic/lib  vcpkg_installed/x64-linux-dynamic/lib/libpython3.11.so && :
ld: error: undefined symbol: _Py_RefTotal
>>> referenced by object.h:500 (/home/ZESLAN/jschaub/vcpkgtest/vcpkg_installed/x64-linux-dynamic/include/python3.11/object.h:500)
>>>               CMakeFiles/Test.dir/main.cc.o:(Py_INCREF)
>>> referenced by object.h:500 (/home/ZESLAN/jschaub/vcpkgtest/vcpkg_installed/x64-linux-dynamic/include/python3.11/object.h:500)
>>>               CMakeFiles/Test.dir/main.cc.o:(Py_INCREF)
>>> referenced by object.h:520 (/home/ZESLAN/jschaub/vcpkgtest/vcpkg_installed/x64-linux-dynamic/include/python3.11/object.h:520)
>>>               CMakeFiles/Test.dir/main.cc.o:(Py_DECREF)
>>> referenced 1 more times

ld: error: undefined symbol: _Py_NegativeRefcount
>>> referenced by object.h:523 (/home/ZESLAN/jschaub/vcpkgtest/vcpkg_installed/x64-linux-dynamic/include/python3.11/object.h:523)
>>>               CMakeFiles/Test.dir/main.cc.o:(Py_DECREF)
collect2: error: ld returned 1 exit status
ninja: build stopped: subcommand failed.
~/vcpkgtest$ 

Take note that it passes -DPy_DEBUG but does not link against the debug, but against the release library. This casues a linker error because the needed symbols are not defined.

[RealLitb]

Neumann-A on Discord recommended we do a "--trace-expand" to find out why the toolchain file is not used by CMake on your system.

[DimRochette]

We have similar error python3 boost-python 1.83 using python 3.11 here is our ugly workaround for debug link

if(VCPKG_TOOLCHAIN AND ${CMAKE_BUILD_TYPE} STREQUAL "Debug")
	if (UNIX)
		target_link_libraries(${PROJECT_NAME} PRIVATE python${Python3_VERSION_MAJOR}.${Python3_VERSION_MINOR}d.a)
	endif()
    target_link_directories(${PROJECT_NAME} PRIVATE ${_VCPKG_INSTALLED_DIR}/${VCPKG_TARGET_TRIPLET}/debug/lib)
endif()

[github-actions[bot]]

This is an automated message. Per our repo policy, stale issues get closed if there has been no activity in the past 180 days. The issue will be automatically closed in 14 days. If you wish to keep this issue open, please add a new comment.