Cross-compilation support

[LecrisUT]

trafficstars

I want to work on this feature, but I need more info on how the cross-compilation support for this should look like:

• User input:
  • Specify a cmake.toolchain pointing to the CMake toolchain file created for the specific cross-compilation target
  • Specify the wheel.py-api for the target python environment
  • Do we need also a wheel.arch to overwrite the archs part specifically in WheelTag?
• Passing the appropriate Python hints:
  • Do we rely on some tool like crossenv to provide the location of the target python headers?
  • Do we rely on the CMake toolchain to be built such that it provides the appropriate hints to find the target python?
  • Would build-isolation get in our way and should we gate it by being in a specific setup?
• How/what combo to test?
  • We could use the gcc cross-compilers for the tests, unless there are other recommended ways to do it?
  • Typical use-case would be android and embedded afaik, should we add something testing those specifically?

[henryiii]

There's work on cross-compiling PEP ("current state" is in https://peps.python.org/pep-0720/, I think there's work on providing a standard file with CPython for a future PEP), we should be compatible with that. PyCon 2025 is in mid May, likely to come up or be something we could discuss there in the packaging summit.

I think android support is planned for cibuildwheel (iOS just landed).

Monthly dev meeting is in 30 mins, we could discuss there.

[LecrisUT]

Monthly dev meeting is in 30 mins, we could discuss there.

I don't think I have access to those. Is it the same room as the community meeting?

[henryiii]

Yes, it's identical.

[LecrisUT]

Ok, I have finally managed to get myself a python cross compilation environment (learned the hard way that gcc --sysroot is broken on Fedora 🙁)

Anyway I've tried doing a build with crossenv with the current main and for the most part it seems to work fine. Some things that I've noticed:

• the wheel filename does not seem to resolve correctly to be installable, i.e. aarch64 instead of linux_aarch64
• ~~SKBUILD_SOABI still expanded to the system's tag cpython-313-x86_64-linux-gnu~~ (scratch this, I was not cross-compiling it correctly). Python_SOABI is empty, probably needs cmake 4.1 with this MR
• Python_EXECUTABLE is expanded to the system's python executable (probably correct?) and Python_INCLUDE_DIR expands to the cross-compiled environment path. But Python_ROOT_DIR is expanding to the host's environment as well, which doesn't seem right
• Development.Embed and making python_add_library(SHARED) fails because Python_LIBRARIES is pointing to the system's file. With some coaxing of Python_LIBRARY it does build correctly though

Otherwise the artifacts created work well afaict. At least trying a simple C binding and checking it with readelf. So what do you think we should approach this?

[LecrisUT]

Oh, it seems that we only need to correct Python_ROOT_DIR and make sure Interpreter is not used in the FindPython, otherwise the CMAKE_CROSSCOMPILING check gets in the way.

My current thinking is that if a PEP517 builder is used then we are in a python environment and we need to detect the target's variables to override. Iiuc PEP720 things like sysconfig.get_config_var should point to the target's environment similar to crossenv environment faking (probably conda-forge as well). Let's see if there is a fix that we can do for this that will work for PEP720. But there seems to be missing a variable to define that we are in a cross-compiling environment. Can we assume sys.base_prefix would point to the target's environment?

[minrk]

There may need an update here related to the cmake 4.1 changes, as cross-compiling on conda-forge using crossenv has started to fail with cmake 4.1: https://github.com/conda-forge/pyzmq-feedstock/pull/120

(maybe the fix belongs in crossenv and/or cmake, I'm not sure).

With cmake 4.1, the failure in cross-compile envs [is]:

Processing $SRC_DIR
  Added file://$SRC_DIR to build tracker '/tmp/pip-build-tracker-8f06ga00'
  Created temporary directory: /tmp/pip-modern-metadata-fwjw8fdp
  Preparing metadata (pyproject.toml): started
  Running command Preparing metadata (pyproject.toml)
  WARNING: Use build.verbose instead of cmake.verbose for scikit-build-core >= 0.10
  *** scikit-build-core 0.11.5 using CMake 4.1.0 (metadata_wheel)
  Preparing metadata (pyproject.toml): finished with status 'done'
  Source in $SRC_DIR has version 27.0.2, which satisfies requirement pyzmq==27.0.2 from file://$SRC_DIR
  Removed pyzmq==27.0.2 from file://$SRC_DIR from build tracker '/tmp/pip-build-tracker-8f06ga00'
Created temporary directory: /tmp/pip-unpack-w9zuqedb
Building wheels for collected packages: pyzmq
  Created temporary directory: /tmp/pip-wheel-cd9z8cvg
  Destination directory: /tmp/pip-wheel-cd9z8cvg
  Building wheel for pyzmq (pyproject.toml): started
  Running command Building wheel for pyzmq (pyproject.toml)
  WARNING: Use build.verbose instead of cmake.verbose for scikit-build-core >= 0.10
  *** scikit-build-core 0.11.5 using CMake 4.1.0 (wheel)
  *** Configuring CMake...
  2025-08-21 07:57:22,984 - scikit_build_core - WARNING - Unsupported CMAKE_ARGS ignored: -DCMAKE_BUILD_TYPE=Release
  2025-08-21 07:57:22,984 - scikit_build_core - WARNING - Unsupported CMAKE_ARGS ignored: -DCMAKE_INSTALL_PREFIX=/home/conda/feedstock_root/build_artifacts/pyzmq_1755762847554/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_place
  2025-08-21 07:57:22,986 - scikit_build_core - WARNING - Unsupported CMAKE_ARGS ignored: -DCMAKE_BUILD_TYPE=Release
  2025-08-21 07:57:22,987 - scikit_build_core - WARNING - Unsupported CMAKE_ARGS ignored: -DCMAKE_INSTALL_PREFIX=/home/conda/feedstock_root/build_artifacts/pyzmq_1755762847554/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_place
  loading initial cache file /tmp/tmpedeq1g8b/build/CMakeInit.txt
  -- The C compiler identification is GNU 14.3.0
  -- The CXX compiler identification is GNU 14.3.0
  -- Detecting C compiler ABI info
  -- Detecting C compiler ABI info - done
  -- Check for working C compiler: /home/conda/feedstock_root/build_artifacts/pyzmq_1755762847554/_build_env/bin/powerpc64le-conda-linux-gnu-cc - skipped
  -- Detecting C compile features
  -- Detecting C compile features - done
  -- Detecting CXX compiler ABI info
  -- Detecting CXX compiler ABI info - done
  -- Check for working CXX compiler: /home/conda/feedstock_root/build_artifacts/pyzmq_1755762847554/_build_env/bin/powerpc64le-conda-linux-gnu-c++ - skipped
  -- Detecting CXX compile features
  -- Detecting CXX compile features - done
  -- Building pyzmq for Linux
  CMake Error at /home/conda/feedstock_root/build_artifacts/pyzmq_1755762847554/_build_env/share/cmake-4.1/Modules/FindPackageHandleStandardArgs.cmake:227 (message):
    Could NOT find Python (missing: Interpreter Development.Module) (found
    version "3.13.5")
  Call Stack (most recent call first):
    /home/conda/feedstock_root/build_artifacts/pyzmq_1755762847554/_build_env/share/cmake-4.1/Modules/FindPackageHandleStandardArgs.cmake:591 (_FPHSA_FAILURE_MESSAGE)
    /home/conda/feedstock_root/build_artifacts/pyzmq_1755762847554/_build_env/share/cmake-4.1/Modules/FindPython.cmake:727 (find_package_handle_standard_args)
    CMakeLists.txt:14 (find_package)


  -- Configuring incomplete, errors occurred!

with cmake 4.0.3 and all other things the same, build succeeds in all cross compile envs. If there's more info I can share or a known workaround, that would probably be useful to conda-forge.