[Windows ARM] `sys_tags` produces `win_amd64`

[JP-Ellis]

Description

Summary

I have recently wanted to enable support for Windows ARM wheels being generated, but for some reason, AMD64 (i.e. x86_64) is consistently being detected.

Background

I am trying to build wheels which include a binary extension. The binary extension effectively wraps around a pre-compiled binary, and the build script uses packaging's sys_tags function in order to determine the target platform and therefore the appropriate library to download.

The high level logs are:

Building cp39-win_arm64 wheel
CPython 3.9 Windows on ARM 64bit
Installing Python cp39...
                                                             ✓ 16.75s
Installing native Python AMD64 for cross-compilation...
                                                             ✓ 13.20s
Setting up build environment...
Notice: Setting DIST_EXTRA_CONFIG=C:\Users\runneradmin\AppData\Local\Temp\cibw-run-i_504v45\cp39-win_arm64\build\extra-setup.cfg for cross-compilation
                                                             ✓ 12.75s
Installing build tools...
                                                              ✓ 0.36s
Running before_build...
                                                              ✓ 0.01s
Building wheel...
                                                             ✕ 41.87s

The build script is clearly detecting the AMD64 platform. To help debug, I have added:

logger.debug("Sys tags: %s", list(sys_tags()))
logger.debug("Sysconfig platform: %s", sysconfig.get_platform())
logger.debug("Sys version: %s", sys.version.lower())

which produces the following:

    01:38:27.339 [DEBUG   ] hatch_build.py: Sys tags: [<cp39-cp39-win_amd64 @ 2222568532224>, <cp39-abi3-win_amd64 @ 2222568533056>, <cp39-none-win_amd64 @ 2222568531904>, <cp38-abi3-win_amd64 @ 2222568532096>, <cp37-abi3-win_amd64 @ 2222568533248>, <cp36-abi3-win_amd64 @ 2222568533696>, <cp35-abi3-win_amd64 @ 2222568533952>, <cp34-abi3-win_amd64 @ 2222568558848>, <cp33-abi3-win_amd64 @ 2222568559104>, <cp32-abi3-win_amd64 @ 2222568559360>, <py39-none-win_amd64 @ 2222568560000>, <py3-none-win_amd64 @ 2222568532480>, <py38-none-win_amd64 @ 2222568560192>, <py37-none-win_amd64 @ 2222568560384>, <py36-none-win_amd64 @ 2222568560640>, <py35-none-win_amd64 @ 2222568560896>, <py34-none-win_amd64 @ 2222568561152>, <py33-none-win_amd64 @ 2222568561408>, <py32-none-win_amd64 @ 2222568561664>, <py31-none-win_amd64 @ 2222568561920>, <py30-none-win_amd64 @ 2222568562176>, <cp39-none-any @ 2222568566848>, <py39-none-any @ 2222568562432>, <py3-none-any @ 2222568567104>, <py38-none-any @ 2222568567424>, <py37-none-any @ 22225
    01:38:27.339 [DEBUG   ] hatch_build.py: Sysconfig platform: win-amd64
    01:38:27.339 [DEBUG   ] hatch_build.py: Sys version: 3.9.10 (tags/v3.9.10:f2f3f53, jan 17 2022, 14:17:25) [msc v.1929 64 bit (amd64)]

Note that I am using hatch to build the wheel, though I don't understand why/how this might be the issue.

I am asking first here in case there's something obviously wrong with my use of CIBuildWheel, and if not, I will pose this question over with Hatch too.

Build log

https://github.com/pact-foundation/pact-python/actions/runs/10033168088/job/27725905437?pr=737

CI config

https://github.com/pact-foundation/pact-python/blob/debug/win-arm-build2/.github/workflows/build.yml

[Czaki]

It looks like you are using custom code for building binary PactBuildHook It may not pass cross compilation setup to the compiler. (it was a common problem when people are using own cmake wrapping code in setup.py). But I do not have experience with hatchling and compiled code to suggest a proper solution.

[JP-Ellis]

The compilation itself is done using cffi, and I actually believe that is working fine. At a high level, the steps are:

1. Detect the platform,
2. Download the pre-compiled library from upstream, along with headers,
3. Build the Python extension with cffi.

The last step fails because of linking errors, and which I believe is because cffi is trying to link an ARM Python extension and an x86_64 library.

Are you able to elaborate a little on what this 'cross compilation setup' is? How is the compiler meant to know to cross-compile? Is it done purely through environment variables being set? Or are there any flags I should look for?

[Czaki]

based on https://cibuildwheel.pypa.io/en/stable/faq/#windows-arm64 and https://github.com/pypa/cibuildwheel/blob/5af5df4e4fb8b50198e0b8c6182903cd6e349c8a/cibuildwheel/windows.py#L163 it depends on VSCMD_ARG_TGT_ARCH environment variable, but you need to check setuptools code to validate how to translate it into compiler options.

[JP-Ellis]

Thanks for pointing that out. From inspecting that file, it looks like cibuildwheel configures various backends automatically. I will raise the question over with Hatch as well to raise awareness.

Having said that, my code ultimately relies on the logic within the packaging library. Should cibuildwheel be compatible with packaging in this scenario?

[henryiii]

it depends on VSCMD_ARG_TGT_ARCH environment variable

That's not in a release yet, correct? It will be in the next one (soon).

[joerick]

it depends on VSCMD_ARG_TGT_ARCH environment variable

That's not in a release yet, correct? It will be in the next one (soon).

Right. But this is just for setuptools.

Having said that, my code ultimately relies on the logic within the packaging library. Should cibuildwheel be compatible with packaging in this scenario?

Packaging (see here) uses sysconfig.get_platform(), which is unaffected by environment variables.

So packaging sys_tags isn't going to give you the answer you're after. (Indeed, the function's docstring states- "Returns the sequence of tag triples for the running interpreter.", which isn't what you're after either).

I'd recommend looking for an environment variable in your script (you can set one in your pyproject.toml via an override, or wait for VSCMD_ARG_TGT_ARCH), and falling back to sysconfig.get_platform() if that's not set.

[joerick]

The VSCMD_ARG_TGT_ARCH env var is the method you'll probably want to use to get the target architecture on windows from now on.

[XuehaiPan]

I'd recommend looking for an environment variable in your script (you can set one in your pyproject.toml via an override, or wait for VSCMD_ARG_TGT_ARCH), and falling back to sysconfig.get_platform() if that's not set. The VSCMD_ARG_TGT_ARCH env var is the method you'll probably want to use to get the target architecture on windows from now on.

Hi, I wonder if there is any example or best practice I can refer to?

I ran into a similar Windows ARM64 cross-compiling issue that was built with the latest cibuildwheel on GitHub-hosted Windows x86_64 runner.

https://github.com/metaopt/optree/actions/runs/12769845006/job/35593365002#step:10:416

  copying build\lib.win-arm64-cpython-313\optree\version.py -> build\bdist.win-arm64\wheel\.\optree
- copying build\lib.win-arm64-cpython-313\optree\_C.cp313-win_amd64.pyd -> build\bdist.win-arm64\wheel\.\optree
  copying build\lib.win-arm64-cpython-313\optree\_C.pyi -> build\bdist.win-arm64\wheel\.\optree
  copying build\lib.win-arm64-cpython-313\optree\__init__.py -> build\bdist.win-arm64\wheel\.\optree

I think cibuildwheel has set VSCMD_ARG_TGT_ARCH.

https://github.com/pypa/cibuildwheel/blob/25e87d8558d8349c29b00c567e30a945f2e903ff/cibuildwheel/windows.py#L158-L163

The built C extension is suffixed with .cp313-win_amd64.pyd (https://github.com/metaopt/optree/issues/182#issuecomment-2589853703). It cannot be imported with ARM Python on Windows:

>>> import _C
ModuleNotFoundError: No module named '_C'

If I manually rename the suffix to .cp313-win_arm64.pyd, it raises:

>>> import _C
ImportError: DLL load failed while importing _C: %1 is not valid Win32 application