conda-forge.github.io icon indicating copy to clipboard operation
conda-forge.github.io copied to clipboard
verified publisher icon conda-forge

enable windows arm64

Open beckermr opened this issue 2 years ago • 53 comments

This is a tracking issue to enable windows arm64.

We think the path forward is to cross-compile and do out of band testing on native hardware.

Let's put todo items here:

  • [x] https://github.com/conda-forge/conda-forge-pinning-feedstock/pull/4357
  • [x] Add vs2022 as compiler for arm64 https://github.com/conda-forge/conda-forge-pinning-feedstock/pull/4415
  • [x] Add compilers to vc-feedstock https://github.com/conda-forge/vc-feedstock/pull/55
  • [x] https://github.com/conda-forge/vc-feedstock/pull/58
  • [x] https://github.com/conda-forge/conda-smithy/pull/1742
    • [x] https://github.com/conda-forge/conda-forge-ci-setup-feedstock/pull/248
  • [x] Fix deprecated msvc_env_cmd in conda-build (make it a no-op for win-arm64 platform) https://github.com/conda/conda-build/pull/4867
    • [x] https://github.com/conda-forge/conda-build-feedstock/pull/205
  • [x] https://github.com/conda-incubator/setup-miniconda/pull/299
  • [x] https://github.com/conda-forge/conda-forge-ci-setup-feedstock/pull/300
  • [x] https://github.com/conda-forge/conda-smithy/pull/1808
  • [ ] Build a python entrypoint launcher (cli-arm64.exe) for conda-build
    • [X] https://github.com/conda/conda-launchers/pull/9
    • [ ] https://github.com/conda/conda/pull/14401
    • [ ] https://github.com/conda/conda-build/pull/5545
  • [x] Write a migrator like we did for linux-aarch64/linux-ppc64le resp. osx-arm64 (c.f. this file).
    • [x] Initial list of packages in pinning repo https://github.com/conda-forge/conda-forge-pinning-feedstock/pull/5440
    • [x] https://github.com/regro/cf-scripts/pull/3194

cc @zooba @isuruf @jakirkham @jezdez @chenghlee

beckermr avatar Apr 19 '23 17:04 beckermr

x-refs for related ecosystem issues:

  • Issue conda/conda#11472
  • PR conda/conda#11778
  • PR conda/conda-build#4579`
  • ContinuumIO/anaconda-issues#12957

chenghlee avatar Apr 19 '23 17:04 chenghlee

@zooba I added a few TODOs above if you or someone else want to help.

isuruf avatar Apr 20 '23 16:04 isuruf

@finnagin is looking at the conda-build changes, and will probably have some Qs about any user visible changes (such as how best to inform conda-build it should be doing a cross-compile). Hopefully that will unblock us to start looking at recipe/feedstock changes and figuring out the best way to automatically migrate existing ones (if needed).

zooba avatar Apr 24 '23 16:04 zooba

@zooba, @finnagin, task 4 needs to be done to build some packages. Task 5 is needed only after python itself is built.

isuruf avatar Apr 24 '23 20:04 isuruf

Task 4 is definitely first, yeah, and part of that seems to be plugging in the arch and host_arch configs to allow running an ARM64 targeted build on AMD64 machine. That plus a few recipe updates should enable building Python - we don't actually need conda/conda-build itself to run natively (it'll be nice, but it's not essential).

zooba avatar Apr 25 '23 11:04 zooba

msvc_env_cmd is a deprecated function, so we need not update it to actually work. We just need it to not error out. So, a check for bits to see if it is arm64 and returning without doing anything should be enough.

isuruf avatar Apr 25 '23 16:04 isuruf

@isuruf You mentioned in the call that you got the zlib build working. Can you post how you did that for @finnagin's benefit?

zooba avatar May 03 '23 18:05 zooba

See https://github.com/conda-forge/zlib-feedstock/pull/72

conda build recipe -m .ci_support/win_arm64_.yaml

isuruf avatar May 03 '23 21:05 isuruf

Ah neat, it's using CMake's support. I guess that means the env variables needed were already being set? Or did those get slipped in recently without me noticing?

zooba avatar May 03 '23 22:05 zooba

No, we should add support for an env variable like CMAKE_ARGS which we use in Linux/macOS. The two systems are so similar that even if cmake is not told that it is cross compiling, things just work because the correct compiler is used.

isuruf avatar May 03 '23 22:05 isuruf

Oh cool, I didn't see where the compiler was being picked. Just a vcvarsall call? That should work for anything with MSBuild or setuptools (if relatively up to date) too.

zooba avatar May 03 '23 22:05 zooba

Just a vcvarsall call?

Yes.

C:\Program Files\Microsoft Visual Studio\2022\Enterprise>CALL "VC\Auxiliary\Build\vcvarsamd64_arm64.bat" -vcvars_ver=14.35 10.0.22621.0

isuruf avatar May 03 '23 22:05 isuruf

Thanks @isuruf! This is very helpful!

finnagin avatar May 04 '23 20:05 finnagin

There should be a zlib package for win-arm64 now. It's untested though.

isuruf avatar Jul 06 '23 06:07 isuruf

Can someone with hardware access check the zstd package? Download https://anaconda.org/conda-forge/zstd/1.5.2/download/win-arm64/zstd-1.5.2-h45ec737_7.conda and run

conda build -t /path/to/zstd-1.5.2-h45ec737_7.conda

isuruf avatar Jul 11 '23 19:07 isuruf

Hey @isuruf I can try running that for you! However, I'm currently waiting for access to the network that the machine I would use is on so I likely won't be able to do it until tomorrow.

finnagin avatar Jul 11 '23 21:07 finnagin

Can someone with hardware access check the zstd package? Download https://anaconda.org/conda-forge/zstd/1.5.2/download/win-arm64/zstd-1.5.2-h45ec737_7.conda and run

conda build -t /path/to/zstd-1.5.2-h45ec737_7.conda

yes, I download it, but how to run it? I get the error ">conda build -t zstd-1.5.2-h45ec737_7.conda 'conda' is not recognized as an internal or external command, operable program or batch file."

shijunz avatar Jul 12 '23 09:07 shijunz

yes, I download it, but how to run it?

You need to have conda installed and conda build available. Both of these don't exist for win-arm64 yet, so I think it will be tricky to test these artefacts natively until we've built conda & conda-build...

h-vetinari avatar Jul 12 '23 10:07 h-vetinari

Looking at conda install -n test conda conda-build and filtering out the noarch/ packages, that would mean building at least:

The following NEW packages will be INSTALLED:

  brotli-python      conda-forge/win-64::brotli-python-1.0.9-py311h12c1d0e_9
  bzip2              conda-forge/win-64::bzip2-1.0.8-h8ffe710_4
  ca-certificates    conda-forge/win-64::ca-certificates-2023.5.7-h56e8100_0
  cffi               conda-forge/win-64::cffi-1.15.1-py311h7d9ee11_3
  chardet            conda-forge/win-64::chardet-5.1.0-py311h1ea47a8_0
  conda              conda-forge/win-64::conda-23.5.0-py311h1ea47a8_1
  conda-build        conda-forge/win-64::conda-build-3.25.0-py311h1ea47a8_0
  cryptography       conda-forge/win-64::cryptography-41.0.2-py311h28e9c30_0
  libarchive         conda-forge/win-64::libarchive-3.6.2-h6f8411a_1
  libexpat           conda-forge/win-64::libexpat-2.5.0-h63175ca_1
  libffi             conda-forge/win-64::libffi-3.4.2-h8ffe710_5
  libiconv           conda-forge/win-64::libiconv-1.17-h8ffe710_0
  liblief            conda-forge/win-64::liblief-0.12.3-h63175ca_0
  libsqlite          conda-forge/win-64::libsqlite-3.42.0-hcfcfb64_0
  libxml2            conda-forge/win-64::libxml2-2.11.4-hc3477c8_0
  libzlib            conda-forge/win-64::libzlib-1.2.13-hcfcfb64_5
  lz4-c              conda-forge/win-64::lz4-c-1.9.4-hcfcfb64_0
  lzo                conda-forge/win-64::lzo-2.10-he774522_1000
  m2-msys2-runtime   conda-forge/win-64::m2-msys2-runtime-2.5.0.17080.65c939c-3
  m2-patch           conda-forge/win-64::m2-patch-2.7.5-2
  m2w64-gcc-libgfor~ conda-forge/win-64::m2w64-gcc-libgfortran-5.3.0-6
  m2w64-gcc-libs     conda-forge/win-64::m2w64-gcc-libs-5.3.0-7
  m2w64-gcc-libs-co~ conda-forge/win-64::m2w64-gcc-libs-core-5.3.0-7
  m2w64-gmp          conda-forge/win-64::m2w64-gmp-6.1.0-2
  m2w64-libwinpthre~ conda-forge/win-64::m2w64-libwinpthread-git-5.0.0.4634.697f757-2
  markupsafe         conda-forge/win-64::markupsafe-2.1.3-py311ha68e1ae_0
  menuinst           conda-forge/win-64::menuinst-1.4.19-py311h1ea47a8_1
  msys2-conda-epoch  conda-forge/win-64::msys2-conda-epoch-20160418-1
  openssl            conda-forge/win-64::openssl-3.1.1-hcfcfb64_1
  psutil             conda-forge/win-64::psutil-5.9.5-py311ha68e1ae_0
  py-lief            conda-forge/win-64::py-lief-0.12.3-py311h12c1d0e_0
  pycosat            conda-forge/win-64::pycosat-0.6.4-py311ha68e1ae_1
  python             conda-forge/win-64::python-3.11.4-h2628c8c_0_cpython
  python-libarchive~ conda-forge/win-64::python-libarchive-c-4.0-py311h1ea47a8_2
  python_abi         conda-forge/win-64::python_abi-3.11-3_cp311
  pyyaml             conda-forge/win-64::pyyaml-6.0-py311ha68e1ae_5
  ripgrep            conda-forge/win-64::ripgrep-13.0.0-h7f3b576_2
  ruamel.yaml        conda-forge/win-64::ruamel.yaml-0.17.32-py311ha68e1ae_0
  ruamel.yaml.clib   conda-forge/win-64::ruamel.yaml.clib-0.2.7-py311ha68e1ae_1
  tk                 conda-forge/win-64::tk-8.6.12-h8ffe710_0
  ucrt               conda-forge/win-64::ucrt-10.0.22621.0-h57928b3_0
  vc                 conda-forge/win-64::vc-14.3-h64f974e_17                           ✅ exists for win-arm64
  vc14_runtime       conda-forge/win-64::vc14_runtime-14.36.32532-hfdfe4a8_17          ✅ exists for win-arm64
  vs2015_runtime     conda-forge/win-64::vs2015_runtime-14.36.32532-h05e6639_17        ✅ exists for win-arm64
  xz                 conda-forge/win-64::xz-5.2.6-h8d14728_0
  yaml               conda-forge/win-64::yaml-0.2.5-h8ffe710_2
  zstandard          conda-forge/win-64::zstandard-0.19.0-py311he5d195f_2
  zstd               conda-forge/win-64::zstd-1.5.2-h12be248_7                         ✅ exists for win-arm64

I think it should be possible to stub out m2-patch (at least as long as we're not applying patches, which isn't necessary for testing artefacts), so we avoid having to build the msys2-stack

h-vetinari avatar Jul 12 '23 10:07 h-vetinari

Just install an x86_64 version of conda and conda-build (and windows 11 for x86_64 emulation) and run the command.

isuruf avatar Jul 12 '23 11:07 isuruf

@isuruf when I run conda build -t /path/to/zstd-1.5.2-h45ec737_7.conda I get the following error:

Solving environment: ...working... failed
failed to get install actions, retrying.  exception was: Unsatisfiable dependencies for platform win-arm64: {'cmake', 'pkg-config'}
WARNING:conda_build.build:failed to get install actions, retrying.  exception was: Unsatisfiable dependencies for platform win-arm64: {'cmake', 'pkg-config'}

looks like cmake and pkg-config need arm64 versions first?

finnagin avatar Jul 12 '23 17:07 finnagin

@isuruf when I run conda build -t /path/to/zstd-1.5.2-h45ec737_7.conda I get the following error:

Solving environment: ...working... failed
failed to get install actions, retrying.  exception was: Unsatisfiable dependencies for platform win-arm64: {'cmake', 'pkg-config'}
WARNING:conda_build.build:failed to get install actions, retrying.  exception was: Unsatisfiable dependencies for platform win-arm64: {'cmake', 'pkg-config'}

looks like cmake and pkg-config need arm64 versions first?

(base) C:\Users\shijunz\Downloads>conda build -t zstd-1.5.2-h45ec737_7.conda TEST START: zstd-1.5.2-h45ec737_7.conda Copying package to conda-build croot. No packages otherwise alongside yours will be available unless you specify -c local. To avoid this warning, your package must reside in a channel structure with platform-subfolders. See more info on what a valid channel is at https://conda.io/docs/user-guide/tasks/create-custom-channels.html WARNING:conda_build.build:Copying package to conda-build croot. No packages otherwise alongside yours will be available un less you specify -c local. To avoid this warning, your package must reside in a channel structure with platform-subfolders . See more info on what a valid channel is at https://conda.io/docs/user-guide/tasks/create-custom-channels.html INFO:conda_index.index.convert_cache:Migrate database INFO:conda_index.index.convert_cache:CONVERT .cache INFO:conda_index.index.convert_cache:Migrate database INFO:conda_index.index:noarch cached 0 B from 0 packages at 0 B/second INFO:conda_index.index.convert_cache:Migrate database INFO:conda_index.index.convert_cache:CONVERT .cache INFO:conda_index.index.convert_cache:Migrate database INFO:conda_index.index:win-arm64 cached 253 KB from 1 packages at 22.5 MB/second INFO:conda_index.index:Subdir: noarch Gathering repodata INFO:conda_index.index:noarch Writing pre-patch repodata INFO:conda_index.index:noarch Applying patch instructions INFO:conda_index.index:noarch Writing patched repodata INFO:conda_index.index:noarch Building current_repodata subset INFO:conda_index.index:noarch Writing current_repodata subset INFO:conda_index.index:noarch Writing index HTML INFO:conda_index.index:Completed noarch INFO:conda_index.index:Subdir: win-arm64 Gathering repodata INFO:conda_index.index:win-arm64 Writing pre-patch repodata INFO:conda_index.index:win-arm64 Applying patch instructions INFO:conda_index.index:win-arm64 Writing patched repodata INFO:conda_index.index:win-arm64 Building current_repodata subset INFO:conda_index.index:win-arm64 Writing current_repodata subset INFO:conda_index.index:win-arm64 Writing index HTML INFO:conda_index.index:Completed win-arm64 INFO:conda_index.index:Channeldata subdir: noarch INFO:conda_index.index:Channeldata subdir: win-arm64 Multiple meta files found. The meta.yaml file in the base directory (C:\Users\shijunz\AppData\Local\Temp\tmpgcv9dcgo\info\r ecipe) will be used. WARNING:conda_build.utils:Multiple meta files found. The meta.yaml file in the base directory (C:\Users\shijunz\AppData\Loc al\Temp\tmpgcv9dcgo\info\recipe) will be used. No numpy version specified in conda_build_config.yaml. Falling back to default numpy value of 1.22 WARNING:conda_build.metadata:No numpy version specified in conda_build_config.yaml. Falling back to default numpy value of 1.22 Collecting package metadata (repodata.json): ...working... done Solving environment: ...working... failed failed to get install actions, retrying. exception was: Unsatisfiable dependencies for platform win-arm64: {'cmake', "libz lib[version='>=1.2.13,<1.3.0a0']", 'pkg-config', "vc14_runtime[version='>=14.36.32532']", "vc[version='>=14.3,<15']", 'vs20 22_win-arm64'} WARNING:conda_build.build:failed to get install actions, retrying. exception was: Unsatisfiable dependencies for platform win-arm64: {'cmake', "libzlib[version='>=1.2.13,<1.3.0a0']", 'pkg-config', "vc14_runtime[version='>=14.36.32532']", "vc[ver sion='>=14.3,<15']", 'vs2022_win-arm64'}

Leaving build/test directories: Work: C:\ProgramData\miniconda3\conda-bld\work Test: C:\ProgramData\miniconda3\conda-bld\test_tmp Leaving build/test environments: Test: source activate C:\ProgramData\miniconda3\conda-bld_test_env Build: source activate C:\ProgramData\miniconda3\conda-bld_build_env

TESTS FAILED: zstd-1.5.2-h45ec737_7.tar.bz2

(base) C:\Users\shijunz\Downloads>

shijunz avatar Jul 13 '23 02:07 shijunz

@isuruf when I run conda build -t /path/to/zstd-1.5.2-h45ec737_7.conda I get the following error:

Solving environment: ...working... failed
failed to get install actions, retrying.  exception was: Unsatisfiable dependencies for platform win-arm64: {'cmake', 'pkg-config'}
WARNING:conda_build.build:failed to get install actions, retrying.  exception was: Unsatisfiable dependencies for platform win-arm64: {'cmake', 'pkg-config'}

looks like cmake and pkg-config need arm64 versions first?

cmake should be ok, I build a lot windows on arm64 app using the x64 version cmake. it support crossing build. maybe we should not check the it? or just tell the cmake -A arm64.

but the pkg-config I have no idea.

shijunz avatar Jul 13 '23 03:07 shijunz

Just install an x86_64 version of conda and conda-build (and windows 11 for x86_64 emulation) and run the command.

llecting package metadata (repodata.json): ...working... done Solving environment: ...working... failed failed to get install actions, retrying. exception was: Unsatisfiable dependencies for platform win-arm64: {'pkg-config', "vc[version='>=14.3,<15']", "vc14_runtime[version='>=14.36.32532']", 'cmake', "libzlib[version='>=1.2.13,<1.3.0a0']", 'vs2022_win-arm64'} WARNING:conda_build.build:failed to get install actions, retrying. exception was: Unsatisfiable dependencies for platform win-arm64: {'pkg-config', "vc[version='>=14.3,<15']", "vc14_runtime[version='>=14.36.32532']", 'cmake', "libzlib[version='>=1.2.13,<1.3.0a0']", 'vs2022_win-arm64'}

yes, I run it on the windows 11 for x86-64 emulation mode) and get the similar error.

shijunz avatar Jul 13 '23 03:07 shijunz

cmake should be ok, I build a lot windows on arm64 app using the x64 version cmake. it support crossing build.

If it's relevant, in addition to cross-compiling with the x64 version cmake also has arm64 Windows releases.

finnagin avatar Jul 13 '23 03:07 finnagin

We'll need to disable those tests for now. In the meantime, can you do,

set CONDA_SUBDIR=win-arm64
conda create -n arm64 zstd
set CONDA_SUBDIR=
conda activate arm64
conda config --env --set subdir win-arm64

and then try zstd --version?

isuruf avatar Jul 13 '23 03:07 isuruf

Sure!

>zstd --version
*** Zstandard CLI (64-bit) v1.5.5, by Yann Collet ***

finnagin avatar Jul 13 '23 03:07 finnagin

(arm64) C:\Windows\System32>conda install zstd Collecting package metadata (current_repodata.json): done Solving environment: failed with initial frozen solve. Retrying with flexible solve. Collecting package metadata (repodata.json): done Solving environment: failed with initial frozen solve. Retrying with flexible solve.

PackagesNotFoundError: The following packages are not available from current channels:

  • zstd

Current channels:

  • https://repo.anaconda.com/pkgs/main/win-arm64
  • https://repo.anaconda.com/pkgs/main/noarch
  • https://repo.anaconda.com/pkgs/r/win-arm64
  • https://repo.anaconda.com/pkgs/r/noarch
  • https://repo.anaconda.com/pkgs/msys2/win-arm64
  • https://repo.anaconda.com/pkgs/msys2/noarch

To search for alternate channels that may provide the conda package you're looking for, navigate to

https://anaconda.org

and use the search bar at the top of the page.

shijunz avatar Jul 13 '23 06:07 shijunz

@shijunz, you need to add conda-forge to the list of channels. @finnagin, thanks.

isuruf avatar Jul 13 '23 15:07 isuruf

@shijunz, you need to add conda-forge to the list of channels. @finnagin, thanks.

(env_win-arm64) C:>conda install -c conda-forge zstd Collecting package metadata (current_repodata.json): done Solving environment: done

Package Plan

environment location: C:\Users\shijunz\AppData\Local\miniconda3\envs\env_win-arm64

added / updated specs: - zstd

The following packages will be downloaded:

package                    |            build
---------------------------|-----------------
libzlib-1.2.13             |       h50ba5a6_5          52 KB  conda-forge
vc-14.3                    |      h6ed9334_17          16 KB  conda-forge
vc14_runtime-14.36.32532   |      he64ce00_17        14.4 MB  conda-forge
zstd-1.5.2                 |       h45ec737_7         253 KB  conda-forge
------------------------------------------------------------
                                       Total:        14.7 MB

The following NEW packages will be INSTALLED:

libzlib conda-forge/win-arm64::libzlib-1.2.13-h50ba5a6_5 vc conda-forge/win-arm64::vc-14.3-h6ed9334_17 vc14_runtime conda-forge/win-arm64::vc14_runtime-14.36.32532-he64ce00_17 zstd conda-forge/win-arm64::zstd-1.5.2-h45ec737_7

Proceed ([y]/n)? y

Downloading and Extracting Packages

Preparing transaction: done Verifying transaction: done Executing transaction: done

shijunz avatar Jul 14 '23 00:07 shijunz