WHL: MacOS arm64 Wheels

[snowman2]

Currently don't have CI runners to build/test them. There are options to potentially cross compile, but the wheels wouldn't be able to be tested. Not sure what the best way to handle this is.

Related:

• https://github.com/rasterio/rasterio-wheels/issues/75
• https://github.com/geopandas/pyogrio/pull/64

[vot4anto]

I built wheel for pyproj 3.3.1 for arm64 python3.9 and python3.10. If you teel me how put on pypip I Will do with pleasure. I test these wheel with pytest and the tests are all passed

[snowman2]

@vot4anto, there are a couple of ways to do it:

1. You can upload the files here or share a link to the files and I can upload them manually.
2. If you able to allow the pyproj project to use your GitHub Action ARM64 runners, we could use them to build & upload wheels automatically to pypi. This will be helpful for future releases.

[vot4anto]

Unfortunately our GitHub Action ARM64 runners are not always online. Since now i have to poweron It to use, I still work how use a scripts with MacVm to create It on demand. Follow link to the wheels and you can download .

https://wheelhouse.openquake.org/macos/arm64/py39/pyproj-3.3.1-cp39-cp39-macosx_11_0_arm64.whl

And

https://wheelhouse.openquake.org/macos/arm64/py310/pyproj-3.3.1-cp310-cp310-macosx_11_0_arm64.whl

Hope these wheels works fine, with pur OpenQuake engine works as expected

[snowman2]

First of all, thank you for sharing links to the wheels you uploaded :+1:

I looked at the wheels recently uploaded to pypi for x86_64 and it has these libraries included:

This is the script we use to prepare the wheels for MacOS: https://github.com/pyproj4/pyproj/blob/main/ci/proj-compile-wheels.s

These are the libraries included in the wheels you provided:

You mentioned that everything worked for you as expected. I wanted to clarify and ask if you tested the PROJ Network capabilities with the wheels you generated?

[vot4anto]

Hi, I use this script to build the wheel: https://github.com/gem/oq-builders/blob/master/.github/workflows/pyproj_cibuild_m1.yml

And essentially I copy your script and use the source of 3.3.1 to build the wheel. Unfortunately there was a lot of failure due the configuration of the runner, for example the configuration of sudo for user runner and other things like that and always I need to recreate from scratch the VM of the runner

[vot4anto]

I run again the actions and I have on the wheel the same contents, I also see the follows in the test section of cibuildwheel:

  test/test_transformer.py::test_network__disable[transformer0] PASSED
  test/test_transformer.py::test_network__disable[transformer1] PASSED
  test/test_transformer.py::test_network__disable[transformer2] PASSED
  test/test_transformer.py::test_network__enable[transformer0] PASSED
  test/test_transformer.py::test_network__enable[transformer1] PASSED
  test/test_transformer.py::test_network__enable[transformer2] PASSED
  test/test_transformer.py::test_network__default[transformer0] PASSED
  test/test_transformer.py::test_network__default[transformer1] PASSED
  test/test_transformer.py::test_network__default[transformer2] PASSED

@snowman2 Are those the test that you mention?

[snowman2]

Looks good. It looks like you are testing with PROJ_NETWORK=ON, so I think it should be good. Would you mind adding PROJ 8.2 to the wheels? It is what the other wheels were build with for the 3.3.1 release (https://github.com/pyproj4/pyproj/discussions/1063).

[vot4anto]

I compile the wheel with the env variabile PROJ=8.2.1 in first versione of wheel and after with PROJ=9.0.0 It Is enough? Or I need to change something in the scripts to adding PROJ 8.2 ?

[snowman2]

PROJ_VERSION was 8.2.0 for the 3.3.1 wheels. It would be nice to keep it consistent.

When the 3.4.0 pyproj release happens, the wheels will contain the latest version of PROJ at the time of the release.

[vot4anto]

I set PROJ_VERSION to 8.2.0 and recreate the wheels attached

pyproj_wheel.zip

[snowman2]

Thanks @vot4anto 👍. They are on pypi now.

[dansd]

Does that mean that we should be able to install pyproj by pip downloading pre-built wheels? I have a M1 Macbook, and can not get it to install without further requirements:

$ python --version
Python 3.10.6

$ python -m pip install pyproj
Collecting pyproj
  Using cached pyproj-3.4.0.tar.gz (217 kB)
  Installing build dependencies ... done
  Getting requirements to build wheel ... error
  error: subprocess-exited-with-error
  
  × Getting requirements to build wheel did not run successfully.
  │ exit code: 1
  ╰─> [1 lines of output]
      proj executable not found. Please set the PROJ_DIR variable. For more information see: https://pyproj4.github.io/pyproj/stable/installation.html
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
error: subprocess-exited-with-error

× Getting requirements to build wheel did not run successfully.
│ exit code: 1
╰─> See above for output.

note: This error originates from a subprocess, and is likely not a problem with pip.

After installing proj with brew install proj, it did work:

$ python -m pip install pyproj
Collecting pyproj
  Using cached pyproj-3.4.0.tar.gz (217 kB)
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
Collecting certifi
  Downloading certifi-2022.9.14-py3-none-any.whl (162 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 162.5/162.5 kB 3.3 MB/s eta 0:00:00
Building wheels for collected packages: pyproj
  Building wheel for pyproj (pyproject.toml) ... done
  Created wheel for pyproj: filename=pyproj-3.4.0-cp310-cp310-macosx_12_0_arm64.whl size=413134 sha256=dabb12412967ba21fc4d88731a7ff189337034cb9240470842ef322c85435572
  Stored in directory: /Users/user/Library/Caches/pip/wheels/88/9b/b4/515b18b3dcb8e3af3b25ed03a391e4ba0f0246fa7d465dc9e5
Successfully built pyproj
Installing collected packages: certifi, pyproj
Successfully installed certifi-2022.9.14 pyproj-3.4.0

[vot4anto]

I'm sorry but I have some trouble and can't have time to build whl for 3.4.0. I will do asap

[vot4anto]

cibuildwheel project now add M1 support with Cirrus CI https://github.com/pypa/cibuildwheel Try to use it with fork of proj 3.4.0 since I have test failed with old method:

 test/test_sync.py::test_get_transform_grid_list__contains FAILED

 -- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
  =========================== short test summary info ============================
  FAILED test/test_sync.py::test_get_transform_grid_list__contains - AssertionE...
  ============ 1 failed, 838 passed, 1 skipped, 47 warnings in 18.51s ============

[vot4anto]

If I compile the wheel from main I don't have any errors, instead the build faild on tag and new branch https://github.com/vot4anto/pyproj/tree/3.4.0-arm64 that I have created to built the wheel. As you can see from the follow: Failed: https://github.com/vot4anto/pyproj/actions/runs/3077093832 Success: https://github.com/vot4anto/pyproj/actions/runs/3076994603

In the successfull build there are the wheel for python3.9, 3.10:

  Length      Date    Time    Name
---------  ---------- -----   ----
  3792707  2022-09-18 11:24   pyproj-3.4.1.dev0-cp310-cp310-macosx_11_0_arm64.whl
  3798222  2022-09-18 11:24   pyproj-3.4.1.dev0-cp39-cp39-macosx_11_0_arm64.whl

Unfortunately with [email protected] don't compile python 3.11 and with higher version I have this error:

Version 3.7 - 3.10 was not found in the local cache
Error: Version 3.7 - 3.10 with arch arm64 not found
The list of all available versions can be found here: https://raw.githubusercontent.com/actions/python-versions/main/versions-manifest.json

[snowman2]

The test failure is okay - see #1145. You can safely skip it.

If you update to cibuildwheel 2.9, you should be able to build Python 3.11 wheels.

[vot4anto]

I can build the wheel for python 3.9 and 3.10 after you suggestion and attach to this comment. Unfortunately if I use cibuildwheel 2.9 or 2.10 the all process fail with the follow error:

Run actions/setup-python@v4
Version 3.7 - 3.10 was not found in the local cache
Error: Version 3.7 - 3.10 with arch arm64 not found

As you can see: https://github.com/vot4anto/pyproj/actions/runs/3080187090/jobs/4977249167

I think something is changed from those release of cibuildwheel

wheel_arm64.zip

[vot4anto]

I change the workflow and now I can use also python 3.11 But the test of the wheel fails for python 3.11

 ==================================== ERRORS ====================================
  ______________________ ERROR collecting test/test_geod.py ______________________
  test/test_geod.py:18: in <module>
      from shapely.geometry import (
  /private/var/folders/jd/2kp_brz51hl61lhbbsxp9dy80000gn/T/cibw-run-gefupi54/cp311-macosx_arm64/venv-test/lib/python3.11/site-packages/shapely/geometry/__init__.py:4: in <module>
      from .base import CAP_STYLE, JOIN_STYLE
  /private/var/folders/jd/2kp_brz51hl61lhbbsxp9dy80000gn/T/cibw-run-gefupi54/cp311-macosx_arm64/venv-test/lib/python3.11/site-packages/shapely/geometry/base.py:20: in <module>
      from shapely.coords import CoordinateSequence
  /private/var/folders/jd/2kp_brz51hl61lhbbsxp9dy80000gn/T/cibw-run-gefupi54/cp311-macosx_arm64/venv-test/lib/python3.11/site-packages/shapely/coords.py:10: in <module>
      from shapely.geos import lgeos
  /private/var/folders/jd/2kp_brz51hl61lhbbsxp9dy80000gn/T/cibw-run-gefupi54/cp311-macosx_arm64/venv-test/lib/python3.11/site-packages/shapely/geos.py:164: in <module>
      _lgeos = load_dll('geos_c', fallbacks=alt_paths)
  /private/var/folders/jd/2kp_brz51hl61lhbbsxp9dy80000gn/T/cibw-run-gefupi54/cp311-macosx_arm64/venv-test/lib/python3.11/site-packages/shapely/geos.py:54: in load_dll
      raise OSError(
  E   OSError: Could not find lib geos_c or load any of its variants ['/Library/Frameworks/GEOS.framework/Versions/Current/GEOS', '/opt/local/lib/libgeos_c.dylib', '/usr/local/lib/libgeos_c.dylib', '/opt/homebrew/lib/libgeos_c.dylib'].
  =============================== warnings summary ===============================

https://github.com/vot4anto/pyproj/actions/runs/3080277927/jobs/4977431916

[snowman2]

Thanks @vot4anto :+1:

For the 3.11 wheels, you should be able to skip installing shapely and the tests should skip the tests requiring shapely.

[vot4anto]

Ok, I will update the workflow to skip installation only on python 3.11 and I will upload the wheel asap

[snowman2]

cibuildwheel project now add M1 support with Cirrus CI https://github.com/pypa/cibuildwheel

That is good to know, thanks :+1:. Hopefully we can get that setup so we can automate this.

[vot4anto]

As suggested I skip installation of shapely and the wheel are created also for 3.11

artifact.zip

[snowman2]

Thanks @vot4anto :+1:

[jorisvandenbossche]

@snowman2 FYI I just added MacOS arm64 wheels to pyogrio's cibuildwheel setup (using github actions and cross-compiling, so without being able to run tests (but that's also what eg conda-forge does)): https://github.com/geopandas/pyogrio/pull/64 (if this works for GDAL, it should also work for PROJ ;))

[snowman2]

Thanks for the update @jorisvandenbossche :+1: