Support macOS arm64 processors

[tomwhite]

I have a new Mac Mini with an Apple M1 chip. It would be good to be able to run (and develop) sgkit on this architecture.

[jeromekelleher]

I'll be curious to hear what the install experience is like for our dependencies :wink: (See discussion in #518 )

I'd imagine pip install is particularly entertaining?

[tomwhite]

@jeromekelleher you were right about pip! Numpy installs fine, but scipy has a build error (https://github.com/scipy/scipy/issues/13409), and there are no llvmlite wheels yet (needed for numba, https://pypi.org/project/llvmlite/#files). There is progress being made on these issues, but it's not a smooth experience yet.

Things are better for conda. I installed miniforge, which explicitly supports arm64 on macOS, via brew install miniforge. Then I used this env (which I derived from requirements.txt and requirements-dev.txt):

name: sgkit-dev-3.8
dependencies:
  - python=3.8
  - numpy
  - xarray
  - dask[array] != 2021.5.1
  - distributed != 2021.5.1
  - scipy
  - typing-extensions
  - numba
  - zarr
  - fsspec != 2021.6.0
  - codecov
  - pre-commit
  - pytest
  - pytest-cov
  - pytest-datadir
  - pytest-mock
  - hypothesis
  - statsmodels
  #- msprime>=1.0
  - scikit-learn
  - partd
  #- bed-reader
  - rechunker == 0.3.3
  #- cbgen
  #- cyvcf2
  - yarl
  - matplotlib
  - asv
  - networkx
  - aiohttp
  - requests
  - pip:
    - callee
    - scikit-allel

With this I was able to successfully run all non-IO tests.

The commented out libraries in the above env still need work though:

• msprime
  • There is no osx-arm64 version on https://anaconda.org/conda-forge/msprime
  • Need to add tskit too (first?), see https://github.com/conda-forge/tskit-feedstock/issues/25
• bed-reader
  • There is no osx-arm64 version on https://anaconda.org/conda-forge/bed-reader
  • https://github.com/conda-forge/bed-reader-feedstock/issues/6
• cbgen
  • There is no osx-arm64 version on https://anaconda.org/conda-forge/cbgen
  • https://github.com/conda-forge/cbgen-feedstock/issues/6
• cvcf2
  • There is no osx-arm64 version on https://anaconda.org/bioconda/cyvcf2, but I'm not sure bioconda has support for this architecture yet
  • So probably better to try to get a pip build going, so I opened https://github.com/brentp/cyvcf2/issues/221

I found these posts useful:

• M1 background: https://uwekorn.com/2021/01/04/first-two-weeks-with-the-m1.html
• Another perspective: https://ericmjl.github.io/blog/2021/1/27/experience-with-m1-macbook-air/ 6 months later, I think arm64 by default is the way to go for Homebrew and Python (rather than Rosetta)

[jeromekelleher]

Thanks @tomwhite - noble work! I'll try to follow up on the tskit/msprime side of things on conda-forge.

[tomwhite]

Thanks @jeromekelleher

[jeromekelleher]

msprime and tskit should be accounted for now @tomwhite

[tomwhite]

That's great! I just successfully ran the sgkit popgen tests on Mac Mini.

[tomwhite]

There are now macos arm64 packages for cyvcf2 on PyPi (since release 0.30.12).

I was able to run all tests (except bgen and plink) using the environment file in this branch: https://github.com/tomwhite/sgkit/tree/apple-m1.

[tomwhite]

Numba wheels should be available in the next release according to https://github.com/numba/numba/issues/7951#issuecomment-1126043337

[tomwhite]

I updated my branch at https://github.com/tomwhite/sgkit/tree/apple-m1. I can now successfully run the bgen tests now that there is a cbgen build for osx-arm64 on https://anaconda.org/conda-forge/cbgen.

The only package missing now is bed-reader, see https://github.com/conda-forge/bed-reader-feedstock/issues/6 and https://anaconda.org/conda-forge/bed-reader

[tomwhite]

New issues for arm64 wheels for bed-reader and cbgen: https://github.com/fastlmm/bed-reader/issues/13, https://github.com/limix/cbgen/issues/14