Python wheels for macOS on Apple Silicon

[crisluengo]

The new macOS computers use the new Apple Silicon architecture (arm64), which is not compatible with x86_64. Consequently, a native Python installation on those machines will not be able to install or use the current DIPlib distributions on PyPI.

We need to build specifically for the arm64 architecture. On a macOS 12 (Monterey) machine, even if it is x86_64, it is possible to cross-compile for arm64 (it requires Xcode 12.2, on macOS 11 I can only get Xcode 12.0, don't see how to upgrade). But GitHub Actions does not yet have any macOS 12 images available. See https://github.com/actions/virtual-environments/issues/2187.

For the 3.2 release I will manually build arm64 wheels.

[r614]

Are you planning on uploading the arm64 build anytime soon? Could use it but don't want to build from source atm. :eyes:

[crisluengo]

@r614 We're getting ready for a new release, I'll create arm64 wheels for various Python versions when that's ready. Shouldn't be too long!

[crisluengo]

Created a deploy script, it should be straight-forward to deploy these wheels when we coin the release. 157797b7d68a72951b1556afb318597b41113b66

macOS + Apple Silicon will be supported for Python versions 3.8, 3.9 and 3.10, which are the versions of Python that Homebrew delivers for this platform.

[crisluengo]

@r614 You should be able to install DIPlib for your arm64 macOS machine from PyPI now. Please let me know if you run into any issues!

[r614]

Awesome! I was able to install it but running an example code to load an image runs into an issue with OpenMP.

OMP: Error #15: Initializing libomp.dylib, but found libomp.dylib already initialized.  
OMP: Hint This means that multiple copies of the OpenMP runtime have been linked into the program.  
That is dangerous, since it can degrade performance or cause incorrect results.  
The best thing to do is to ensure that only a single OpenMP runtime is linked into the process, e.g. by avoiding static linking of the OpenMP runtime in any library.  
As an unsafe, unsupported, undocumented workaround you can set the environment variable KMP_DUPLICATE_LIB_OK=TRUE to allow the program to continue to execute, but that may cause crashes or silently produce incorrect results. For more information, please see http://openmp.llvm.org/
zsh: abort      python analyses/dip_test.py

Setting KMP_DUPLICATE_LIB_OK=TRUE results in a segfault. I am not sure if it's a build problem or my own env issue, but figured I'll post here just in case.

[crisluengo]

That is interesting... Do you import other packages that might be using OpenMP?

Which Python are you using?

[r614]

3.9, with sklearn installed from Conda forge - I think that's the only one using OpenMP, though I am also using Pandas, OpenCV, Numpy, Matplotlib etc.

[crisluengo]

Could you do a test running DIPlib without importing any of the other packages? If that works, import packages one by one and see if it still works.

I have not been able to reproduce your issue on my end when combining with any of the packages you mentioned.

[r614]

I tried this in a python script without importing anything else. I was trying to reproduce the alignment example on this StackOverflow question with my own images.

I still get the same error in a fresh conda environment. For reference, here's how it was setup:

conda create -n test_capstone python=3.9
conda activate test_capstone 
conda install openjdk
pip install diplib matplotlib

Python script:

import diplib as dip

# load data
image = dip.ImageRead('IMG_9869.jpg')
template = dip.ImageRead('stl_screenshot_bin.jpg')
template = template.TensorElement(0) 

sz = [max(image.Size(0), template.Size(0)), max(image.Size(1), template.Size(1))]

image = image.Pad(sz)
template = template.Pad(sz)

res = dip.FourierMellinMatch2D(template, image)

dip.JoinChannels((template,res,res)).Show()

[crisluengo]

Thank you for trying that out. This is really strange, I will have to look more deeply into this. :/

[r614]

No problem, happy to help. I'll try running it on another M1 machine and see if I find the same bug.

[crisluengo]

My new arm64 iMac arrived today. I installed Miniconda, then followed your setup and ran the Python script without problems.

[r614]

Thanks for confirming, I'll dig through my modules and see what's causing the issue - glad to see its not the wheels though!

[crisluengo]

Closing because of inactivity. Feel free to reopen if necessary.