devbio-napari
devbio-napari copied to clipboard
haesleinhuepf
Installing arm64 variant on Mac M1
I'm attempting to install devbio-napari on a Mac M1. I do have rosetta installed and there everything works great. I'm failing to setup an environment using the arm64 architecture.
I followed the instructions here and use this command to setup an environment:
CONDA_SUBDIR=osx-arm64 mamba create --name devbio-napari-env python=3.9 devbio-napari redlionfish -c conda-forge
And receive this error:
Looking for: ['python=3.9', 'devbio-napari', 'redlionfish']
conda-forge/noarch Using cache
pkgs/main/noarch [====================] (00m:00s) No change
pkgs/r/osx-arm64 [====================] (00m:00s) Done
pkgs/r/noarch [====================] (00m:00s) No change
pkgs/main/osx-arm64 [====================] (00m:00s) Done
conda-forge/osx-arm64 [====================] (00m:01s) Done
Encountered problems while solving.
Problem: nothing provides __linux needed by devbio-napari-0.6.0-linux_h41d4057_1
Maybe @thawn or @psobolewskiPhD can give me hints? (-:
Thanks!
I can confirm same error using arm64 miniforge3 and fresh env. This seems like a conda/conda-forge issue. Not sure why it's looking for linux? Maybe @goanpeca has some ideas?
Executing the query devbio-napari
conda-forge/osx-arm64 @ 2.0MB/s 1.8s
conda-forge/noarch @ 2.5MB/s 3.9s
Name Version Build Channel
───────────────────────────────────────────────────────────
devbio-napari 0.7.0 linux_h41d4057_0 conda-forge/noarch
devbio-napari 0.7.0 osx_hd1c38e8_0 conda-forge/noarch
devbio-napari 0.7.0 win_h08f2357_0 conda-forge/noarch
Also, I don't get how it's noarch and yet has architectures...
Thanks for the feedback Peter!
Ok, then it might be related to:
- https://github.com/conda-forge/devbio-napari-feedstock/pull/17
CC @kevinyamauchi @jaimergp
CONDA_SUBDIR=osx-arm64 mamba create --name devbio-napari-env python=3.9 "devbio-napari=*=*osx*" redlionfish -c conda-forge
should work. I'll see why it's pulling __linux though.
Alas, looks like a depend isn't arm64 yet:
Looking for: ['python=3.9', 'devbio-napari=[build=*osx*]']
conda-forge/osx-arm64 Using cache
conda-forge/noarch Using cache
Encountered problems while solving:
- nothing provides simpleitk needed by platymatch-0.0.3-pyhd8ed1ab_0
Executing the query simpleitk
conda-forge/osx-arm64 Using cache
conda-forge/noarch Using cache
No entries matching "simpleitk" found
I'll make a PR to add it to the migrator like I did for redlionfish
Done: https://github.com/conda-forge/conda-forge-pinning-feedstock/pull/3480
Yep, that's the conclusion I reached too. We are using noarch packages with __virtual dependencies per OS to implement conditional dependencies. However, the OSX tree is not satisfied (as you saw with platymatch), so the solver sees a conflict for the preferred OSX variant. Then it tries with the other variants, which also conflict because of the __linux and __win dependencies. However, these are "simpler" (shallower, just one leaf) than the platymatch tree, and I guess the simpler conflict is reported.
Log details:
info libsolv ANALYZE UNSOLVABLE ----------------------
info libsolv JOB Rule #231328:
info libsolv devbio-napari-0.7.0-linux_h41d4057_0 [60296] (w1) Conflict.level1
info libsolv devbio-napari-0.7.0-osx_hd1c38e8_0 [60297] (w2) Conflict.level1
info libsolv devbio-napari-0.7.0-win_h08f2357_0 [60298] Conflict.level1
info libsolv next rules: 0 0
info libsolv Rule #68522:
info libsolv !devbio-napari-0.7.0-osx_hd1c38e8_0 [60297] (w1) Conflict.level1
info libsolv platymatch-0.0.3-pyhd8ed1ab_0 [96902] (w2) Conflict.level1
info libsolv next rules: 68523 68576
info libsolv Rule #68542:
info libsolv !devbio-napari-0.7.0-linux_h41d4057_0 [60296] (w1) Conflict.level1
info libsolv next rules: 0 0
info libsolv Rule #68432:
info libsolv !devbio-napari-0.7.0-win_h08f2357_0 [60298] (w1) Conflict.level1
info libsolv next rules: 0 0
info libsolv Rule #23740:
info libsolv !platymatch-0.0.3-pyhd8ed1ab_0 [96902] (w1) Conflict.level1
info libsolv next rules: 0 0
info libsolv enabledisablelearntrules called
Fingers crossed for simpleitk. It hasn't received much attention in a while, apparently.
The ITK devs are active on GitHub, but maybe not following the feedstock repo carefully? Maybe a case for pinging them on their repo: https://github.com/InsightSoftwareConsortium/SimpleITK Or maybe their forums? https://discourse.itk.org
Edit: Once the migrator makes the PR obviously.
Oddly, that one does have arm64 binaries:
Executing the query libsimpleitk
conda-forge/osx-arm64 No change
conda-forge/noarch @ 2.7MB/s 3.7s
Name Version Build Channel
───────────────────────────────────────────────────────
libsimpleitk 2.1.1 hbdafb3b_0 conda-forge/osx-arm64
libsimpleitk 2.1.1 hbdafb3b_1 conda-forge/osx-arm64
libsimpleitk 2.1.0 hbdafb3b_3 conda-forge/osx-arm64
I guess it was manually migrated by the devs?
o.o Then that PR should have been closed because the bot doesn't know otherwise :(
I am fixing 2.2.0 for libsimpleitk, which takes 2h to build or so, apparently. I also closed the arm64 migration PR, so the bot will realize at some point that it can actually migrate simpleitk itself. So at some point we will have 2.2.0 ready and hopefully a green PR for arm64. Now we wait :)
Ok the (lib)simpleitk 2.2.x depends on itk 5.3, not yet released on conda-forge (it's in RC state, so maybe soon), so I am going to focus on the 2.1.x series. In this case, libsimpleitk is available, but the Python bindings (simpleitk) fail on Windows. It could be a trivial fix or something else; we'll see.
I'll keep you posted!
The arm64 bot has made the PR for simpleitk:
https://github.com/conda-forge/simpleitk-feedstock/pull/26
so there's progress!
Thanks for all your efforts Jaime! ❤️
Awesome, big thanks @jaimergp @psobolewskiPhD
Also enjoy your weekend! 🌞
Yup, it's live now. Trying devbio-napari now...
It's going 389 packages!
Definitely one to use mamba and not conda 🤣
Napari is outdated, 0.4.15, but it runs.
Trying some plugins now.
napari-cellpose is fine.
But I get an error with the assistant and clesperanto:
========================= Errors for plugin 'clEsperanto' =========================
napari version: 0.4.15
ERROR #1: Error while importing module napari_pyclesperanto_assistant._napari_plugin
---------------------------------------------------------------------------
LogicError Traceback (most recent call last)
File ~/Dev/miniforge3/envs/test-env/lib/python3.9/site-packages/napari_plugin_engine/manager.py:318, in PluginManager._load_and_register(self=, mod_name='napari_pyclesperanto_assistant._napari_plugin', plugin_name='clEsperanto')
317 try:
--> 318 module = load(mod_name)
mod_name = 'napari_pyclesperanto_assistant._napari_plugin'
319 if self.is_registered(module):
File ~/Dev/miniforge3/envs/test-env/lib/python3.9/site-packages/napari_plugin_engine/manager.py:1042, in load(value='napari_pyclesperanto_assistant._napari_plugin')
1041 raise ValueError(f"malformed entry point string: {value}")
-> 1042 module = importlib.import_module(match.group('module'))
match =
1043 attrs = filter(None, (match.group('attr') or '').split('.'))
File ~/Dev/miniforge3/envs/test-env/lib/python3.9/importlib/__init__.py:127, in import_module(name='napari_pyclesperanto_assistant._napari_plugin', package=None)
126 level += 1
--> 127 return _bootstrap._gcd_import(name[level:], package, level)
level = 0
name = 'napari_pyclesperanto_assistant._napari_plugin'
name[level:] = 'napari_pyclesperanto_assistant._napari_plugin'
package = None
_bootstrap =
File :1030, in _gcd_import(name='napari_pyclesperanto_assistant._napari_plugin', package=None, level=0)
File :1007, in _find_and_load(name='napari_pyclesperanto_assistant._napari_plugin', import_=)
File :972, in _find_and_load_unlocked(name='napari_pyclesperanto_assistant._napari_plugin', import_=)
File :228, in _call_with_frames_removed(f=, *args=('napari_pyclesperanto_assistant',), **kwds={})
File :1030, in _gcd_import(name='napari_pyclesperanto_assistant', package=None, level=0)
File :1007, in _find_and_load(name='napari_pyclesperanto_assistant', import_=)
File :986, in _find_and_load_unlocked(name='napari_pyclesperanto_assistant', import_=)
File :680, in _load_unlocked(spec=ModuleSpec(name='napari_pyclesperanto_assistant'...9/site-packages/napari_pyclesperanto_assistant']))
File :850, in exec_module(self=, module=)
File :228, in _call_with_frames_removed(f=, *args=( at 0x17d3d7c90, file "/Use...ari_pyclesperanto_assistant/__init__.py", line 1>, {'__builtins__': {'ArithmeticError': , 'AssertionError': , 'AttributeError': , 'BaseException': , 'BlockingIOError': , 'BrokenPipeError': , 'BufferError': , 'BytesWarning': , 'ChildProcessError': , 'ConnectionAbortedError': , ...}, '__cached__': '/Users/piotrsobolewski/Dev/miniforge3/envs/test-...nto_assistant/__pycache__/__init__.cpython-39.pyc', '__doc__': None, '__file__': '/Users/piotrsobolewski/Dev/miniforge3/envs/test-...ckages/napari_pyclesperanto_assistant/__init__.py', '__loader__': , '__name__': 'napari_pyclesperanto_assistant', '__package__': 'napari_pyclesperanto_assistant', '__path__': ['/Users/piotrsobolewski/Dev/miniforge3/envs/test-...n3.9/site-packages/napari_pyclesperanto_assistant'], '__spec__': ModuleSpec(name='napari_pyclesperanto_assistant'...9/site-packages/napari_pyclesperanto_assistant'])}), **kwds={})
File ~/Dev/miniforge3/envs/test-env/lib/python3.9/site-packages/napari_pyclesperanto_assistant/__init__.py:1
----> 1 from ._gui import Assistant
2 from ._convert_to_numpy import *
File ~/Dev/miniforge3/envs/test-env/lib/python3.9/site-packages/napari_pyclesperanto_assistant/_gui/__init__.py:1
----> 1 from ._Assistant import Assistant
File ~/Dev/miniforge3/envs/test-env/lib/python3.9/site-packages/napari_pyclesperanto_assistant/_gui/_Assistant.py:2
1 from __future__ import annotations
----> 2 from ._select_gpu import select_gpu
3 from napari.viewer import Viewer
File ~/Dev/miniforge3/envs/test-env/lib/python3.9/site-packages/napari_pyclesperanto_assistant/_gui/_select_gpu.py:8
3 from napari_tools_menu import register_action
7 @magicgui(Select_GPU={
----> 8 "choices": cle.available_device_names(),
cle =
9 },
10 Note={"widget_type":"Label"},
11 call_button='Select')
12 def gpu_selector(Select_GPU : str, Note:str = "Hint: Do not change the GPU while images are open."):
13 print("Selected device:", cle.select_device(Select_GPU))
File ~/Dev/miniforge3/envs/test-env/lib/python3.9/site-packages/pyclesperanto_prototype/_tier0/_available_device_names.py:32, in available_device_names(dev_type=None, score_key=None)
5 """Retrieve a list of names of available OpenCL-devices
6
7 Parameters
(...)
29 >>> print("Available CPU OpenCL devices:" + str(cpu_devices))
30 """
---> 32 devices = filter_devices(dev_type=dev_type, score_key=score_key)
dev_type = None
score_key = None
33 device_names = [device.name for device in devices]
File ~/Dev/miniforge3/envs/test-env/lib/python3.9/site-packages/pyclesperanto_prototype/_tier0/_device.py:101, in filter_devices(name=None, dev_type=None, score_key=None)
100 devices = []
--> 101 for platform in cl.get_platforms():
cl =
102 for device in platform.get_devices():
LogicError: clGetPlatformIDs failed: PLATFORM_NOT_FOUND_KHR
The above exception was the direct cause of the following exception:
PluginImportError Traceback (most recent call last)
File ~/Dev/miniforge3/envs/test-env/lib/python3.9/site-packages/napari_plugin_engine/manager.py:264, in PluginManager.discover(self=, path=None, entry_point=None, prefix=None, ignore_errors=True)
261 continue
263 try:
--> 264 if self._load_and_register(mod_name, name):
name = 'napari_skimage_regionprops2'
mod_name = 'napari_skimage_regionprops._utilities'
self =
265 count += 1
266 self._id_counts[name] = 1
File ~/Dev/miniforge3/envs/test-env/lib/python3.9/site-packages/napari_plugin_engine/manager.py:322, in PluginManager._load_and_register(self=, mod_name='napari_pyclesperanto_assistant._napari_plugin', plugin_name='clEsperanto')
320 return None
321 except Exception as exc:
--> 322 raise PluginImportError(
plugin_name = 'clEsperanto'
323 f'Error while importing module {mod_name}',
324 plugin_name=plugin_name,
325 cause=exc,
326 )
327 if not (inspect.isclass(module) or inspect.ismodule(module)):
328 raise PluginRegistrationError(
329 f'Plugin "{plugin_name}" declared entry_point "{mod_name}"'
330 ' which is neither a module nor a class.',
331 plugin=module,
332 plugin_name=plugin_name,
333 )
PluginImportError: Error while importing module napari_pyclesperanto_assistant._napari_plugin
================================================================================
Definitely one to use mamba and not conda
Guess why we recommend mamba in the installation instructions 😉
Looking at the pyopencl related packages different between this env (fresh) and my normal env (where pycleesperanto works) is khronos-opencl-icd-loader no idea what this is, but seems like the problem.
Doing a simple test of:
mamba create --name test-env2 python=3.9 napari napari-pyclesperanto-assistant
gives the same issue even with:
In [2]: import pyclesperanto_prototype as cle
In [3]: cle.available_device_names()
But if I mamba install just napari and then pip install napari-pyclesperanto-assistant:
In [1]: import pyclesperanto_prototype as cle
In [2]: cle.available_device_names()
Out[2]: ['Apple M1']
and no khronos-opencl-icd-loader.
If I conda install it into this env...it still works.
So 🤷♂️
Trying mamba create --name test-env2 python=3.9 pyclesperanto-prototype
And pooped on:
Python 3.9.13 | packaged by conda-forge | (main, May 27 2022, 17:01:00)
[Clang 13.0.1 ] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import pyclesperanto-prototype as cle
File "<stdin>", line 1
import pyclesperanto-prototype as cle
^
SyntaxError: invalid syntax
>>> import pyclesperanto_prototype as cle
>>> cle.available_device_names()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/Users/piotrsobolewski/Dev/miniforge3/envs/test-env2/lib/python3.9/site-packages/pyclesperanto_prototype/_tier0/_available_device_names.py", line 32, in available_device_names
devices = filter_devices(dev_type=dev_type, score_key=score_key)
File "/Users/piotrsobolewski/Dev/miniforge3/envs/test-env2/lib/python3.9/site-packages/pyclesperanto_prototype/_tier0/_device.py", line 101, in filter_devices
for platform in cl.get_platforms():
pyopencl._cl.LogicError: clGetPlatformIDs failed: PLATFORM_NOT_FOUND_KHR
The pyclesperanto installation instructions say:
Mac-users please also install this:
conda install -c conda-forge ocl_icd_wrapper_apple
Would you mind giving this a try @psobolewskiPhD ?
@haesleinhuepf I don't have that in my regular napari/CLE env, but I will try:
mamba create --name test-env2 python=3.9 pyclesperanto-prototype ocl_icd_wrapper_apple
Edit: that works.
Python 3.9.13 | packaged by conda-forge | (main, May 27 2022, 17:01:00)
[Clang 13.0.1 ] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import pyclesperanto_prototype as cle
>>> cle.available_device_names()
['Apple M1']
So if khronos-opencl-icd-loader is present then it seems ocl_icd_wrapper_apple is needed. Both or none.
No idea what any of those do and neither seems actually used if installing by pip.
So if
khronos-opencl-icd-loaderis present then it seemsocl_icd_wrapper_appleis needed. Both or none.
This is consistent with what is stated at https://github.com/jrprice/ocl_icd_wrapper
So the combo is needed to permit switchiing/overriding Apple's OpenCL implementation. I'm not sure how wise this is on M1? 🤷♂️
All conda-forge libraries that link to OpenCL do so through the khronos implementation for good ABI compatibility. The ICD loader mechanism allows to replace it with a different implementation, so it should be safe. I don't fully know what those words mean, but it was the conclusion of the OpenMM migration, which also relies on OpenCL.
So it seems the pyclesperanto-prototype conda-forge package needs to also include ocl_icd_wrapper_apple otherwise it's borked, as far as I can tell.
Edit:
Actually, it's probable pyopencl:
mamba create --name test-env2 python=3.9 pyopencl
And then the example from: https://documen.tician.de/pyopencl/
fails with the same error.
If I mamba install ocl_icd_wrapper_apple then it works.
Edit2: @jaimergp should I make an issue on pyopencl-feedstock?
Maybe it's sufficient to add ocl_icd_wrapper_apple here:
https://github.com/conda-forge/pyopencl-feedstock/blob/486e7d8b21036662d63752ed9e82054089045c11/recipe/meta.yaml#L27-L34
Technically you can install any other OpenCL implementation on the environment and it will work. ocl_icd_wrapper just exposes Apple's implementation to the khronos ICD.
It's up to the downstream users of an OpenCL app to provide their implementation of choice. For some other projects, Apple's implementation doesn't cut it so the env should not contain the ICD wrapper.
In this case, devbio-napari can add ocl_icd_wrapper to its runtime deps if that should be the default implementation available.
OK, i just checked, the conda-forge pyopencl breaks on arm64 starting here:
https://github.com/conda-forge/pyopencl-feedstock/commit/7ac3d405734b474c6cbb2ee824f777f50e4a2a98
So
╰─ mamba create --name test-env2 python=3.9 pyopencl=2021.1.6 (base) ─╯
The example works.
Everything after is broken on arm64, as far as I can tell—i've not tested every build, just a few here and there, but same error.
It's broken in the sense that no implementation can be found. But adding pocl (another implementation) should make it work, right?
@jaimergp I have no idea, I can try. Is that the expected behavior though? that packages just don't work? I thought dependencies should be installed along with packages?
Edit:
If I do:
mamba create --name test-env2 python=3.9 pyopencl=2021.2.13
Then pyopencl is non-functional as above.
if I then do mamba install pocl (adds 12 packages), then it works, but interestingly doesn't find my Apple M1, but rather:
[0] <pyopencl.Platform 'Portable Computing Language' at 0x10fd71c30>
🤷♂️
Sorry for taking so much of your time and for being so dense, but this stuff just doesn't make much sense to me.