CaImAn
CaImAn copied to clipboard
map_async causing issue when doing CNMF using ipyparallel
Your setup:
- Operating System (Linux, MacOS, Windows): Linux
- Hardware type (x86, ARM..) and RAM: x86, 128 GB
- Python Version (e.g. 3.9): 3.11.8
- Caiman version (e.g. 1.9.12): 1.11.0
- Which demo exhibits the problem (if applicable): N/A
- How you installed Caiman (pure conda, conda + compile, colab, ..): conda + compile
- Details:
I'm getting the following error when running CNMF with ipyparallel:
Traceback (most recent call last):
File "/home/ethan/mambaforge/envs/caiman/lib/python3.11/site-packages/ipyparallel/client/asyncresult.py", line 712, in __getattr__
return self.__getitem__(key)
^^^^^^^^^^^^^^^^^^^^^
File "/home/ethan/mambaforge/envs/caiman/lib/python3.11/site-packages/ipyparallel/client/asyncresult.py", line 699, in __getitem__
values = [md[key] for md in self._metadata]
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/ethan/mambaforge/envs/caiman/lib/python3.11/site-packages/ipyparallel/client/asyncresult.py", line 699, in <listcomp>
values = [md[key] for md in self._metadata]
~~^^^^^
KeyError: 'sort'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/mnt/bigdata/eblackwood/2p_analysis/cmcode/caiman_analysis.py", line 559, in do_cnmf
self.cnmf_fit2 = self.cnmf_fit1.refit(images, dview=cluster_info['dview'])
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/synology/code/CalciumImaging/CaImAn/build/__editable__.caiman-1.10.5-cp311-cp311-linux_x86_64/caiman/source_extraction/cnmf/cnmf.py", line 405, in refit
return cnm.fit(images)
^^^^^^^^^^^^^^^
File "/synology/code/CalciumImaging/CaImAn/build/__editable__.caiman-1.10.5-cp311-cp311-linux_x86_64/caiman/source_extraction/cnmf/cnmf.py", line 522, in fit
self.update_spatial(Yr, use_init=True)
File "/synology/code/CalciumImaging/CaImAn/build/__editable__.caiman-1.10.5-cp311-cp311-linux_x86_64/caiman/source_extraction/cnmf/cnmf.py", line 914, in update_spatial
update_spatial_components(Y, C=self.estimates.C, f=self.estimates.f, A_in=self.estimates.A,
File "/synology/code/CalciumImaging/CaImAn/build/__editable__.caiman-1.10.5-cp311-cp311-linux_x86_64/caiman/source_extraction/cnmf/spatial.py", line 238, in update_spatial_components
A_ = threshold_components(A_, dims, dview=dview, medw=medw, thr_method=thr_method,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/synology/code/CalciumImaging/CaImAn/build/__editable__.caiman-1.10.5-cp311-cp311-linux_x86_64/caiman/source_extraction/cnmf/spatial.py", line 511, in threshold_components
res.sort(key=lambda x: x[1])
^^^^^^^^
File "/home/ethan/mambaforge/envs/caiman/lib/python3.11/site-packages/ipyparallel/client/asyncresult.py", line 714, in __getattr__
raise AttributeError(
AttributeError: 'AsyncMapResult' object has no attribute 'sort'. Did you mean: 'abort'?
This seems to be because of the following code in cnmf/spatial.py
, which calls map_async
on the dview and then tries to sort the return value:
if dview is not None:
if 'multiprocessing' in str(type(dview)):
res = dview.map_async(
threshold_components_parallel, pars).get(4294967)
else:
res = dview.map_async(threshold_components_parallel, pars)
else:
res = list(map(threshold_components_parallel, pars))
res.sort(key=lambda x: x[1])
I think most of the other code uses map_sync
for the ipyparallel case - is that what was intended here? Oddly, this code doesn't seem to have changed recently; seems like someone would have run into this already.