CaImAn icon indicating copy to clipboard operation
CaImAn copied to clipboard

map_async causing issue when doing CNMF using ipyparallel

Open ethanbb opened this issue 1 month ago • 5 comments

Your setup:

  1. Operating System (Linux, MacOS, Windows): Linux
  2. Hardware type (x86, ARM..) and RAM: x86, 128 GB
  3. Python Version (e.g. 3.9): 3.11.8
  4. Caiman version (e.g. 1.9.12): 1.11.0
  5. Which demo exhibits the problem (if applicable): N/A
  6. How you installed Caiman (pure conda, conda + compile, colab, ..): conda + compile
  7. Details:

I'm getting the following error when running CNMF with ipyparallel:

Traceback (most recent call last):                                                                     
  File "/home/ethan/mambaforge/envs/caiman/lib/python3.11/site-packages/ipyparallel/client/asyncresult.py", line 712, in __getattr__                                                                           
    return self.__getitem__(key)                                                                       
           ^^^^^^^^^^^^^^^^^^^^^                                                                                                                                                                               
  File "/home/ethan/mambaforge/envs/caiman/lib/python3.11/site-packages/ipyparallel/client/asyncresult.py", line 699, in __getitem__                                                                           
    values = [md[key] for md in self._metadata]                                                                                                                                                                
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^                                                                                                                                                                
  File "/home/ethan/mambaforge/envs/caiman/lib/python3.11/site-packages/ipyparallel/client/asyncresult.py", line 699, in <listcomp>                                                                            
    values = [md[key] for md in self._metadata]                                                                                                                                                                
              ~~^^^^^                                                                                  
KeyError: 'sort'                                   
                                                                                                                                                                                                               
During handling of the above exception, another exception occurred:                                                                                                                                            
                                                                                                                                                                                                               
Traceback (most recent call last):                                                                     
  File "<stdin>", line 1, in <module>                                                                  
  File "/mnt/bigdata/eblackwood/2p_analysis/cmcode/caiman_analysis.py", line 559, in do_cnmf                                                                                                                   
    self.cnmf_fit2 = self.cnmf_fit1.refit(images, dview=cluster_info['dview'])                                                                                                                                 
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^                                                                                                                                 
  File "/synology/code/CalciumImaging/CaImAn/build/__editable__.caiman-1.10.5-cp311-cp311-linux_x86_64/caiman/source_extraction/cnmf/cnmf.py", line 405, in refit                                              
    return cnm.fit(images)                                                                             
           ^^^^^^^^^^^^^^^                                                                             
  File "/synology/code/CalciumImaging/CaImAn/build/__editable__.caiman-1.10.5-cp311-cp311-linux_x86_64/caiman/source_extraction/cnmf/cnmf.py", line 522, in fit                                                
    self.update_spatial(Yr, use_init=True)                                                             
  File "/synology/code/CalciumImaging/CaImAn/build/__editable__.caiman-1.10.5-cp311-cp311-linux_x86_64/caiman/source_extraction/cnmf/cnmf.py", line 914, in update_spatial                                     
    update_spatial_components(Y, C=self.estimates.C, f=self.estimates.f, A_in=self.estimates.A,                                                                                                                
  File "/synology/code/CalciumImaging/CaImAn/build/__editable__.caiman-1.10.5-cp311-cp311-linux_x86_64/caiman/source_extraction/cnmf/spatial.py", line 238, in update_spatial_components                       
    A_ = threshold_components(A_, dims, dview=dview, medw=medw, thr_method=thr_method,                                                                                                                         
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^                                                                                                                         
  File "/synology/code/CalciumImaging/CaImAn/build/__editable__.caiman-1.10.5-cp311-cp311-linux_x86_64/caiman/source_extraction/cnmf/spatial.py", line 511, in threshold_components                            
    res.sort(key=lambda x: x[1])                                                                       
    ^^^^^^^^                                       
  File "/home/ethan/mambaforge/envs/caiman/lib/python3.11/site-packages/ipyparallel/client/asyncresult.py", line 714, in __getattr__                                                                           
    raise AttributeError(                          
AttributeError: 'AsyncMapResult' object has no attribute 'sort'. Did you mean: 'abort'?  

This seems to be because of the following code in cnmf/spatial.py, which calls map_async on the dview and then tries to sort the return value:

    if dview is not None:
        if 'multiprocessing' in str(type(dview)):
            res = dview.map_async(
                threshold_components_parallel, pars).get(4294967)
        else:
            res = dview.map_async(threshold_components_parallel, pars)
    else:
        res = list(map(threshold_components_parallel, pars))

    res.sort(key=lambda x: x[1])

I think most of the other code uses map_sync for the ipyparallel case - is that what was intended here? Oddly, this code doesn't seem to have changed recently; seems like someone would have run into this already.

ethanbb avatar May 10 '24 23:05 ethanbb