support for in-memory movies with patch mode?

[mschachter]

I would like to use the CNMF object in patch mode, but with a matrix Y that is in-memory, and not from a memmapped file. However when I try this, the code fails because it's required that Y is memory-mapped. Would it be possible to remove this constraint?

[epnev]

Not at the moment. I modified the code in the past to do that but it ended copying the entire dataset on each worker causing all sorts of memory overload. We would need to modify the code at a deeper level to achieve that.

[mschachter]

Thanks for letting me know. It makes sense that you did it that way, given that you're using ipcluster on a distributed network with a network mounted drive. I'm working on an implementation that applies greedyROI_corr to patches, but from smaller sequences of movies, so memory usage can be limited even more than by setting the number of processes. However, I'm also benchmarking the CNMF object in 1P mode in tandem, so it's not like I'm blocked or anything. Thanks for the response and all the help from the CaImAn team!

[agiovann]

@mschachter a solution that a some point one could try to implement is to use some shared memory techniques. Less efficiently, but easier to directly load chunks of tiff files in each process. Both are possible but will need some thinking and work.

[EricThomson]

Closing as issue #750 seems to push the discussion along further, I'll leave that one open.