MouseLand
Calculate memory needed
When applying a custom model to a ~3 GB 3D data set (1000x1024x ~1500 slices, 16-bit grayscale), I encountered issues with running out of memory.
I've seen several issues on this topic, but I can no longer find a specific comment where someone gave a way to calculate how much memory is needed to process data of a given size. Could someone share with me how to do this? I have access to clusters at my university and need to specify how much memory I need for various data sets, which can be up to 20GB.
My PC has 64 GB memory cpu and 24GB gpu. It's not clear to me which memory is used when cellpose is called using GPU or also why it's so much higher than the dataset size. I apologize if this is a naive question.
Thanks!
The main memory usage on the CPU RAM will be for saving the flows. It will be 6x the dataset size for all the flows + in float64 so 4x more than that in your case.
I noticed excessive memory usage in 3D and it should be resolved now, you likely do still need around 8x the image size in terms of ram. you can install the latest version with the update: pip install git+https://github.com/mouseland/cellpose.git
I will also create plots so people know memory required by image size
I've updated the mask creation code to reduce RAM usage, and posted updated benchmarking here: https://cellpose.readthedocs.io/en/latest/benchmark.html. your tiff in 2-channel, float32 format for cellpose would be ~12GB and thus require ~82gb of CPU ram (~6-7x data size) and ~32gb of GPU ram (~2-3x data size). Alternatively you can tile your dataset and run each tile with lower memory usage.