visium-clustergrammer2 icon indicating copy to clipboard operation
visium-clustergrammer2 copied to clipboard

switch to parquet to lower memory footprint

Open cornhundred opened this issue 4 years ago • 0 comments

Currently we are loading data from a CSV and we have to load the entire CSV into memory. Since this example is running on mybinder, which has limited memory ~1GB (https://github.com/voila-dashboards/voila/issues/521#issuecomment-580135675), we want to be careful with memory usage.

Switching to parquet (and adding the pyarrow requirement) will enable us to load data from select cells (or genes in a transposed parquet) instead of loading everything.

cornhundred avatar Feb 01 '20 14:02 cornhundred