nancorrmp icon indicating copy to clipboard operation
nancorrmp copied to clipboard

Add storing result on drive, when keeping only intermidiate result in memory

Open bukson opened this issue 5 years ago • 2 comments

Is your feature request related to a problem? Please describe. It helps to calculate correlations for large arrays with little memory footprint

Describe the solution you'd like Store ready results on drive, using hd5 or something similar

Describe alternatives you've considered A clear and concise description of any alternative solutions or features you've considered.

Additional context https://stackoverflow.com/questions/30109297/pearson-correlation-on-big-numpy-matrices

bukson avatar Feb 07 '20 11:02 bukson

https://stackoverflow.com/questions/15704010/write-data-to-hdf-file-using-multiprocessing

bukson avatar Feb 07 '20 18:02 bukson

I would appreciate this as well since computing a high-dimensional correlation matrix is not possible at the moment due to memory issues.

g3rfx avatar May 08 '20 20:05 g3rfx