Add storing result on drive, when keeping only intermidiate result in memory
Is your feature request related to a problem? Please describe. It helps to calculate correlations for large arrays with little memory footprint
Describe the solution you'd like Store ready results on drive, using hd5 or something similar
Describe alternatives you've considered A clear and concise description of any alternative solutions or features you've considered.
Additional context https://stackoverflow.com/questions/30109297/pearson-correlation-on-big-numpy-matrices
https://stackoverflow.com/questions/15704010/write-data-to-hdf-file-using-multiprocessing
I would appreciate this as well since computing a high-dimensional correlation matrix is not possible at the moment due to memory issues.