Optimize RAM usage

Open kdreher opened this issue 4 years ago • 2 comments

Currently, the RAM size required by highly resolved simulations is very high. Since we generated the code focussing mostly on performance, it might be beneficial to revisit especially the core modules to see where we could reduce the RAM usage.

Apr 23 '21 07:04 kdreher

I added some manual garbage collection and fixed a memory leak caused by struct.(un)pack in 761e2ab, 2669ac6 and 643c203. At the moment, a simulation of the entire pipeline with a spacing of 0.1mm has a peak RAM usage of ~20GB instead of ~60GB. However, I guess we are far from optimal RAM usage, so this will probably be an ongoing issue.

Apr 29 '21 14:04 kdreher

we might want to consider adding the option to memmap the data fields? While this will most certainly drastically increase simulation time, it might enable high-resolution simulations even without infinite amounts of RAM?

Nov 07 '21 20:11 jgroehl