simpa icon indicating copy to clipboard operation
simpa copied to clipboard

Simulations are slowed down considerably when compressing output

Open leoyala opened this issue 3 years ago • 1 comments

The simulation time is extended a couple of seconds when Tags.DO_FILE_COMPRESSION=True is in the Settings. This slows down even more other simulations modules such as MCXAdapterReflectance. A first attempt to alleviating this is to save the HDF5 file only once and/or saving results with half precision.

leoyala avatar Feb 17 '22 10:02 leoyala

I think we should make available other compression filters in SIMPA. At the moment we are using gzip as default. However, other filters such as lzf would considerably speed up the compression. A preliminary experiment with the minimal_optical_simulation example in the repo generates something like:

  1. using lzf: time=0.98 seconds resulting in a 177.6MB .hdf5 file
  2. using gzip: time=4.82 seconds resulting in a 158.5MB .hdf5 file

More comparisons can be found here http://www.h5py.org/lzf/. I think we should open these options to the users @jgroehl @kdreher.

leoyala avatar Nov 23 '22 16:11 leoyala