wepy icon indicating copy to clipboard operation
wepy copied to clipboard

Reading weights from the HDF5 is slow

Open alexrd opened this issue 1 year ago • 3 comments

For weighted ensemble analyses it is common to require access to all of the weights at once. Currently, this takes tens of minutes to read the weights from reasonably-sized HDF5 files. In contrast, reading a newly computed observable can be done in seconds.

Could this potentially be helped by arranging them all in their own folder? E.g. ['runs/0/weights'] Or perhaps there is another way of ensuring that the weights are written to some contiguous region of the disk?

alexrd avatar Nov 27 '23 18:11 alexrd