Write into multiple files (per each timestep) for hdf5 engine

[haykh]

Hi, I was wondering if it is possible to configure adios2 to write into multiple hdf5 files (one per each timestep). It is oftentimes very useful, for instance, if I want to read the data in python with h5py, while the simulation is still running (when it's a single file it just crashes the simulation when it tries to read). Also, in many scenarios after the simulation is done, I want to delete most of the timesteps and only leave one for future comparison (of course, I can do that through python, but I think it's much easier to just rm the extra files).

If it's not that straightforward, a possible workaround would be to io.Open("mydata_" + step + ".h5", adios2::Mode::Write) different files at every timestep, but then it automatically creates the master Step0 dataset and writes everything there which doesn't really make sense (maybe there is a way to modify this behavior?).

[eisenhauer]

@guj, just making sure you see this...

[guj]

Thanks for the suggestion. Will put it on our desired feature list.

[anagainaru]

Hey @haykh we discussed your usecase and we recommend using the reorganize utility (https://adios2.readthedocs.io/en/latest/ecosystem/utilities.html#adios-reorganize) to convert adios2 files to HDF5 format instead of writing HDF5 files through the adios2 interface. This way you can use all the facilities we have in adios (streaming, receiving Kokkos::Views directly, applying operators on your datasets etc.) but you can still have your data in HDF5 format for other purposes. From what I remember, your dataset is not too large so you could apply the reorganize command on one core.