openPMD-standard icon indicating copy to clipboard operation
openPMD-standard copied to clipboard

Proposed: make option "recordBased" for iterationEncoding

Open cosmonaut-ok opened this issue 4 years ago • 1 comments

It's not comfortable to make separate file/group for each iteration in case of active working with time series (e.g. if required to select single point from every iteration). Much better to make n+1-dimensional array and write data to it. But, it breaks openPMD compatibility.

cosmonaut-ok avatar Jan 04 '21 15:01 cosmonaut-ok

Hi @cosmonaut-ok,

Thank you for the idea!

We plan something very similar, currently drafted as "variableBased" encoding related to #221. In that, we will use the intrinsic capabilities of e.g. the ADIOS2 format to encode steps inside a data set (variable) if the file format supports this.

n+1 dimensionality in other formats, e.g. HDF5, is thinkable as a work-around for those formats. Yet we need to be aware that contrary to ADIOS1/2 "step" encoding, HDF5 would then be limited to stay with the same shape of the data set for all iterations; which is limiting for some applications (e.g. particle data shape changes over time).

ax3l avatar Jan 07 '21 22:01 ax3l