LLNL
Add support for ADIOS2
Conduit supports currently only ADIOS 1, and the latest ADIOS 1 release happened 7 years ago. Please add support for ADIOS2.
As context: The Einstein Toolkit collaboration https://einsteintoolkit.org is looking for a widely-supported metadata standard for structured grids with both adapative mesh refinement and multi-patch systems. In particular, we're looking for a standard that is already supported by visualization tools such as VisIt.
It is important for us that we can perform efficient I/O on HPC systems. We've had problems with HDF5 in the past, and ADIOS2 seems to be a library with many of the same features as HDF5 but which is much more focussed on performance.
I don't want to promise anything, but we might also consider contributing towards ADIOS2 support in Conduit.
I wanted to chime in to say that this would also be extremely useful for our codes, especially for writing extracts from Ascent.
I would love for this to exist and we can collaborate on this.
We have Fides (a mesh data model on top of ADIOS2) support in Ascent, however generic Conduit to ADIOS2 would open up many more use cases and more performant I/O.
I've created a working branch for ADIOS2 support: https://github.com/LLNL/conduit/pull/1389
One main difference between HDF5 and ADIOS2 is that HDF5 executes all actions (e.g. "add attribute", "write dataset") directly. ADIOS2, on the other hand, only remembers what needs to be done, and executes all actions at once when the file is flushed.
This means that datasets that are written must not be modified before the file is flushed, and datasets that are read cannot be access before the file is flushed.
How would I best model this in Conduit/Relay? It is obviously possible to execute all actions right away, but for best performance actions need to be delayed. Is there an API or a model for this?
@eschnett
Here is an idea that could help with write and perhaps read if we care careful with documenting the semantics.
We can add a flush to method to the RelayIOHandle Interface.
That will allow the user to choose when to flush and queue up multiple operations.
The simple interfaces (conduit::relay::io::read) that take a path will still be forced to flush, but the RelayIOHandle interface will provide a more efficient option for folks that need it.
The simple interfaces can be implemented in terms of the RelayIOHandle interface
@eschnett https://github.com/LLNL/conduit/pull/1461 added a flush method to the Relay IO Handle Interface.
Also, if needed - we could also expand this to provide options (as a conduit node) to the flush method. For example - if we want to flush reads vs writes independently.