Reading an incomplete Series fails

[lucafedeli88]

Describe the bug I am using WarpX ( https://github.com/ECP-WarpX/WarpX , b4067afbd5f487683d6c0f1e5c8ac835d3373fb6) with openPMD output format (ADIOS2 backend). When I try to read a Series from a simulation which is still running or a Series from an output folder generated by a crashed simulation, reading fails with this error message:

[ADIOS2] Warning: Attribute with name /openPMD has no type in backend.
[ADIOS2] Warning: Attribute with name /openPMD has no type in backend.
terminate called after throwing an instance of 'std::runtime_error'
  what():  [ADIOS2] Requested attribute (/openPMD) not found in backend.
Aborted (core dumped)

Data can be read only if the last timestep is deleted.

To Reproduce Run a simulation with WarpX using openPMD output format and make it stop prematurely (e.g. by killing it). Then read some data with the following instructions:

import openpmd_api as io 
series = io.Series("diags/sp_bp/simData_%T.bp", io.Access_Type.read_only) 

Software Environment

• version of openPMD-api: 0.12.0 (for writing), 0.11.1 (for reading)
• installed openPMD-api via: from source (for writing), from conda (for reading)
• operating system: Red Hat Enterprise Linux Server release 7.6 (Maipo) for writing (7.7 for reading)
• machine: Summit (OLCF) for writing, Rhea (OLCF) for reading
• name and version of Python implementation: Python 3.8.2
• version of ADIOS2: 2.5.0

[ax3l]

Thank you for documenting this @lucafedeli88 !

cc @franzpoeschel As discussed elsewhere in the past, we want to address this generally by introducing an Iteration::close() method to release handles as early as possible. Most workflows will know a definite point in time in which a write will not contribute to an iteration anymore. Do you want to take the lead on that one since it is a cut-topic with staging efforts in #570 or shall I draft an implementation for the current methods?

Independent of that, I will submit a PR that reading of incomplete series does work gracefully by gracefully skipping of such files as I did in https://github.com/openPMD/openPMD-viewer/pull/262

[franzpoeschel]

I already have this functionality on my streaming branch, will separate this into a PR next week @ax3l

[ax3l]

Link: #746 :)