Cannot read "stepless" BP5 files with `adios2_mode_read`

[eschnett]

The test case in Julia package ADIOS2.jl first writes a file, and then reads it back in.

These are the steps to write the file:

• adios2_init_serial
• adios2_declare_io
• adios2_define_variable
• adios2_open
• adios2_put
• adios2_perform_puts
• adios2_close
• adios2_finalize Note that the variables are defined before the engine is opened.

These are the steps to read the file:

• adios2_init_serial
• adios2_declare_io
• adios2_open (with adios2_mode_read)
• adios2_inquire_all_variables The last step succeeds, but reports 0 variables in the file. This is wrong, and bpls confirms that all the variables are indeed present in the file.

I find that I can avoid the error by opening the file with adios2_mode_readRandomAccess. This is also what bpls seems to be doing.

Things are also working fine when I choose BP4 as engine instead of the default BP5.

Given that adios2_mode_readRandomAccess was only introduced in ADIOS2 2.9, and that adios2_mode_read was working fine in previous versions of ADIOS2, this looks like a bug.

I am not using steps in this code. Is adios2_mode_read supposed to work in this case?

[eisenhauer]

Unfortunately it was necessary to separate out our two types of reading methods in BP5. Now adios2_mode_read requires the use of BeginStep/EndStep, and adios2_mode_readRandomAccess forbids their use. This change does represent an incompatibility with prior behavior, but it was deemed necessary because the prior semantics incurred large, unavoidable but unnecessary overheads on file Open() (requiring the loading and processing of all metadata immediately, even if the use of Begin/EndStep meant that data was going to be processed a step at a time). For what it's worth, there is a comment about it in the release notes (https://adios2.readthedocs.io/en/latest/introduction/whatsnew.html). Sorry for any inconvenience, but you'll have to use readRandomAccess with BP5. BP3/4 remain unchanged, with readRandomAccess simply being an alias for adios2_mode_read as far as they are concerned.

[pnorbert]

Also note that there is no "stepless" file in adios. The write side API seems to be stepless in that does not require steps, but the resulting file has 1 step

[pnorbert]

What you have been used to was a side effect of the read implementation in previous file engines. BP5 and staging is now fully compatible if the code uses Read mode and BeginStep/EndStep.

It can also support random access mode that staging engines don't support, and in this case BeginStep/EndStep is forbidden.

[eschnett]

Thanks for the explanation.

Would it be possible to detect this application behaviour at run time and emit a warning or an error? If reading from a file using mode_read without begin_step is not supported, could that return an error instead of hiding all variables? This would make it easier for others to diagnose this problem.

On a related note, the error message that old ADIOS libraries produce when trying to read a BP5 file is unclear. I first thought I had somehow corrupted a file. Going forward, you could e.g. encode the file format name as string in the file's header, allowing the current ADIOS version to emit a clear error message when encountering BP6.

[eisenhauer]

Actually, there should be an exception raised in the C++ interface, but maybe Julia goes through the C interface and that gets lost. It may be a matter of making sure that all the error return codes are checked, that exceptions set them appropriately, etc. It's probably the case that our test suite for other languages focuses more on checking that things work than checking that it fails appropriately when it doesn't...