Error while reading BP5 files serially

[shreybh1]

Hi, I am benchmarking IO performance of ADIOS2. In this program, i first write data using the particular IO engine and then read it back serially. When i read this data back serially, the error comes up as:

" [ADIOS2 ERROR] <Helper> <adiosSystem> <ExceptionToError> : adios2_variable_ndims: [Tue May 10 17:40:37 2022] [ADIOS2 EXCEPTION] <Helper> <adiosType> <CheckForNullPtr> : found null pointer for const adios2_variable, in call to adios2_variable_ndims "

Desktop:

• OS/Platform: Ubuntu 20.04.4 LTS
• Build gcc (Ubuntu 9.4.0-1ubuntu1~20.04.1) 9.4.0

Thank you for reading, and please let me know if i can give any more information.

[eisenhauer]

We will indeed need a bit more to go on. Which engine? When you read it back, what does the code look like? Which version of ADIOS? Maybe just attaching your reader code might help.

[shreybh1]

Thank you for your reply. This is the code sample for my read file. The engine i am using is BP5 to read and write the data. The ADIOS2 version is 2.7.1.978. This code sample works well for BP4, but doesnt work for BP5. The only thing i change is when i specify the IO_ENGINE variable from "BP4" to "BP5". Thank you.

adios2_adios *adios = adios2_init_serial(); 
adios2_io *io = adios2_declare_io(adios, IO_ENGINE); //IO handler declaration
adios2_engine *engine = adios2_open(io, FILENAME, adios2_mode_read);
adios2_variable *var_iodata = adios2_inquire_variable(io, "iodata"); 

/*
* 1st set of tests, to check if the shape and dimensions of variable matches those of iodata. 
*/ 
adios2_variable_ndims(&ndims, var_iodata);
if (ndims != NDIM)
{
    printf("Number of dimensions is not matching. Data verification failed \n"); 
    exit(1); 
}
adios2_variable_shape(shape, var_iodata); 
for (i = 0; i < NDIM; i++)
{
    if (shape[i] != arraygsize[i])
    {
        printf("Global shape of array is not matching. Data verification failed \n"); 
        exit(1); 
    }
}

/*
* start step to initiate data stream 
* adios2_get function to insert contents of var_iodata (adios2 variable) into iodata_read
* end step to stop data stream
*/ 
adios2_step_status status = 0; 
adios2_begin_step(engine, adios2_step_mode_read, times, &status); 
adios2_get(engine, var_iodata , iodata_read, adios2_mode_sync);
ierr = adios2_end_step(engine);

ierr = adios2_close(engine);
engine = NULL; 
adios2_finalize(adios); 

[eisenhauer]

In general, if you use BeginStep/EndStep, you should do Inquire Variable inside the begin/end pair. The above will work with some old file engines, but it won't work with any of the streaming engines or with the BP5 file engine.

(This is due to an evolution in how we handle metadata. The old file engines generally read all the metadata for the file (all information about variables for every timestep), at open(). This was bad because it wasn't possible for streaming, and even for files it made open() horrifically expensive if you had a lot of timesteps and variables. Beginning in ADIOS 2.8, we're trying to separate "random access" mode, where you get all the metadata at open() and don't use begin_step(), and "streaming" mode, where you read metadata only per timestep and do use begin/end step(). The latter works for both file and streaming engines like SST, SSC, Dataman, etc.)

[pnorbert]

Please also note that adios2_io *adios2_declare_io(adios2_adios *adios, const char *name) does not specify the engine of your choice. The second parameter is a string to assign a name to the IO object, which can be used in the XML/YAML config file to describe the parameters including the choice of engine. So you have been using the default engine all along (which is BP4 right now).

You need to use adios2_error adios2_set_engine(adios2_io *io, const char *engine_type); function to set the engine after adios2_declare_io and before adios2_open().

[shreybh1]

Hi, Apologies i didnt give the full description of my code. Indeed that is true, i run my program with a config file in which I have defined the BP5 IO engine. At first i wasn't sure if the engine was correctly working so i did a diff across the output files and they are different and the BP5 file has the unique md.0 file in it. I took this as proof that my program was calling the BP5 engine but please let me know if i have missed something. Thank you.

[shreybh1]

If possible i would also like to know more about the unique diferences between BP5 and BP4 IO engines as I am investigating the performance of these layers in various HPC machines in the UK. I have got some results from the Darshan IO tool, but i was wondering if theres more documentation that i could perhaps read and explain some of the results i have obtained. Apologies for the long messages, and thank you for taking the time to answer my questions.

[pnorbert]

It would be great if we could chat about your results if you could set up a meeting with me. Thanks. Norbert