MPI_File calls not logged when called through HDF5

[roblatham00]

Consider this simple test case:

#include <stdio.h>
#include <mpi.h>
#include <netcdf.h>
#include <netcdf_par.h>

int main(int argc, char* argv[])
{
    int my_rank;
    MPI_Info info;
    MPI_Comm newcomm;
    int ncid;

    MPI_Init(&argc, &argv);

    MPI_Comm_rank(MPI_COMM_WORLD, &my_rank);

    MPI_Info_create(&info);
    MPI_Info_set(info, "romio_no_indep_rw", "true");

    nc_create_par("test.nc", NC_CLOBBER | NC_MPIIO | NC_NETCDF4, MPI_COMM_WORLD, info, &ncid);

    printf("Before calling nc_close, rank = %d\n", my_rank);
    fflush(stdout);

    nc_close(ncid);

    printf("After calling nc_close, rank = %d\n", my_rank);
    fflush(stdout);

    MPI_Finalize();

    return 0;
}

it uses NetCDF-4, which in turn makes HDF5 calls, which in turn calls MPI-IO routines.

However tracing with pilgrim only shows me top-level MPI calls made from the test case:

 % LD_PRELOAD=${HOME}//work/soft/pilgrim/lib/libpilgrim.so mpiexec -np 2 ./hdf-2434
 % ~/work/soft/pilgrim/bin/pilgrim2text pilgrim-logs
 % cat pilgrim-logs/_text/1.txt
 MPI_Comm_rank()
 MPI_Info_create()
 MPI_Info_set()

Weird, right?

I build HDF5 and NetCDF-4 with spack... should I have used a special compiler or linker flag to make sure Pilgrim would see the MPI-IO calls?

[wangvsa]

Are you using the latest Pilgrim code? There's no compiler/linker flag to enable/disable MPI-IO tracing. Pilgrim intercepts everything by default. But in some older versions I probably commented out MPI-IO tracing for debugging purpose.

I don't have a NetCDF installed, but I just tried FLASH (it uses HDF5 for I/O) and it works fine.

[wangvsa]

By the way, if you want to see I/O traces, I would recommend my other tracing tool, Recorder :) https://github.com/uiuc-hpc/Recorder

Recorder intercepts both HDF5, MPI-IO, and POSIX calls, and has some post-processing/visualization support.