Unable to capture MPI Functions when instrumenting fortran code

[deanchester]

I have instrumented a fortran code using Caliper but when I run the application it doesn't capture MPI communication.

I have built Caliper with the following configuration:

cmake -DCMAKE_INSTALL_PREFIX=$HOME/local/caliper-gcc -DCMAKE_C_COMPILER=/csc/tinis/software/Core/GCCcore/7.3.0/bin/gcc -DCMAKE_CXX_COMPILER=/csc/tinis/software/Core/GCCcore/7.3.0/bin/g++ -DWITH_FORTRAN=On -DWITH_TOOLS=On -DWITH_MPI=On -DMPI_C_COMPILER=/csc/tinis/software/Compiler/GCC/7.3.0-2.30/OpenMPI/3.1.1/bin/mpicc -DCMAKE_Fortran_COMPILER=/csc/tinis/software/Core/GCCcore/7.3.0/bin/gfortran ..

When I run my application I set the following in my script:

export CALI_SERVICES_ENABLE=trace,event,mpi,timestamp,recorder
export CALI_TIMER_SNAPSHOT_DURATION=true
export CALI_TIMER_INCLUSIVE_DURATION=true
export CALI_MPI_WHITELIST=all
export CALI_RECORDER_FILENAME="./caliper-$SLURM_JOB_ID/caliper-%mpi.rank%.cali"

In the caliper output files for the code I only have the instrumented areas of the code with the start and end routines:

  call cali_begin_byname('sweep')
  c CODE... 
  call cali_end_byname('sweep')

Any ideas whats going wrong?

[daboehme]

Hi @deanchester ,

By default Caliper relies on library constructors to initialize its MPI component, but sometimes that fails. In that case, you can explicitly initialize it with the cali_mpi_init() function. I've added a Fortran binding for that function in the latest commit to master (#232). With that, adding call cali_mpi_init() somewhere at program start should help. You can set CALI_LOG_VERBOSITY=1 and see if the mpi service gets initialized.

[andrewreisner]

I am having an issue with this as well. Using cali_mpi_init() results in undefined reference to cali_mpi_init_. If I remove the preprocessor guard from https://github.com/LLNL/Caliper/blob/master/src/interface/c_fortran/wrapfcaliper.F#L457 everything works as expected and I get caliper output. Is CALIPER_HAVE_MPI from caliper-config.h supposed to propagate to this wrapper?

[daboehme]

Hi @andrewreisner ,

The CALIPER_HAVE_MPI flag is supposed to propagate to the wrapper, so that might be a bug. I'll take a look. Calling cali_mpi_init() is no longer necessary in newer Caliper versions (as of v2.5.0 at least) though, so you can safely remove it. Capturing MPI functions in Fortran codes is unrelated to that: we simply don't have the wrapper functions for the Fortran MPI functions in Caliper, which are different from the C MPI functions. I hope I can add them at some point, but it's a significant effort.

[andrewreisner]

Thanks for the information. I updated my Caliper version and removed cali_mpi_init() and everything works as expected.