kokkos-tools
kokkos-tools copied to clipboard
Profiling requires MPI be initialized
This might be a corner case. I have some tests in Trilinos that are single node only. They do not need MPI and, therefore, don't call MPI_Init in main (in cmake we restrict to one MPI process). However, if Trilinos is configured/built with MPI enabled, then it seems the stacked timer profiler requires all executables to initialize mpi. This causes a bunch of unit test failures in phalanx. An example is shown below. It would be convenient for the profiler could check if mpi was initialized and perform the init if needed. That way I could leave the profiling flag on during unit testing. When I unset the KOKKOS_PROFILE_LIBRARY flag, the error goes away and all the tests run fine.
[rppawlo@gge Kokkos]$ mpirun -np 1 ./Phalanx_tKokkosNestedLambda.exe
Kokkos::OpenMP::initialize WARNING: OMP_PROC_BIND environment variable not set
In general, for best performance with OpenMP 4.0 or better set OMP_PROC_BIND=spread and OMP_PLACES=threads
For best performance with OpenMP 3.1 set OMP_PROC_BIND=true
For unit testing set OMP_PROC_BIND=false
*** The MPI_Comm_rank() function was called before MPI_INIT was invoked.
*** This is disallowed by the MPI standard.
*** Your MPI job will now abort.
[gge.srn.sandia.gov:23656] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
-------------------------------------------------------
Primary job terminated normally, but 1 process returned
a non-zero exit code.. Per user-direction, the job has been aborted.
-------------------------------------------------------
--------------------------------------------------------------------------
mpirun detected that one or more processes exited with non-zero status, thus causing
the job to be terminated. The first process to do so was:
Process name: [[60414,1],0]
Exit code: 1
--------------------------------------------------------------------------