[Ember]: How to enable MPI event statistics for the motifs?

[saichennaintel]

Hello,

I'm trying to dump the MPI event time statistics for the ember motifs. I see that there are 21 statistics for motifs, one for each MPI event as shown below.

        STATISTIC 0 = time-Init [Time spent in Init event] (ns) Enable level = 0
        STATISTIC 1 = time-Finalize [Time spent in Finalize event] (ns) Enable level = 0
        STATISTIC 2 = time-Rank [Time spent in Rank event] (ns) Enable level = 0
        STATISTIC 3 = time-Size [Time spent in Size event] (ns) Enable level = 0
        STATISTIC 4 = time-Send [Time spent in Recv event] (ns) Enable level = 0
        STATISTIC 5 = time-Recv [Time spent in Recv event] (ns) Enable level = 0
        STATISTIC 6 = time-Irecv [Time spent in Irecv event] (ns) Enable level = 0
        STATISTIC 7 = time-Isend [Time spent in Isend event] (ns) Enable level = 0
        STATISTIC 8 = time-Wait [Time spent in Wait event] (ns) Enable level = 0
        STATISTIC 9 = time-Waitall [Time spent in Waitall event] (ns) Enable level = 0
        STATISTIC 10 = time-Waitany [Time spent in Waitany event] (ns) Enable level = 0
        STATISTIC 11 = time-Compute [Time spent in Compute event] (ns) Enable level = 0
        STATISTIC 12 = time-Barrier [Time spent in Barrier event] (ns) Enable level = 0
        STATISTIC 13 = time-Alltoallv [Time spent in Alltoallv event] (ns) Enable level = 0
        STATISTIC 14 = time-Alltoall [Time spent in Alltoall event] (ns) Enable level = 0
        STATISTIC 15 = time-Allreduce [Time spent in Allreduce event] (ns) Enable level = 0
        STATISTIC 16 = time-Reduce [Time spent in Reduce event] (ns) Enable level = 0
        STATISTIC 17 = time-Bcast [Time spent in Bcast event] (ns) Enable level = 0
        STATISTIC 18 = time-Gettime [Time spent in Gettime event] (ns) Enable level = 0
        STATISTIC 19 = time-Commsplit [Time spent in Commsplit event] (ns) Enable level = 0
        STATISTIC 20 = time-Commcreate [Time spent in Commcreate event] (ns) Enable level = 0 

However, these statistics are commented out https://github.com/sstsimulator/sst-elements/blob/master/src/sst/elements/ember/libs/emberMpiLib.cc#L40. I uncommented those lines but it failed to build saying that the registerStatistic method definition has not been found. Can you point me which header file to include to resolve this error and/or how to dump these MPI event statistics during the network simulation?

[deanchester]

I tried this a little while ago (2/3 years ago), it doesn't work as expected as the underlying implementation of the functions migrated to a module within Ember (see here) and modules cannot have statistics registered to them so you can't retrieve them from the core.

[nahkoots]

To be clear, this means that it's currently not possible to get timing statistics for specific events from Ember? What was the motivation for moving the implementation to a module within Ember, given that the change broke event time statistics? Is it possible to fix statistics while keeping the implementations where they are? What would have to change within Ember for that to happen?

[researcherben]

My team is trying to use metrics from SST's ember but the metrics are not working. Is there any status update on this issue?

[feldergast]

We don't currently have an active developer for Ember/Firefly, so there are no plans to work on this in the near term.

I think the change to module happened when the library interface was abstracted to allow different libraries to be loaded for different motifs. I looked at the code and nothing stood out as to why the libraries need to be modules, so it may be possible to change them to be loaded as anonymous SubComponents. If someone is interested in pursuing this, the place to start is where it loads the library in EmberEngine::createApiMap() (in emberEngine.cc). This should point you to where other changes in the data structures would have to be made.

[researcherben]

Thanks for explaining.