A way to get performance loadgen statistics in a program

[d-uzlov]

What is the intended way to get statistics from loadgen?

Currently in the mlperf mobile app we use the ReportLatencyResults function to calculate performance from latency. However, recently this function was removed, so our app is not compatible with the r2.1 release.

Here is the list of data that we currently gather:

• latency percentile values
• benchmark duration
• total number of finished queries
• result validity

As far as I can see, current loadgen API doesn't provide any way to get these values. Am I missing something?

I see that percentile values and result validity are available in the mlperf_log_detail.txt and mlperf_log_summary.txt. Summary logs are likely intended to be human readable but it seems like detailed log file uses JSON format, so it's probably intended to be parsed by other programs?

Questions:

1. Is the format of mlperf_log_detail.txt stable? Can I parse it without worrying that parser will break after updating loadgen to some future version?
2. How can I get total number of finished queries?
3. How can I get benchmark duration without loadgen overhead?