mobile_app_open icon indicating copy to clipboard operation
mobile_app_open copied to clipboard

Published 20 hours ago • verified publisher icon mlcommons

Does performance test need to test accuracy?

Open d-uzlov opened this issue 3 years ago • 5 comments

Currently we have the following comment on the ProcessOutput method: https://github.com/mlcommons/mobile_app_open/blob/38327317a45fd0593fb92240b14c4c0e99a46292/flutter/cpp/dataset.h#L56-L61

However, this is incorrect, because ProcessOutput is called unconditionally: https://github.com/mlcommons/mobile_app_open/blob/38327317a45fd0593fb92240b14c4c0e99a46292/flutter/cpp/mlperf_driver.cc#L55-L56 https://github.com/mlcommons/mobile_app_open/blob/38327317a45fd0593fb92240b14c4c0e99a46292/flutter/cpp/mlperf_driver.cc#L72-L74

We should either fix documentation or add a condition to avoid calling this in performance mode.

d-uzlov avatar Oct 31 '22 10:10 d-uzlov

What was the conclusion, is ProcessOutput counted in loadgen latency? e.g. topK calculation for Imagenet?

Mostelk avatar Aug 16 '23 06:08 Mostelk

What was the conclusion, is ProcessOutput counted in loadgen latency? e.g. topK calculation for Imagenet?

It's counted. @anhappdev fixed the doc.

freedomtan avatar Aug 16 '23 06:08 freedomtan

The doc still reads like this , so is ProcessOutput only called in accuracy mode? / ProcessOutput processes the output data before sending to mlperf. This // function only get called on Accuracy mode so we don't need to care about // its performance. The argument sample_idx should only be used to help the // Dataset calculating the accuracy; it should not affect the returned // result by any means.

Mostelk avatar Aug 16 '23 21:08 Mostelk

Right now looks Imagenet topK calculation is done in ProcessOutput https://github.com/mlcommons/mobile_app_open/blob/8d92e1bd8a379f188c0db33fc22052be18a15eff/flutter/cpp/datasets/imagenet.cc#L116

Mostelk avatar Aug 16 '23 23:08 Mostelk

The doc still reads like this , so is ProcessOutput only called in accuracy mode? / ProcessOutput processes the output data before sending to mlperf. This // function only get called on Accuracy mode so we don't need to care about // its performance. The argument sample_idx should only be used to help the // Dataset calculating the accuracy; it should not affect the returned // result by any means.

The line: This function only get called on Accuracy mode so we don't need to care about its performance. is already removed from the comment.

https://github.com/mlcommons/mobile_app_open/blob/8d92e1bd8a379f188c0db33fc22052be18a15eff/flutter/cpp/dataset.h#L56-L61

anhappdev avatar Aug 17 '23 05:08 anhappdev