Anton Lokhmotov
Anton Lokhmotov
Thank you @MatthewARM. For the official [MLPerf Inference v2.0](https://mlcommons.org/en/inference-edge-20/) submission, we measured 339 ms for Odroid N2+ and 349 ms for Raspberry Pi 4. Seems like a RPi4 regression to...
@ckstanton Does this look like a manifestation of the same bug to you? ``` ================================================ MLPerf Results Summary ================================================ SUT name : ArmNN_TFLite_SUT Scenario : SingleStream Mode : PerformanceOnly 90th...
A question to the WG. We have already collected **lots** of results like above, which would be rather onerous to rerun before the submission deadline. I'm wondering if we can...
@nv-jinhosuh That's my feeling too. But IIRC we decided that the early estimate would be used as the metric regardless. And it is what the submission checker appears to do....
@tjablin Updated as requested.
Taken from [the latest meeting slides](https://docs.google.com/presentation/d/1x7goS8XnuwlS2PVlsoE93KvbQQA-GhDjEEYGORQ5pfk/edit#slide=id.g12733238521_0_1) ``` Accuracy target FP32: 0.375 mAP Latency target WG Approves:100 msec ``` [Copy of 2022.08.02 MLCommons Inference WG Meeting - RetinaNet.pdf](https://github.com/mlcommons/inference/files/9259454/Copy.of.2022.08.02.MLCommons.Inference.WG.Meeting.-.RetinaNet.pdf)
Sorry, I missed [PR1168](https://github.com/mlcommons/inference/pull/1168) (submission checker change) 5 weeks ago. But [PR](https://github.com/mlcommons/inference_policies/pull/251) was only two weeks ago. According to [the attached slide](https://github.com/mlcommons/inference/files/9261585/Copy.of.2022.07.05.MLCommons.Inference.WG.Meeting.-.v2.1.timeline.pdf) from [the WG meeting slide deck](https://docs.google.com/presentation/d/1NYiCaWGFUc45gKZvAbH-Jooqw37lR_d71W9RDbKqc-E/edit#slide=id.gbb938d29b3_3_0) 4 weeks...
Where do we draw a line between cases like this and (non-WG approved) late changes like https://github.com/mlcommons/inference/pull/1206?
Thank you Arjun. Your accuracy is still higher than 37.55%.
> [Inference Policies](https://github.com/mlcommons/inference_policies/blob/master/inference_rules.adoc#minimum-requirements-of-a-datacenter-system) say the accuracy requirement as 99% of FP32 model, but does not clearly say which exact model (lowest accuracy one?) nor say up to how many digits...