policies icon indicating copy to clipboard operation
policies copied to clipboard
verified publisher icon mlcommons

Open/Preview submissions from Inference v2.1

Open psyhtest opened this issue 3 years ago • 3 comments

In the MLPerf Inference v2.1 round, Qualcomm and their partners submitted a number of RetinaNet results, which the Review Committee eventually accepted to the Open division under the Preview category. Also, ResNet50 and BERT results from the same systems were accepted to the Closed division under the Available category.

Given the extenuating circumstances surrounding these RetinaNet submissions, I believe it would be reasonable NOT to require ALL of such submissions to be replayed in the v3.0 round under the Available category, for example, if a system is no longer available for experimentation.

psyhtest avatar Dec 20 '22 17:12 psyhtest

Discussed two options:

  1. Submit in v3.0 and withdraw during review period so that preview results stay in v2.1 table.
  2. Not submitted in v3.0, and mark v2.1 preview results as retracted/invalid (exact action TBD based on the discussion from chairs meeting)

Option 2 was chosen

rnaidu02 avatar Jan 03 '23 17:01 rnaidu02

David is going to propose rule changes for Preview->Available

rnaidu02 avatar Jan 10 '23 17:01 rnaidu02

This issue has been open for 1.5 years. Let's close it?

psyhtest avatar May 14 '24 09:05 psyhtest