Submission Events and Timeline

[bitfort]

We need to enumerate when things will happen for the submission and review.

[bitfort]

Week0 - June 22nd Monday June 22 - Receive (over email) information for github uploads, reach out if you do not. Thursday June 25 - Notify SWG Chairs if you will not submit. Friday June 26 - 1PM PDT must have completed the submission. Friday June 26 - 1:30PM PDT Submitters receive email from SWG with our understand of results.

Week 0.5 - June 29th Holiday Week

Week 1 - July 6 Monday - Begin drafting neutral press release Tuesday - (4:30PM-5:30PM PDT) Review committee meeting, discuss objections Thursday - (8:30AM-9:30AM PDT) Review committee meeting Friday - 5PM PDT Objections due in github

Week 2 - July 13 Monday - 5PM PDT response to objections due: see document Tuesday - (4:30PM-5:30PM PDT) Review committee meeting, fix easy to resolve issues and discuss more difficult Wednesday - 5PM PDT written statements on disputed issues not resolved Tuesday Thursday - (8:30AM-9:30AM PDT) Review committee meeting, voting to resolve issues if necessary.

Week 3 - July 20 Tuesday - (4:30PM-5:30PM PDT) Review committee meeting Wednesday - 5PM PDT Final code due Thursday - (8:30AM-9:30AM PDT) Review committee meeting, final voting decisions taking into account code fixes, approval of press release. Friday - 1PM PDT Final logs due, 1:30PM final results distributed by SWG

Week 4 - July 27 Monday - Press and analyst pre-briefings allowed under embargo, all briefings to include neutral press release; 1PM PDT draft of results page available for comment Tuesday - 1PM PDT corrects to result page due, 5PM mock of results available Wednesday - 10AM PDT Results and PR public, press embargo ends

[bitfort]

NOTICE TO ALL SUBMITTERS:

DO NOT DISCUSS RESULTS WITH PUBLIC UNTIL JULY 29th 10AM PDT. All results are private until July 29th 10AM PDT.

[christ1ne]

@bitfort @petermattson the line about results withdraw is missing. should be the Friday of week 3.

[jonathan-cohen-nvidia]

I think there is a principle: once the submission deadline is passed, no one should be surprised by anyone's resubmissions.

Resubmissions can happen for 2 reasons:

1. To address a compliance issue, apply a fix, then rerun
2. To steal hyper parameters

All such resubmissions should be declared in advance so everyone knows what to expect.

So there should be a deadline for declaring intent to resubmit a result with stolen HPs. I would propose Week 3 Monday (July 20).

[petermattson]

SWG: Approve week 3 Monday declaration of borrowing w/ specific hps. Make withdraw deadline more obvious.

[jonathan-cohen-nvidia]

Just want to add that today we identified a valid 3rd reason for resubmit:

3. In case the rules were slightly wrong/ambiguous, there are special cases where we will allow resubmission. I think these are (exhaustive list):

• RN50 label smoothing - if submitted with 0, allowed to resubmit with 0.1
• RN50 can switch from norm to trunc norm or vice versa
• Transformer eval using the wrong dataset, can resubmit with the corrected dataset (which should improve score)