training
training copied to clipboard
mlcommons
v0.6 tag missing
The Git repository comes with a tag for version 0.5 but not 0.6. It would be great to add the respective tag. Especially since the MiniGo benchmark is currently broken one could use this tag to go back to the version officially used in the last competition.
Where do you need the tag to be created? Where are the 0.5 version tags?
At https://github.com/mlcommons/training/ you see at the top, that there is only one tag. This should be largely extended to previous submission rounds.
We're in the process of cleaning up the repo. I fully agree with the sentiment that this repo should have tags corresponding to the repo states at each submission deadline. And that moving forward those tags should be in place on the repo at the reference freeze date. (So when we freeze the new Large Language Model and Recommender models in mid-January 2023 we should tag the repo "v3.0").
That said, the repo is (and has for a long time) been a mess, so I don't think there is any tagging scheme that will actually get us to sets of references that actually consistently work. (I don't think there's any version of the minigo reference that is likely to work since we've never generated RCPs for minigo. I don't think anyone has run the minigo reference for years.)
For past rounds I think the best we can currently do is go back and search for the commits closest to each submission deadline, and tag those with v0.6, v0.7, v1.0, v1.1, v2.0.
Paging through the logs and doing some diffs my best guess for the tagging would be:
Commit e8237dc2, May 10, 2019: tag "v0.6" Commit 373dfeda, Jun 21, 2020: tag "v0.7" Commit c5495718, Jun 27, 2021: tag "v1.0" Commit d0a86d67, Aug 13, 2021: tag "v1.1" Commit 8959736b, Jun 15, 2022: tag "v2.0" Commit e6f45a4db, Nov 3, 2022: tag "v2.1"
I just closed https://github.com/mlcommons/training/issues/406 as a duplicate of this. Hopefully when we do the tagging above it will (sort of) address issue #406 . (Although as I said above: won't really address 406, since the BERT reference at time of v0.7 submission (and for quite some time afterwards) was broken.)
