logging
logging copied to clipboard
mlcommons
Document RCPs method of production
It would be useful if each RCP point also notes the method of production that was used to generate it, in the RCP jsons. Intention would be to have a path of reproducibility if needed or record for future reference- especially when at submission time we will have few PRs for different submitters with new RCPs, probably run on different HW platforms & different framework versions.
Method of production can be noted via additional fields per RCP point such as:
- "hw_notes": for example tpuv3-128 or DGXA100 etc.
- "framework": for example TF2.1.1, pytorch etc
- "reference git commit hash": To note the reference commit that was used to generate the RCP
- "notes": For optional details, for example if a rcp is from a submission code repo vs the reference repo for temporary reasons
This would avoid us all scratching our heads post v1.0 about different RCP points.
Sure, we could add more fields in the json that document when the RCP was generated.
Not everything is easy to obtain (reliably) for 1.0 -- the TF vs pytorch is easy but the git commit hash is not for example. Due to other things to fix, I am not planning to add it for 1.0. I can add some info, but reviewing will take time so the fields will not be 100% reliable.
What you can do is refer to this spreadsheet for RCPs: https://docs.google.com/spreadsheets/d/1HZSh3THnWEldiZYUiqYSyWitB5JfYfDL_RORDcB3rAw/edit#gid=0 Most of them are from NVDA, so John can point you to the right source.
Also RCPs for resnet: https://docs.google.com/spreadsheets/d/1CDCSizkWfJKIR0pS4E1k36gNuWVVEORCfIMxM2U-PDQ/edit?resourcekey=0-nOaUKM7iS1jvI0CJmPtMSw#gid=1086699069
And RCPs for Bert: https://docs.google.com/spreadsheets/d/1vD3UkqJr20dB1jOoZGuLmIbf5iFgsk1aObTMC2QhvSY/edit#gid=252780832