inference_policies
inference_policies copied to clipboard
mlcommons
Issues related to MLPerf™ Inference policies, including rules and suggested changes
According to the current rules, a submitter may intentionally or unintentionally introduce sparsity in the results table. For example, if they choose a different system name for each workload they...
Hi, I see many criterions for accuracy, for example the accuracy must be within 99% of the FP-32 accuracy. Is there a similar idea for perplexity scores in language modelling?
Neither the `/inference` or the `/inference_policies` repo holds the submission UI usage documentation that Pablo had created and added into the private 2.0 submission repo. Could we pull that markdown...
Hi, the rules show that min duration is 600 for all workloads (I was looking at datacenter) while it shall be 60 for most of them. https://github.com/mlcommons/inference_policies/blob/master/inference_rules.adoc#3-scenarios e.g,. looking at...
This may not be suitable for discussion here. In MLPerf HPC v0.7, the summary from each institute is uploaded as submitter_summaries.adoc in https://github.com/mlcommons/hpc_results_v0.7. In inference v1.0, The same kind of...
In the [available systems definition](https://github.com/mlcommons/policies/blob/master/submission_rules.adoc#731-available-systems) of inference policies, here is the snippet of the definition "_Available systems must use an Available software stack. A software stack consists of the set...
The rationale for start_from_device is that submissions should not need to incur the overhead of transfer from system DRAM if there is a mechanism whereby network inputs can be delivered...
