Shriya Rishab

Clarify rules regarding the review committee

10

The current rules are not clear about review chair selection process and the review committee composition. So this PR aims to clarify those reviews.

Formalize rule regarding benchmark lifecycle

1

Address comment https://github.com/mlcommons/policies/pull/171#issuecomment-2102731288

Why are we using olympic scores stdev for RCP interpolation?

3

The general idea of stdev is to get the variance of a population, but Olympic scoring drops the extreme values. So we should consider using stdev of the entire population...

Package checker and the compliance checker expect submission_status but have different allowed values

1

Package checker looks at sysjson to verify that [submission_status is one of [ "Available on-premise", "Available cloud", "Research, Development, or Internal (RDI)", "Preview"]](https://github.com/mlcommons/logging/blob/master/mlperf_logging/system_desc_checker/system_desc_checker.py#L99-L104) Compliance checker looks at individual logs to...

Add rule to require logging of seeds from v4.1 onwards

1

Addresses https://github.com/mlcommons/training_policies/issues/546 Based on https://github.com/mlcommons/training_policies/pull/444 Corresponding change to the logging library https://github.com/mlcommons/logging/pull/384

Currently, the seed checker within the package checker ensures that logged seeds are compliant. But it only raises a warning if seeds are not logged as [stated here](https://github.com/mlcommons/logging/blob/master/mlperf_logging/package_checker/seed_checker.py#L59-L65). But we...

# What does this PR do ? Enable async ckpt, ckpt every 15mins and reduce preemption time from 5mins to 1min. Disable async ckpt save for SSMs as it is...