algorithmic-efficiency
algorithmic-efficiency copied to clipboard
mlcommons
MLCommons Algorithmic Efficiency is a benchmark and competition measuring neural network training speedups due to algorithmic improvements in both training algorithms and models.
## Description Add CI test that checks if reported times and number of evals are inline with expectations. For each submission we report: - total wall clock time - submission...
According to the rules, it is expected that the model parameters can be reset by calling `workload.init_model_fn()`. However, creating a new model instance, like it is currently done in the...
We should demonstrate the capabilities that our API gives users, to give them examples of what they can explore when developing submissions. We should make some example submissions, probably in...
We likely can delete a lot of our code to use https://docs.scipy.org/doc/scipy/reference/stats.qmc.html
Currently `update_batch_norm` just runs the librispeech workloads in train mode, which also runs dropout in train mode. The purpose of having separate `mode` and `update_batch_norm` kwargs to `model_fn()` was so...
Based on the [CALL_FOR_SUBMISSIONS](https://github.com/mlcommons/algorithmic-efficiency/blob/main/CALL_FOR_SUBMISSIONS.md#dates), results were to be announced on July 15th. Is there an updated release date?
Updated the CONTRIBUTING readme to include detailed steps for installing and configuring the Google Cloud Ops Agent. This will help users set up monitoring for their VM logs.
## Control over batch-norm running_mean/var buffers Following up on the request in the recent working group meeting regarding future improvements to the challenge, it would be extremely useful if we...
In `librispeech_conformer` the `model_fn` returns `logits_batch` as a Tuple of tensors, not a tensor. The return type is hence wrong: https://github.com/mlcommons/algorithmic-efficiency/blob/ddf5efc4e13a9a4e620ad719e9bf42303f064fac/algorithmic_efficiency/workloads/librispeech_conformer/librispeech_pytorch/workload.py#L119 It should be: ``` def model_fn(...) -> Tuple[Tuple[spec.Tensor, spec.Tensor],...
## Description This pull request introduces a `prepare_for_eval` function and updates the code to support it. The implementation follows the blueprint of @fsschneider in https://github.com/mlcommons/algorithmic-efficiency/issues/719#issuecomment-2328797610 and fixes the bug of...