algorithmic-efficiency
algorithmic-efficiency copied to clipboard
Add tests for expected number of evals submission and measured times
Description
Add CI test that checks if reported times and number of evals are inline with expectations.
For each submission we report:
- total wall clock time
- submission time
- eval time
- logging time
- number of evals
We should add the following tests on at least 1 workload (probably mnist).
- We should check that submission_time + eval_time + logging time ~ total _wallclock_time.
- We should check that the number of evals is roughly equal to total_submission_time // eval_period