algorithmic-efficiency icon indicating copy to clipboard operation
algorithmic-efficiency copied to clipboard

Add tests for expected number of evals submission and measured times

Open priyakasimbeg opened this issue 2 years ago • 0 comments

Description

Add CI test that checks if reported times and number of evals are inline with expectations.

For each submission we report:

  • total wall clock time
  • submission time
  • eval time
  • logging time
  • number of evals

We should add the following tests on at least 1 workload (probably mnist).

  1. We should check that submission_time + eval_time + logging time ~ total _wallclock_time.
  2. We should check that the number of evals is roughly equal to total_submission_time // eval_period

priyakasimbeg avatar Jul 25 '23 23:07 priyakasimbeg