astro-sdk icon indicating copy to clipboard operation
astro-sdk copied to clipboard

[BENCHMARK] Create threshold based checks

Open utkarsharma2 opened this issue 3 years ago • 0 comments

Please describe the feature you'd like to see Benchmarks are useful when we have to compare them with old releases and make sure they have not gone down/degraded.

Describe the solution you'd like

  1. Get old gitsha from old version. This should be ideally be mapped to tag.
  2. Use the gitsha to get old results from bq.
  3. Modify the config to add a threshold in the [datasets/database]
  4. Modify the analyse.py script to have a parameter fail if the thresholds are not met.
  5. Use analyse.py as the last step in CI benchmark job

utkarsharma2 avatar Oct 07 '22 15:10 utkarsharma2