multi-model-server icon indicating copy to clipboard operation
multi-model-server copied to clipboard

CircleCI + Performance Regression

Open prashantsail opened this issue 5 years ago • 0 comments

Description of changes:

Primary goal of this PR is -

  1. Integration of performance regression suite with CircleCI
    • Add new performance regression job in circleci config
    • Add corresponding shell script for job execution
    • Added new ci_linux_medium environment for comparing against thresholds specific to CircleCI env.
      • Renamed xlarge environment to macos_xlarge (more specific)

Along with the primary goal, we have identified and fixed following -

  1. Movement of API tests(newman) suite from test/ to tests/api/
  2. CircleCI's API tests(newman) job now continues with execution of all 3 collections(management, inference, https), even if any one of them fails.
  3. CircleCI's shell scripts tweaked to exit with error when any of the important commands fail.
  4. CircleCI Jobs now copy the entire MMS's logs directory along with console logs (instead of only copying mms console logs)
  5. Removed - /performance_regression. The directory was part of POC and is no longer required.

All Jobs are working and are successful after these changes - Build #443 Screenshot 2020-07-09 at 15 05 33

Performance regression Job Performance regression logs and results

Note:
The commit count is high because the previous PRs of CircleCI and Performance regression have been squashed & merged.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

prashantsail avatar Jul 02 '20 13:07 prashantsail