Azure
test: detect regressions with build performance program
What type of PR is this?
/kind test
What this PR does / why we need it:
This PR does the following:
- Ingests local build performance data into kusto for later analysis
- Queries aggregate performance data for the relevant SKU in kusto
- Kusto will contain mean and stdev for every section in every script for every SKU
- Compares queried data against performance data recorded for the local build
- Detects regressions and prints them to logs, notifying user with a warning
- Regression is currently defined as any time over 3 Stdev over the mean, but that can change if team would like a different metric
- This is to prevent the logs from being too noisy and only showing regressions for sections that are significantly out of spec
Which issue(s) this PR fixes:
This PR will prevent VHD Build time regression by notifying software engineers of their code changes effect on VHD build times. Additionally, it can illuminate problem areas in the build when retry logic is causing the build to take longer than expected.
Requirements:
- [ x] uses conventional commit messages
- [ ] includes documentation
- [ ] adds unit tests
- [ ] tested upgrade from previous version
Reviewer Notes:
Open to any suggestions related to testing and error handling specifically, as I would like this program to be as robust as possible. Currently, I am wrapping all errors and sending them back to main.
Kusto functions are not currently tested because they are just wrappers around the go SDK functions to keep main clean and readable.
Once code is approved, I will create a new cluster and change a couple variables as I am currently only using a dev/test kusto instance. After that I will merge the code.
Pull Request Test Coverage Report for Build 11297674229
Details
- 0 of 0 changed or added relevant lines in 0 files are covered.
- No unchanged relevant lines lost coverage.
- Overall coverage remained the same at 70.165%
| Totals | |
|---|---|
| Change from base Build 11296795694: | 0.0% |
| Covered Lines: | 2808 |
| Relevant Lines: | 4002 |
💛 - Coveralls
@zachary-bailey If you are still working on this, could we move Kusto interface code/pkg outside the build-performance path? I think I'll be using your code for cve work as well. Not a blocker though, I can do this on my end later as well.
