eval-dev-quality
eval-dev-quality copied to clipboard
Extend the evalatuion.csv with the current maximum values, and keep them updated in the repositories
Please help us keep the number of duplicated issues small. We kindly ask you to add your input to the appropriate issue or PR in case your feature idea is already being tracked.
- [x] I have searched the existing feature requests, latest roadmap and open pull requests.
Problem
Maximum metrics can change over time for a language, repository, task, case, ... to freeze the maximums, we want to save them in the evaluation.csv
Solution (optional)
Tasks:
- [ ] Read maximums from the repository files
- [ ] Insert
- [ ] add files-executed-maximum
- [ ] add coverage-maximum
- [ ] add tests-passing-maximum
- [ ] add response-no-error-maximum
- [ ] add response-with-code-maximum
- [ ] add response-no-excess-maximum
- [ ] Update maximums if the value gets bigger in the repository configuration