SpecFlowOSS
Add Performance Comparison to PR checks
Motivation:
To be better to be able to see the performance impacts of Pull Requests to SpecFlow, we would like to have additional checks in our Pull Request checks.
Idea: There will be a dedicated benchmark project which will be run once with the current released SpecFlow version and once with the freshly build SpecFlow version from the Pull request.
A first prototype exists at https://github.com/SpecFlowOSS/SpecFlow/tree/AddBenchmarksToCI/Tests/Benchmark
It is using Benchmark.NET for running the benchmark. We could probably use one of the reporters of Benchmark.NET and then compare the results.
Benchmark Project: No standard unit test runner is used. The methods of SpecFlow are called manually.
It should have some size, that stuff can be measured. Idea is to have the following in the project:
- 10 Features
- 8 Scenarios each Feature
- 2 Outlines with 10 Examples in each feature file
- 20 Binding classes
- 3-5 bindings per class
- 1 separate Hook class + each hook once
- 20 Step Argument Transformation in 4 classes
- Tags - one per feature, 5 scenarios have 2 per feature file
Integration into Pull Request checks: We will add it to our Release pipeline (which is not public). To be able to do it, the easiest thing would be to have a powershell script that we include then in the pipeline.
I like the idea very much!
One thing to add to the project => Scoped bindings would also be a nice thing to track.
Also the prototype will give you a general overall feeling on what the performance will be. But it does not give you any detail on what features / parts are how performant. What I mean by that, if for example a PR changes the performance around scoped bindings, this test will probably just very slightly improved, but for a user using mostly scoped bindings, the difference will be potentially much higher. So if possible, it would be nice to additionally have more fine granular tests that focus on certain features.
Maybe of some use for reference: https://github.com/PowerShell/PowerShell/pull/15242
Sure we can add additional specialized benchmarks, like with a lot of scoping of the bindings. We thought only about how we can start with it and then improve on that.