[Question] Automate BDN reporting in Continuous Integration workflow

[bouchraRekhadda]

Hello,

We have recently started exploring BenchmarkDotNet and we would like to set it up in our CI workflow (JetBrains TeamCity). We are mainly interested in CPU and memory consumption of certain code areas for a start in the performance testing journey; the idea is to report any regressions (yet to be precisely defined) found upon a commit push (or a range of commits), something like what it was reported by @adamsitnik at https://github.com/dotnet/runtime/issues/59149 could be a great start for us. Do you mind sharing with us how these issues are created?

One last question, is the CI integration still in your roadmap (cf. this comment)?

Thank you in advance.

PS: Unfortunately we cannot use Continuous Benchmark (.NET) since we use TeamCity as CI pipeline.

[adamsitnik]

I would divide it into three problems:

• Ensure that the benchmarks are always going to be executed using the same hardware, so the results are comparable. For example, in the .NET Team we have a dedicated hardware lab where all PCs have the same spec and config.
• Store the results in a database. You can use some of the existing exporters like Elastic.CommonSchema.BenchmarkDotNetExporter or implement your own. If I was writing it from scratch, I would use some Time Series database.
• Use statistical methods to identify the regressions. This is the hard part ;) The bot that .NET Team is using is written and maintained by @DrewScoggins. AFAIK there are no plans to open source it in the near future. I know that @AndreyAkinshin who works at JetBrains was experimenting with a TeamCity plugin for doing that. @AndreyAkinshin can you share a little bit more?

If you can't find any other math tool for detecting the regressions you can always use https://github.com/dotnet/performance/tree/main/src/tools/ResultsComparer which is a simple tool and work for simple regressions.

[bouchraRekhadda]

Thank you @adamsitnik for your detailed answer; we are indeed considering using the ResultComparer for now. @AndreyAkinshin would you be able to provide insights on what Adam has mentioned above regarding TeamCity plugin for regressions identification? Thank you in advance

[AndreyAkinshin]

Currently, we have a standalone service that just fetches data from TeamCity and processes it independently. Sorry, there is no a TeamCity plugin for now.

[bouchraRekhadda]

Thank you @AndreyAkinshin for your feedback. Is there any hope to maybe hear about the plugin in the near future ?

[AndreyAkinshin]

Definitely not in the near future, sorry. Our current approach is highly customized to our own needs, and it's too complicated to make a universal solution based on it.

[adamsitnik]

I believe that all the questions have been answered. Please follow #54 for updates. But to be honest it's a very big feature and without external funding or major contribution I don't see it happening.