Show summary tables for group nodes in TestAdapter

[AndreyAkinshin]

At the moment, the TestAdapter shows statistics for independent benchmark runs (leave nodes of the tree), but doesn't show anything for grouping nodes.

It would be nice to automatically show summary tables for the given grouping nodes that define a subset of running benchmarks.

[caaavik-msft]

I have spoken with one of the devs on the VSTest team about this and they have confirmed to me that this feature is not possible to complete with the current VSTest protocol, I've started a discussion with them to see what it might take to get this added to the VSTest protocol but no promises at this time.

[AndreyAkinshin]

@caaavik-msft thanks for the update, much appreciate it.

I guess, we can support it not on the vstest side, but on the IDE side. At least, I can try to do it in Rider/R#. =) It shouldn't be so hard to parse the results of individual benchmarks and recombine them to a summary table for a grouping node.

However, I'm not sure that it's the right way of supporting it. While it's nice to have the out-of-the-box unit test adapter, it would be better to develop a separate Rider/R# plugin that directly calls BenchmarkDotNet without vstest and properly process the raw results. This would allow to natively support more interesting features. This task has existed in my backlog for a long time, and I hope to solve it one day, but no ETA for now.

[caaavik-msft]

Actually I expose a custom VSTest property exactly for this purpose: https://github.com/dotnet/BenchmarkDotNet/blob/master/src/BenchmarkDotNet.TestAdapter/VSTestProperties.cs#L15

This test property is attached as a custom property onto every testcase. It is definitely possible to build custom behaviour on top of the test adapter that uses this property. The value of this property is a string[] and it's populated here with the nanoseconds values for each iteration: https://github.com/dotnet/BenchmarkDotNet/blob/master/src/BenchmarkDotNet.TestAdapter/VSTestEventProcessor.cs#L123