[BUG] Infinite running benchmarks

[Kashio]

Describe the bug Benchmarks fixture don't seem to run unless in debug with breakpoints

System

• OS: Windows 10 x64
• Compiler and version: msvc 19.16.27034 x32/x64 (both)

To reproduce You can clone my repository https://github.com/Kashio/A5 and build with cmake

Expected behavior Benchmarks should run normally and not get stuck somewhere infinitely

Additional context If I place a break pointer in my first defined benchmark function BENCHMARK_F(MultipleFixedAllocations, CAllocator)(benchmark::State& state) and step through it to completion (by stepping into BenchmarkMultipleAllocations and then setting another breakpoint at the end of BenchmarkMultipleAllocations and jumping to it) Then I manage somehow to get the first benchmark case finished, but unless I'm stepping through code with breakpoints and just running it then it get stuck somewhere and never finishes and memory goes up (probably by my allocators objects since they're never freed) but it's as if it constructing them infinitely or something but I'm not even sure about that since I even placing breakpoints on their constructors don't get hit.

You can go to this commit https://github.com/Kashio/A5/tree/25aa53e372a4c6ae5cc02358383ac1d06f137792 which was before I've changed the benchmark function to fixtures. Before doing that everything worked fine. The reason I had to change that to fixtures instead of globally defined functions is that I wanted some setup code specific for these cases as can be seen on the master.

I'm not sure this is a bug, probably me not doing something right with setting the fixture tests. I'm pretty sure this is not a bug in my library code causing it since I've linked to a commit where everything worked fine without fixtures.

Here's a gif showing the problem that no tests are run and the memory just keeping going up: https://gfycat.com/physicalimmaterialgallinule

And Here's the same run with me just stepping with breakpoint and see how the first test somehow finished but then everything get stuck again: https://gfycat.com/windyphonydamselfly

[Kashio]

I realized my problem the Setup code is being called for each iteration/test case. I think this is counter intuitive that you define test case in fixture with BENCHMARK_F as it doesn't really make it a function of the class but make it a derived class and so Setup will be called for each test case separately instead of one Setup for one fixture. This and the fact that it also run for each iteration apparently (because I see Setup called multiple times for each created derived instance) couldn't be figured from the docs so I think a better documentation on what these Macros actually do and how this might affect you should be added.

[dmah42]

leaving this open as it's a FAQ and we probably do want to figure out the difference between a fixture and a 'test suite'. googletest has this notion with both static setup and teardown for the suite, and non-static setup and teardown per test unit.

i wonder if we should have something similar.