Feature Request: Setup and Teardown for `benchmark_base`

[codecircuit]

To implement a Setup, which is shared among all states of a benchmark_base a current solution could be:

void my_benchmark(nvbench::state& state) {
  static int num_execs = 0;
  if (execs == 0) {
    // SETUP calls here
    // e.g. expensive hard disk I/O
    ++execs;
  }
  state.exec([](nvbench::launch& launch) { 
    my_kernel<<<num_blocks, 256, 0, launch.get_stream()>>>(/* uses data from SETUP */);
  });
}
NVBENCH_BENCH(my_benchmark).add_int64_axis("i", {1, 2, 3, 4, 5, ..., 99});

It would be more convenient to have an explicit setup and teardown functionality (similar to e.g. Boost Test). For example:

void my_benchmark_setup() {
  // SETUP calls here
}

void my_benchmark_teardown() {
  // TEARDOWN calls here
}

void my_benchmark(nvbench::state& state) {
  state.exec([](nvbench::launch& launch) { 
    my_kernel<<<num_blocks, 256, 0, launch.get_stream()>>>(/* uses data from SETUP */);
  });
}
NVBENCH_BENCH(my_benchmark).add_int64_axis("i", {1, 2, 3, 4, 5, ..., 99})
  .register_setup(my_benchmark_setup).register_teardown(my_benchmark_teardown);

A discussion about how the setup and teardown registration should look like, would be helpful.

[jrhemstad]

EDIT: Nevermind. I totally overlooked that you wanted this to be shared among all states of the benchmark. The solution I described above would be unique per state.

~Manually registered setup/teardown functions are a bit inconvenient and means any state has to be passed through global state/side-effects. I think something akin to fixture classes would be more convenient.~

~Perhaps something like this:~

struct my_fixture{
   my_fixture(){ /* setup */}
   ~my_fixture(){ /* tear down */}

   auto get_resources(){ /* return whatever resource(s) this fixture owns */}
};


void my_benchmark(nvbench::state& state, my_fixture& fixture){
   auto resources = fixture.get_resources();
   state.exec([](nvbench::launch& launch) { 
    my_kernel<<<num_blocks, 256, 0, launch.get_stream()>>>(resources);
   });
}


NVBENCH_FIXTURE_BENCH(my_fixture, my_benchmark).add_int64_t_axis(...);

~Internally, when creating the callable wrapper, it can make the callable construct the fixture and pass it to the benchmark function.~

#define NVBENCH_DEFINE_FIXTURE_CALLABLE(fixture_name, function, callable_name)                       \
  struct callable_name                                                        \ \
  {                                                                          \
  {                                                                            \
    void operator()(nvbench::state &state, nvbench::type_list<>)               \
    {                                                                          \
      fixture_name f{};                                                        \
      function(state, f);                                                      \
    }                                                                          \
  }

[jrhemstad]

To have the setup/teardown shared among all states of the benchmark, the same user-facing interface would work. Internally, instead of constructing the fixture before each benchmark invocation, it can be constructed before the iteration through the states of the benchmark https://github.com/NVIDIA/nvbench/blob/610b7767b5922f6c50941cfc271d8d76b9546583/nvbench/runner.cuh#L97

auto fixture = benchmark_type::fixture_type{};
for (nvbench::state &cur_state : states) {
   ...
   if constexpr( std::is_same_v<benchmark_type::fixture_type, no_fixture> )
      kernel_generator{}(cur_state, type_config{});
   else
      kernel_generator{}(cur_state, type_config{}, fixture);
}

https://github.com/NVIDIA/nvbench/blob/ff507596bfa76a3875dffade60cb8747cd8f41fc/nvbench/benchmark.cuh#L53-L54

This would likely require modifying the benchmark type to add an additional template parameter for the FixtureType that can be defaulted to a sentinel tag type:

struct no_fixture{};
template <typename KernelGenerator, typename TypeAxes = nvbench::type_list<>, typename Fixture = no_fixture>
struct benchmark{
   using fixture_type = Fixture;
   ...
}