Add example showing how to parse additional arguments

[alliepiper]

Several users have asked about how to add custom arguments to their benchmarks (e.g. #86).

This is done by implementing an application specific main(argc, argv) function and linking to the nvbench::nvbench CMake target instead of nvbench::main. The custom main will need to parse out additional arguments, and then use the macros defined in main.cuh to parse the remaining args and run NVBench. Something like:

int main(int argc, char** argv)
{
  // parse custom args and remove them from (or rebuild) argc/argv here
  
  NVBENCH_MAIN_BODY(argc, argv);
}

A singleton or some global state is then needed to communicate the custom options to the benchmarks.

We should add an example that shows how to do this.

[jrhemstad]

I had to solve the same problem with using GBench a long time ago.

I found that when defining a custom main, GBench has a ::benchmark::Initialize(arg,argv) function that will strip out any of the command line arguments it recognizes and modify argc/argv accordingly.

See: https://github.com/rapidsai/rmm/blob/4e8bca57a8b7cbc46c05ded1fbb70b57f9042e75/benchmarks/replay/replay.cpp#L332-L334

Instead of making it the users responsibility to parse out custom args, I think it would be nice to do the opposite and have nvbench strip out the args it knows about and leave the rest.

[alliepiper]

I agree -- this process could be made much more user friendly and we should definitely explore these sorts of conveniences. I'd still want to error out for unknown args when using the default main implementation, but I'm on-board with @jrhemstad's suggestion for user-defined main.

[annielytical]

FYI we were able to make it work (here's an example: https://github.com/gunrock/essentials/blob/d1f668cdb062b1c5ef442b1eef2a1cb0405e688f/benchmarks/mst_bench.cu). I'd be happy to add an example here as well. Thanks for your help @allisonvacanti !

[alliepiper]

A PR with a simple example would definitely be welcome! I think we'll want to make some additional changes to NVBench to simplify this usecase, but having the current example will help highlight some of the rough edges.

[alliepiper]

One potential interface for this would be to provide a new variant of the NVBENCH_MAIN macro, that allows default arguments to be specified along with a lambda for parsing custom args. Something like:

NVBENCH_MAIN_ARGS(
  {"-q", "--timeout", "60"}, // Prepended args
  [](nvbench::option_parser& parser)
  {
    for (arg : parser.get_unknown_args())
    {
      if (arg == "--custom_format")
      { // Ok, can't use range-for and get next_arg...
        parser.add_printer<custom_format>(next_arg);
      }
      else if (arg == "--validate")
      { // Benchmarks check this and optionally do validation:
        g_do_validation = true;
      }
    }
  });

[alliepiper]

I was working on our main implementation recently, and added some nicer hooks for doing common things like parsing other options. Check this out to see if it gives you a better approach, the test files have examples: https://github.com/NVIDIA/nvbench/pull/165

I think there's still more work to do in this area that would involve cleaning up the argument parsing machinery and making it directly extensible, but these hooks should address simple cases.