Run benchmarks with `fpm bench`

[milancurcic]

Description

I'd like to have an easy way to run a series of programs that time various parts of the library.

Currently this could be done by placing program source files in test or example, but neither seem topically appropriate.

Cargo has cargo bench. I believe their directory structure assumes benchmark programs in the benches/ directory.

In the absence of such feature, this is still relatively easy to do via tests or examples. However, having something like fpm new --bench auto-generate a little benchmarking boilerplate, and a built-in fpm bench to run them could encourage people to time their libraries and make it easy for users to do so. I'm not convinced yet this is a good idea, but at least initially it seems so to me.

Possible Solution

1. Extend fpm new to allow fpm new --bench to create bench/ directory with a small program that times a library procedure.
2. Introduce fpm bench to scan the bench directory and run them. Just like fpm test.

Additional Information

cargo-bench

[awvwgk]

Would it be feasible to group benchmarks together with examples and executables, with tests or have them entirely separate from those two categories? Grouping with the executables would result in fpm run --benchmark --all while the latter results in fpm test --benchmark (note the different semantics for running all executables).

Meson is grouping benchmarks together with tests, there is a dedicated benchmark() call for creating targets which are run by using meson test --benchmark. For CMake there seems to be no dedicated benchmark command, rather one has to add a prefix or mark tests with attributes which can be selected via CTest later or write a custom command which invokes the benchmark executable.

[urbanjost]

The new subcommand was originally proposed to have a bench option; I was trying to find some of the discussion but did not find it. There was also discussion of a more generic solution where you could add an arbitrary directory name to the fpm.toml file including dependencies, etc. just like test and example that would not be built by default unless named on one of the commands. So imagine test and example are not currently defined but fpm let you define them just like now but the directories would be ignored unless specifically named, like "fpm -run test" or "fpm -run bench" or any other directories. The default name would be app. It would essentially add some conditional processing to fpm.toml (which I would like to see anyway, like if/else/endif; especially for compiler flag sets). Maybe still use a subcommand but if the name ends with a / it changes the directory, like fpm run bench/ prog1. so instead of continuing to add names like bench and profile and example and test and ... you could have some generic mechanism.

In the mean-time I put everything in test. That way everything runs (the default for the test directory is to run everything) with a simple "fpm test", but then I use "fpm test 'test*' and fpm test 'bench* and group the files by name with different prefixes.

[milancurcic]

I didn't know that I could do fpm test bench_* to only run those. Very cool and close enough.

[ivan-pi]

In Go they use go test -bench <regexp> to run Benchmarks. They also define a standard benchmark data format.

[urbanjost]

Looks like they define a entire benchmarking/testing framework. Interesting red.

[urbanjost]

We could easily do simple regex with M_match, which would keep it all Fortran; but from earlier discussions we went with globbing. One problem with that is people are used to quoting regular expressions, but usually not globbing, unless we used other characters not as problematic in comdon't remember all the options butmon shells, like using % instead of *. Other options were to start an optional interactive mode where you are prompted for names (that would also support globbing-like syntax); or a menu and you could enter numbers. I do not remember all the options, but at the time we decided on quoted globbing which is much simpler than regular expressions and more commonly understood. It certainly is not perfect but gives a decent amount of flexibility with just a single routine. Did not implement character ranges "[charset]"; and the principal thing I would like is a pause option that would run one code and then pause; and that verbose mode should print out the name being executed.