Ability to provide raw input data? e.g., in order to cse from C++, other languages, complex benchmarks?

[jonstewart]

Criterion is great! I work across Python, C++, and Rust. One thing I have wanted in a benchmarking tool is the ability to feed in raw benchmark data from outside the framework. For example, in C++, the Catch2 unit test framework has some microbenchmarking facilities that will run benchmark functions for a number of iterations in order to get a good sample.

What I would love to be able to do is feed in the data from Catch2 benchmarks to Criterion, in order to get the commit-aware analysis, reports, and plots. It would also be nice to do similarly with acceptance test-level benchmarks involving complicated cloud data processing pipelines (where results may be measured in minutes) — things that are often scripted out, but without the analysis and reporting support that Criterion provides.

Is this possible with some jiggery-pokery in Criterion, or would ever be possible? Even something hacky would beat rolling my own solutions for other languages.

[iilyak]

Same usecase here. I am in the process of choosing the implementation language. Which mean I am running microbenchmarks available on those languages and comparing the results with rust implementation. I would like to plug-in the results to create the report.

I am considering using --message-format=json to get data-points from rust implementation and do the analysis myself. But this would mean re-implementing a lot of things from scratch.

[bend-n]

--message-format json wont tell you the benchmark results.

Criterion has to accept a --json argument, so you can use it as cargo bench -- --json 2>/dev/null. Ive added a --json argument to my fork of iai.