benchmark-action
Catch2 store fails if estimated time exceeds 1m
Using the catch2 framework to benchmark some tests I encountered the following issue: If your total estimated time for the tests exceeds the 60s, then the display time switches to minutes. If this happens, then the store action fails.
I have adjusted the catch2 example in https://github.com/phbasler/github-action-benchmark/tree/master/examples/catch2 ( added a 0.7s sleep before the fib(10) call so that the 100 executions exceed the 60s.
In the CI workflow the "Build and run benchmarks with Catch2" part is successful, but the "Store benchmark results" sections yields the following error:
`2022-04-05T07:01:00.8358638Z ##[group]Run benchmark-action/github-action-benchmark@v1 2022-04-05T07:01:00.8358959Z with: 2022-04-05T07:01:00.8359184Z name: Catch2 Benchmark 2022-04-05T07:01:00.8359428Z tool: catch2 2022-04-05T07:01:00.8359724Z output-file-path: examples/catch2/benchmark_result.txt 2022-04-05T07:01:00.8360710Z github-token: *** 2022-04-05T07:01:00.8360988Z auto-push: true 2022-04-05T07:01:00.8361230Z alert-threshold: 200% 2022-04-05T07:01:00.8361470Z comment-on-alert: true 2022-04-05T07:01:00.8361716Z fail-on-alert: true 2022-04-05T07:01:00.8401114Z alert-comment-cc-users: @bernedom,@ktrz 2022-04-05T07:01:00.8401461Z gh-pages-branch: gh-pages 2022-04-05T07:01:00.8401784Z benchmark-data-dir-path: dev/bench 2022-04-05T07:01:00.8402088Z skip-fetch-gh-pages: false 2022-04-05T07:01:00.8402344Z comment-always: false 2022-04-05T07:01:00.8402594Z save-data-file: true 2022-04-05T07:01:00.8402834Z ##[endgroup] 2022-04-05T07:01:01.1730005Z ##[error]No benchmark found for bench suite. Possibly mangled output from Catch2:
Catch2_bench is a Catch v2.11.0 host application.
Run with -? for options
-------------------------------------------------------------------------------
Fibonacci
-------------------------------------------------------------------------------
/home/runner/work/github-action-benchmark/github-action-benchmark/examples/catch2/catch2_bench.cpp:8
...............................................................................
benchmark name samples iterations estimated
mean low mean high mean
std dev low std dev high std dev
-------------------------------------------------------------------------------
Fibonacci 10 100 1 1.16689 m
700.121 ms 700.118 ms 700.124 ms
16.061 us 14.141 us 18.543 us
Fibonacci 20 100 2 6.051 ms
30.137 us 29.714 us 32.152 us
4.019 us 80 ns 9.574 us
===============================================================================
test cases: 1 | 1 passed
assertions: - none -`
