github-action-benchmark
github-action-benchmark copied to clipboard
Action Will Not Create Summary Tables if Performance Alerts Occur
We are trying to have the Summary table always appear - even after an alert-threshold
is exceeded.
We use this action to perform benchmarks on several os
and device
types in a matrix. This splits our benchmarks into separate jobs for each ${{ matrix.os }}-${{ matrix.device }}
.
The Summary does not appear for devices where the alert-threshold
is exceeded, even when fail-on-alert: false
is set and summary-always: true
is enabled.
This is true whether comment-on-alert
is set or not, and even when we set a very high fail-threshold
to ensure it is not using the alert-threshold
value by default and make sure that the job does not fail.
After testing several different ways, it looks like the action is not set up to create a summary table at all if a performance alert occurs.
- name: Analyze benchmark results
id: analyze-bm
uses: risc0/[email protected]
with:
name: "${{ matrix.os }}-${{ matrix.device }}"
tool: 'customBiggerIsBetter'
output-file-path: target/hotbench/fib/benchmark.json
external-data-json-path: ./cache/external.json
github-token: ${{ secrets.GITHUB_TOKEN }}
alert-threshold: '120%'
fail-threshold: '10000%'
comment-on-alert: true
fail-on-alert: false
summary-always: true