Action is not creating any comments on PR

[nilsfriess]

I was trying to setup a Github action that runs when I open a PR and add a comment with the content /benchmark. Here's my workflow file:

name: Benchmark test

on:
  issue_comment:
    types: [created, edited, deleted]

jobs:
  run-benchmark:
    if: contains(github.event.comment.html_url, '/pull/') && contains(github.event.comment.body, '/benchmark')
    runs-on: ubuntu-latest
    steps:
    - uses: actions/checkout@v4      
    - name: Run benchmark
      run: |
        python3 gen_random_result.py | tee ${GITHUB_WORKSPACE}/output.json
    - name: Download previous benchmark data
      uses: actions/cache@v4
      with:
        path: ./cache
        key: ${{ runner.os }}-benchmark

    - name: Store new result
      uses: benchmark-action/github-action-benchmark@v1
      with:
        tool: customSmallerIsBetter
        output-file-path: output.json
        external-data-json-path: ./cache/benchmark-data.json
        github-token: ${{ secrets.GITHUB_TOKEN }}
        summary-always: true
        comment-always: true

The gen_random_result.py script does exactly what its name says, it just generates some random data of the form

[
 {
    "name": "Test benchmark",
    "unit": "Seconds",
    "value": "0.4937825918725717"
 }
]

The action runs as expected but it doesn't generate a comment on the PR with the results. Is there something I'm missing?