GitHub merge queue not supported

[Swatinem]

We use the GitHub merge queue over at https://github.com/codecov/shared.

The PR that introduced the codspeed action ran successfully, although with a small issue: https://github.com/codecov/shared/pull/520#issuecomment-2672103196

The merge to the main branch however has failed:

Error: Event merge_group is not supported by CodSpeed
Error: unknown variant `merge_group`, expected one of `push`, `pull_request`, `workflow_dispatch`, `schedule`, `local` at line 1 column 13

https://github.com/codecov/shared/actions/runs/13452735007/job/37590061959#logs

[art049]

Merge queues create a few problems for the current product, but we can probably find a good middle ground for supporting them.

Essentially, what would you expect from running CodSpeed in the merge queue?

Our big question is: How should the regression threshold handle the merge group size? Right now, you can set a threshold for the PR, which will determine whether the status check passes. However, a merge_group containing 3 PRs with small regressions could accumulate into a significant regression that would prevent the merge queue from succeeding because of the performance status check, while individual PRs checks were completely fine.

[Swatinem]

That is a good point I haven’t really thought about.

As you mentioned, if we assume the PRs pass individually, they should also pass when run as a group, right? On the other hand though, when they are merged together, you can’t tell them apart anymore.

For our usecase specifically, I don’t think we have so many PRs in the queue that we would even have multiple merges performed as one. In that case, if the group size is 1, there would obviously be no difference to a normal merge.

In either case, the question of how to deal with the merge group size and thresholds is a different one than flat out erroring out when using the merge queue at all :-D