Walltime measurement not uploading properly using default GitHub Runners

[Swatinem]

I see that the walltime measurement is still considered beta, and is advertised together with your custom hosted runners (https://docs.codspeed.io/instruments/walltime/)

Because the instrumentation based measurement is unbearably slow for us, I tried to use --codspeed-mode=walltime instead.

This works in CI, and has a much faster runtime, but somehow does not properly upload those reports, and the rest of codspeed seems to be unusable for the following reasons:

• The PR comment does not list any benchmarks (https://github.com/codecov/shared/pull/520#issuecomment-2671917738)
• The runs are listed in the codspeed ui (https://codspeed.io/codecov/shared/runs), but the links to the details result in a 404 (https://codspeed.io/codecov/shared/runs/67b83b8b43ec322b90fca5c3)
• The overview and list of benchmarks is empty, similar to the PR comment.

See also https://github.com/codecov/shared/pull/520#issuecomment-2672103196 for some more details of why I wanted to switch to walltime measurement.

[art049]

Yes, right now, we only expect the wall time runs to run on our hosted bare-metal runners to avoid excessive variance related to the provisioning of GitHub Action hosted runners.

If you're up for it, I can let you in the macro-runners beta!

[Swatinem]

I would be certainly up for it, in case the bare-metal runners are free to use for open-source repos. Otherwise I would defer this decision to @trent-codecov.

I think the primary problem for us specifically is that we have "macro benchmarks", that run our (sadly very slow) code with a fairly large fixture, meaning that each iteration takes a long time, and thus the synthetic valgrind-based instrumentation does not run in any reasonable time.

[art049]

Definitely it makes sense to try out the walltime instrument if you have macro benchmarks.

I enabled the beta feature, so you have a first 120mn of free tier but then we have a usage based pricing for those runners. Discussing this by email would be more convenient but feel free to try it out before.

[art049]

Hey @Swatinem, we changed it and now walltime works on all runners anyway!