apache
chore: add codspeed for benchmark
Which issue does this PR close?
Closes #4937 .
Rationale for this change
Trying to add a benchmark workflow to track and compare performance
What changes are included in this PR?
A new workflow that runs on every PR and the default main branch.
The criterion dependency is changed to codspeed-criterion-compat.
This should not affect cargo bench
Are there any user-facing changes?
No
Wait for approval from infra team https://issues.apache.org/jira/projects/INFRA/issues/INFRA-26030?filter=allissues
Hi, @adriencaccia, OpenDAL has a benchmark suite that can run tests over different services based on the environment. Does Codspeed support our case? For example, running Cargo Codspeed multiple times can display their performance in groups based on service.
Hi, @adriencaccia, OpenDAL has a benchmark suite that can run tests over different services based on the environment. Does Codspeed support our case? For example, running Cargo Codspeed multiple times can display their performance in groups based on service.
Hey, at the moment there is no support for grouping benchmarks.
What I would recommend in the meantime, if relevant, is to manually change the name of the benchmarks depending on the service.
For example, for a benchmark myBench in the codebase, you could make its name depending on the service, so that multiple benchmarks are tracked on CodSpeed: service1-myBench, service2-myBench, service3-myBench...
Hi, @adriencaccia, there are some concerns from the ASF INFRA, do you have comments?
Hi, @adriencaccia, there are some concerns from the ASF INFRA, do you have comments?
Hey, I just answered in https://github.com/apache/arrow-rs/pull/6150#issuecomment-2317740163 😉
Hi, @dqhl76, this action has been approved! Please give it a try.
Also FYI @adriencaccia and @alamb
Hello, thank you @dqhl76 for your efforts and thank you @adriencaccia for the updates.
However, the solution proposed in this PR is not what we are looking for. I'm going to close it now. Apologies for any inconvenience, and I hope we can achieve better integration in the future.
Thank you for @adriencaccia's new idea. I re-open this PR to make it possible for @dqhl76 to continue.
It seems we are only permitted to install the Codspeed GitHub app, but the workflow is not being allowed.
https://github.com/apache/opendal/actions/runs/11463180557
I think it still needs to communicate with infra team in jira ticket.
It seems we are only permitted to install the Codspeed GitHub app, but the workflow is not being allowed.
apache/opendal/actions/runs/11463180557
I think it still needs to communicate with infra team in jira ticket.
This is where the action can be allowed: https://docs.github.com/en/organizations/managing-organization-settings/disabling-or-limiting-github-actions-for-your-organization#allowing-select-actions-and-reusable-workflows-to-run
This is where the action can be allowed:
Thanks for the link. However, we don't have the organizations privilege😢, still need to seek for permission from infra team. I will try to do it tomorrow.
Close this PR, I will reopen it on opendal repo's branch. Seems all the obstacles are removed. Thanks for all your work.
This PR is from my forked repo's branch, we cannot run the action that need password.