getsentry
Compare benchmarks with previous versions
Description
Android has some benchmark ui tests (https://github.com/getsentry/sentry-java/pull/2013) that record several values and check they don't pass a certain threshold, e.g. 5% for profiling, 500 ms to init the sdk, etc. It would be cool to have a way to compare them with each version of the sdk, to check whether any performance regression was introduced at some point. Ideally, we could find a way to save all these values in Sentry so that we could retrieve them later and query them in any possible way The same thing applies to iOS and possibly other platforms, too @marandaneto Is it fine?
Ideally, we could find a way to save all these values in Sentry so that we could retrieve them later and query them in any possible way
That could be also done on CI and GH Actions, See the JS SDK https://github.com/getsentry/sentry-javascript/blob/c518955b774e5bbc36e7f1a22410703469911a2a/.github/workflows/build.yml#L102-L130 Using the Action https://github.com/getsentry/size-limit-action or similar.
We could add a file to the GH repo containing a table of benchmark results and commit hashes for storing the results.
We could add a file to the GH repo containing a table of benchmark results and commit hashes for storing the results.
Wouldn't that be prone to concurrency issues?
Wouldn't that be prone to concurrency issues?
I don't really understand your question. Can you please elaborate?
Depending on how you want to do this, there might be multiple builds trying to write their result to the file at the same time. Not sure we need to write every result and commit hash, maybe latest result is enough. Could write that to a file named after the branch so this wouldn't only work for master but maybe also for long lived branches like we had with 6.0.
Depending on how you want to do this, there might be multiple builds trying to write their result to the file at the same time. Not sure we need to write every result and commit hash, maybe latest result is enough. Could write that to a file named after the branch so this wouldn't only work for master but maybe also for long lived branches like we had with
6.0.
That's not a problem, PRs are isolated in a different branch, and even if uploading this file somewhere, let's say using actions/upload-artifact, we could use a file name that avoids that.
PRs only read from main as well, unless it gets merged with newer baselines.
Here's the relevant github action which already does an APK size diff: getsentry/action-app-sdk-overhead-metrics
We could reuse the same infrastructure for storing benchmark results.
