[WIP] Improve benchmark pipeline
Adding benchmarks pipeline based on this action: https://github.com/marketplace/actions/continuous-benchmark
The current pipeline is to backfill all historical benchmark data, which prompted me to do some interesting stuff with the code, so ignore removing nullaway and adding old jsr305 dependencies, this all is needed to be able to compile latest benchmarks together with the historical code 🙃 I also tried to unify all benchmarks to use the same parameters, so we have something that's consistent.
@Breus @donavdey please review the changes to benchmarks (parameters and json path changes), ignoring all the others for now. If we need any changes to the benchmarks and backfill the historical data, then we will need all these hacks to rerun them.
The benchmarks are pushed into https://gavlyukovskiy.github.io/json-masker-benchmarks/, but I plan to change that to json-masker repository, for that we need to enable github-pages for the repository and push into gh-pages branch used to render it (and also remove my clear-text PAT 😅).
The pipeline will need to be configured to only push results from master branch and comment on PRs, it has a couple of options like commenting on specific threshold or always, but it requires merging the pipeline into master to test it out. Or we can play with it in a separate repository.
@gavlyukovskiy would it be nice if I prepare json-masker.blaauwendraad.dev to publish benchmark results and maybe some kind of public documentation page? We can also host our own JavaDoc there :-)
@Breus yes, please enable gh pages (some repo setting, I think), you will need to commit empty .nojekyll into the root of gh-pages branch and then I can transfer everything from my repository (https://github.com/gavlyukovskiy/json-masker-benchmarks) into that one. We don't have to use custom domain as after enabling Github Pages we will have https://breus.github.io/json-masker/ for free, but we can do custom domain as well :)
We could also reuse the same PAT to publish this as we do for gradle wrapper updates (needs additional "pages" permission).
I also had the same thoughts about javadoc 😁 I think we could do that, but also not much is wrong with our current service. In any case, we can use /benchmarks for the benchmarks and leave a room for something else later.
It also probably makes sense to remove src/jmh/benchmark-history folder as the history will be stored somewhere else.
@donavdey yea, it's a long time irrelevant at this point
Quality Gate passed
Issues
0 New issues
0 Accepted issues
Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code
Since we don't really have new feature requests, in our vision the library is feature-complete, there is no point in improving benchmark set-up. We might do this later and then we can reopen the issue.