json-masker icon indicating copy to clipboard operation
json-masker copied to clipboard

[WIP] Improve benchmark pipeline

Open gavlyukovskiy opened this issue 1 year ago • 5 comments

Adding benchmarks pipeline based on this action: https://github.com/marketplace/actions/continuous-benchmark

The current pipeline is to backfill all historical benchmark data, which prompted me to do some interesting stuff with the code, so ignore removing nullaway and adding old jsr305 dependencies, this all is needed to be able to compile latest benchmarks together with the historical code 🙃 I also tried to unify all benchmarks to use the same parameters, so we have something that's consistent.

@Breus @donavdey please review the changes to benchmarks (parameters and json path changes), ignoring all the others for now. If we need any changes to the benchmarks and backfill the historical data, then we will need all these hacks to rerun them.

The benchmarks are pushed into https://gavlyukovskiy.github.io/json-masker-benchmarks/, but I plan to change that to json-masker repository, for that we need to enable github-pages for the repository and push into gh-pages branch used to render it (and also remove my clear-text PAT 😅).

The pipeline will need to be configured to only push results from master branch and comment on PRs, it has a couple of options like commenting on specific threshold or always, but it requires merging the pipeline into master to test it out. Or we can play with it in a separate repository.

gavlyukovskiy avatar Sep 25 '24 11:09 gavlyukovskiy

@gavlyukovskiy would it be nice if I prepare json-masker.blaauwendraad.dev to publish benchmark results and maybe some kind of public documentation page? We can also host our own JavaDoc there :-)

Breus avatar Sep 26 '24 09:09 Breus

@Breus yes, please enable gh pages (some repo setting, I think), you will need to commit empty .nojekyll into the root of gh-pages branch and then I can transfer everything from my repository (https://github.com/gavlyukovskiy/json-masker-benchmarks) into that one. We don't have to use custom domain as after enabling Github Pages we will have https://breus.github.io/json-masker/ for free, but we can do custom domain as well :)

We could also reuse the same PAT to publish this as we do for gradle wrapper updates (needs additional "pages" permission).

I also had the same thoughts about javadoc 😁 I think we could do that, but also not much is wrong with our current service. In any case, we can use /benchmarks for the benchmarks and leave a room for something else later.

gavlyukovskiy avatar Sep 26 '24 09:09 gavlyukovskiy

It also probably makes sense to remove src/jmh/benchmark-history folder as the history will be stored somewhere else.

donavdey avatar Oct 03 '24 17:10 donavdey

@donavdey yea, it's a long time irrelevant at this point

gavlyukovskiy avatar Oct 03 '24 17:10 gavlyukovskiy

Since we don't really have new feature requests, in our vision the library is feature-complete, there is no point in improving benchmark set-up. We might do this later and then we can reopen the issue.

Breus avatar Jul 11 '25 09:07 Breus