merlin
merlin copied to clipboard
Add benchmarks for merlin locate on irmin files
The purpose of this PR is to benchmark merlin locate using the command ocamlmerlin single locate -look-for ml -position 23:22 -filename ./examples/irmin_git_store.ml < ./examples/irmin_git_store.ml | jq. On the Irmin repository.
To do this we added bench.Dockerfile and ./tests/bench.sh.
The bench.Dockerfile file contains all the environment to clone Irmin repository and launch the necessary installations for Merlin and Irmin. The ./tests/bench.sh file allows us to run the benchmark on cases where the merlin locate command takes the longest time to run. We were able to identify these cases by running merlin locate on each word of each Irmin file that we tested on two different versions of ocaml (4.14.0 and 4.12.1).
This comparison allowed us to spot a performance regression of merlin that we have reported on https://github.com/ocaml/merlin/issues/1468.
In this commit we added benchmarks for ocamlmerlin type-enclosing command following the same methodology as locate command.
To launch the benchmarks, you must install the current-bench application "https://github.com/marketplace/ocaml-benchmarks" on the Merlin repository. The benchmarks will launch on every PR.
In this last commit we added the benchmark of the occurrences command following the same methodology as the locate and type-enclosing commands. We also put all callsigns (Query, typer, clock, ppc and pp) in one graph for each case in the benchmarks.
@voodoos > We'll soon be able to run the benchmarks only for PRs that have been marked with a specific github label: for example "bench", but you just have to let us know which tag name works for you? Currently it's a bit hard to watch for a list of modified files but we might get back to you on this in the future :)
Regarding the CI setup, you need someone with enough administrative power over the ocaml organization to install the ocaml-benchmarks GH app at https://github.com/marketplace/ocaml-benchmarks , then enable it only on the merlin repository (there's a dropdown to pick the repo right after the "installation").
I would say something like need-bench ?
I know this PR is old, but current-bench has just merged the PR allowing us to only benchmark PRs with a specific label (https://github.com/ocurrent/current-bench/pull/371).
@voodoos should I go ahead and set need-bench as the necessary label?
Thanks a lot @Zineb-Ada for this precursor work ! We just merged a more complete bench CI so I am closing this one.