merlin icon indicating copy to clipboard operation
merlin copied to clipboard

Add benchmarks for merlin locate on irmin files

Open Zineb-Ada opened this issue 3 years ago • 4 comments
trafficstars

The purpose of this PR is to benchmark merlin locate using the command ocamlmerlin single locate -look-for ml -position 23:22 -filename ./examples/irmin_git_store.ml < ./examples/irmin_git_store.ml | jq. On the Irmin repository. To do this we added bench.Dockerfile and ./tests/bench.sh. The bench.Dockerfile file contains all the environment to clone Irmin repository and launch the necessary installations for Merlin and Irmin. The ./tests/bench.sh file allows us to run the benchmark on cases where the merlin locate command takes the longest time to run. We were able to identify these cases by running merlin locate on each word of each Irmin file that we tested on two different versions of ocaml (4.14.0 and 4.12.1). This comparison allowed us to spot a performance regression of merlin that we have reported on https://github.com/ocaml/merlin/issues/1468.

Zineb-Ada avatar Jun 23 '22 11:06 Zineb-Ada

In this commit we added benchmarks for ocamlmerlin type-enclosing command following the same methodology as locate command. To launch the benchmarks, you must install the current-bench application "https://github.com/marketplace/ocaml-benchmarks" on the Merlin repository. The benchmarks will launch on every PR.

Zineb-Ada avatar Jun 28 '22 15:06 Zineb-Ada

In this last commit we added the benchmark of the occurrences command following the same methodology as the locate and type-enclosing commands. We also put all callsigns (Query, typer, clock, ppc and pp) in one graph for each case in the benchmarks.

Zineb-Ada avatar Jun 30 '22 15:06 Zineb-Ada

@voodoos > We'll soon be able to run the benchmarks only for PRs that have been marked with a specific github label: for example "bench", but you just have to let us know which tag name works for you? Currently it's a bit hard to watch for a list of modified files but we might get back to you on this in the future :)

Regarding the CI setup, you need someone with enough administrative power over the ocaml organization to install the ocaml-benchmarks GH app at https://github.com/marketplace/ocaml-benchmarks , then enable it only on the merlin repository (there's a dropdown to pick the repo right after the "installation").

art-w avatar Jul 11 '22 15:07 art-w

I would say something like need-bench ?

voodoos avatar Jul 12 '22 16:07 voodoos

I know this PR is old, but current-bench has just merged the PR allowing us to only benchmark PRs with a specific label (https://github.com/ocurrent/current-bench/pull/371). @voodoos should I go ahead and set need-bench as the necessary label?

ElectreAAS avatar Feb 01 '23 13:02 ElectreAAS

Thanks a lot @Zineb-Ada for this precursor work ! We just merged a more complete bench CI so I am closing this one.

voodoos avatar Jul 06 '23 12:07 voodoos