grammars-v4
grammars-v4 copied to clipboard
[build] Add performance test of a PR
Antlr4 accepts almost any grammar. But, this isn't necessarily a good thing. Just because Antlr4 accepts the grammar, it doesn't mean the grammar is efficient.
PRs to this repo are usually made to solve syntax problems. Unfortunately, PRs often introduce performance problems. Therefore, we need a way to double-check that PRs aren't going to screw things up any more than a grammar currently is.
While we currently cannot test whether a grammar is ambiguous, we can compare the performance of an updated grammar with the previous version. If a PR causes a significant performance decrease, we should note the problem.
The first assumption we're going to have to make is that whatever was accepted into a grammar prior to the PR was "acceptable". This may not be a great assumption as I have seen just awful PRs get merged.
Next, I want the performance testing to be on a separate workflow, as much as possible disconnected from the nomral "CI" workflow for the PR because I don't want the success of the PR to depend on the perf test. I may just have the test only on a merge to the main branch.
The output of the performance test should be "ok: no statistical difference" or "failed: significant difference". The Student t-test should be used to determine whether there is a slow-down. The sample size is unknown at this point, but 40 usually works for me.
In addition, it would be nice to see Octave bar-graphs for each of the grammar tests, which would show the runtime for the test suite before and after the PR. These .svg files should be archived somewhere, or left as an artifact. However, it would be good to have these available in the readme's for the grammar.