coreruleset
feat: add quantitative testing
what
- 💡 add quantitative testing to the testing tool !
- run using a local engine instead of parsing logs!
- let the tool interface with the corpus to test, instead of pointing to a file
- allow caching corpus files to ease CI/CD testing caching big files
- plenty of variables and highly configurable from the cmd line
- [EXPERIMENTAL] interface for adding more corpus
- right now the tests are run against CRS mostly, but could be extended to additional rulesets
why
- reuse the tool we use for testing and add more features
- speed
future
- get a threshold for knowing what is considered "bad" or "worse" than before
- probably using go funcs to process corpus in parallel to lower times. right now is just line by line, which probably is really underperformant
Could you please link some basic theoretical explanation to this issue to outline the motivation standing behind this PR. Thanks in advance!
Well, I don't know if there is any "theoretical" explanation here, other than plain numbers.
We take a bunch of standard (meaning it doesn't contain attacks) text grabbed from the internet, and we run it against CRS. We get the percentage of the text that matches certain rules.
If you modify a rule and the numbers go up, your change will get more false positives. That's the gist of quantitative testing around rules.
BTW, this is experimental until we have a good notion on what output we want from the tool.
@fzipi there are still two unresolved comments from the previous review.
I always fall in the hidden comments 🤦
- Added some factory methods
- Cleaned up outputs
- Added line number when printing payload in debug mode