Improve falco benchmarking, performance, and regression tooling to better track system resources impact

[happy-dude]

Motivation

Hey team, while evaluating and understanding the relationship between Falco, system resources, and detection rules, I was wondering if there was a way to better monitor and correlate the impact of Falco config and rule changes. With this information, I can better optimize and tune Falco for our unique envirionment.

The generally falls under the lines of a Falco benchmarking or instrumentation toolchain. For comparison, osquery provides a tool that provides some info on it's queries and configuration.

Additionally, it was discussed in the Slack community that something during CI/CD would be useful as well for regression testing.

Feature

• Userspace instrumentation/benchmarking tool to correlate impact of config settings and rules on system resources
• Incorporate CI/CD tooling for rules to better track performance improvements/regressions to code changes
• Provide recommendations on how to improve problematic rules?
• Possible documentation improvements, as there are a few blog posts (falco, sysdig, book) that sufficiently go over performance impact and considerations in depth, and fewer in a consumable "general best practices" way.

Additional context

See #2222, libs#531, Slack thread for more info