Skip detectors for known bad chunks
Description:
This is a POC to fix #1517. Skipping detectors on chunks that are known to be problematic (e.g., #1460) should improve performance by reducing the number of false-positives and extraneous network requests.
Any feedback and suggestions are welcome. (Also, I have yet to test whether this specific code works; emphasis on "concept".)
Checklist:
- [ ] Tests passing (
make test-community)? - [ ] Lint passing (
make lintthis requires golangci-lint)?
@rosecodym I've updated the implementation based on feedback and testing. Let me know your thoughts on this.
@rgmz we just had an internal conversation about this and a question came up: How much of this problem is associated with lockfiles specifically? If the answer is "a lot," does it make sense to either use file exclusions or to consider skipping them at the engine level rather than adding complexity to detectors?
I wouldn't say the problem is specific to lock files. There are lots of files that create a high volume of false detections for specific detectors, that doesn't mean the files should be skipped altogether — even lock files have a one-in-a-million chance to contain secrets.