trufflehog icon indicating copy to clipboard operation
trufflehog copied to clipboard

Skip detectors for known bad chunks

Open rgmz opened this issue 2 years ago • 4 comments

Description:

This is a POC to fix #1517. Skipping detectors on chunks that are known to be problematic (e.g., #1460) should improve performance by reducing the number of false-positives and extraneous network requests.

Any feedback and suggestions are welcome. (Also, I have yet to test whether this specific code works; emphasis on "concept".)

Checklist:

  • [ ] Tests passing (make test-community)?
  • [ ] Lint passing (make lint this requires golangci-lint)?

rgmz avatar Sep 01 '23 04:09 rgmz

CLA assistant check
All committers have signed the CLA.

CLAassistant avatar Jan 13 '24 22:01 CLAassistant

@rosecodym I've updated the implementation based on feedback and testing. Let me know your thoughts on this.

rgmz avatar Jan 13 '24 22:01 rgmz

@rgmz we just had an internal conversation about this and a question came up: How much of this problem is associated with lockfiles specifically? If the answer is "a lot," does it make sense to either use file exclusions or to consider skipping them at the engine level rather than adding complexity to detectors?

rosecodym avatar Jan 22 '24 21:01 rosecodym

I wouldn't say the problem is specific to lock files. There are lots of files that create a high volume of false detections for specific detectors, that doesn't mean the files should be skipped altogether — even lock files have a one-in-a-million chance to contain secrets.

rgmz avatar Jan 22 '24 22:01 rgmz