trufflehog
trufflehog copied to clipboard
Fix broken & inaccurate detector unit tests
The detector unit tests were created by "reverse engineering" the patterns, thus they do not provide any real value or confirm how accurate the detectors are.
Additionally, the structure of the tests are (in my opinion) not maintainable as they are difficult to understand and do not reflect any realistic scenarios.
https://github.com/trufflesecurity/trufflehog/blob/def734a783b2d4542f2b353d326130d618f3c421/pkg/detectors/netsuite/netsuite_test.go#L14-L44
Examples
There are dozens, if not hundreds, of problematic test files. These are illustrative.
BombBomb
The "valid" tests for BombBomb do not match the detector's pattern.
https://github.com/trufflesecurity/trufflehog/blob/def734a783b2d4542f2b353d326130d618f3c421/pkg/detectors/bombbomb/bombbomb.go#L24
https://github.com/trufflesecurity/trufflehog/blob/def734a783b2d4542f2b353d326130d618f3c421/pkg/detectors/bombbomb/bombbomb_test.go#L15
Kraken
The "valid" pattern is nonsensical and not correct base64 encoding. The detector should not match this, that is a defect.
https://github.com/trufflesecurity/trufflehog/blob/def734a783b2d4542f2b353d326130d618f3c421/pkg/detectors/kraken/kraken_test.go#L16
https://support.kraken.com/hc/en-us/articles/360000919966-How-to-create-an-API-key
viewneo
A few hundred detectors contain tests tightly coupled to the current implementation of PrefixRegex. Any changes to the prefix pattern will break the detector tests, which seems inadvisable.
https://github.com/trufflesecurity/trufflehog/blob/def734a783b2d4542f2b353d326130d618f3c421/pkg/detectors/viewneo/viewneo_test.go#L38-L42