trufflehog icon indicating copy to clipboard operation
trufflehog copied to clipboard

Fix broken & inaccurate detector unit tests

Open rgmz opened this issue 10 months ago • 7 comments

The detector unit tests were created by "reverse engineering" the patterns, thus they do not provide any real value or confirm how accurate the detectors are.

Additionally, the structure of the tests are (in my opinion) not maintainable as they are difficult to understand and do not reflect any realistic scenarios.

https://github.com/trufflesecurity/trufflehog/blob/def734a783b2d4542f2b353d326130d618f3c421/pkg/detectors/netsuite/netsuite_test.go#L14-L44

Examples

There are dozens, if not hundreds, of problematic test files. These are illustrative.

BombBomb

The "valid" tests for BombBomb do not match the detector's pattern.

https://github.com/trufflesecurity/trufflehog/blob/def734a783b2d4542f2b353d326130d618f3c421/pkg/detectors/bombbomb/bombbomb.go#L24

https://github.com/trufflesecurity/trufflehog/blob/def734a783b2d4542f2b353d326130d618f3c421/pkg/detectors/bombbomb/bombbomb_test.go#L15

Kraken

The "valid" pattern is nonsensical and not correct base64 encoding. The detector should not match this, that is a defect.

https://github.com/trufflesecurity/trufflehog/blob/def734a783b2d4542f2b353d326130d618f3c421/pkg/detectors/kraken/kraken_test.go#L16

image https://support.kraken.com/hc/en-us/articles/360000919966-How-to-create-an-API-key

viewneo

A few hundred detectors contain tests tightly coupled to the current implementation of PrefixRegex. Any changes to the prefix pattern will break the detector tests, which seems inadvisable.

https://github.com/trufflesecurity/trufflehog/blob/def734a783b2d4542f2b353d326130d618f3c421/pkg/detectors/viewneo/viewneo_test.go#L38-L42

rgmz avatar Dec 26 '24 16:12 rgmz