trufflehog icon indicating copy to clipboard operation
trufflehog copied to clipboard

Fix nondeterminism in custom detectors

Open bradlarsen opened this issue 2 months ago • 3 comments

Previously, a custom detector configured to use multiple patterns would produce its results from a match in a nondeterministic order. This behavior could be seen by running the custom detectors test in a loop, e.g.,

$ cd pkg/custom_detectors
$ while true; do go test; done

This is likely to reveal sporadic test failures.

This nondeterminism was a result of producing results by iterating over a map, which has unspecified order in Go.

The fix: sort the intermediate map data structure prior to producing results.

bradlarsen avatar Sep 05 '25 14:09 bradlarsen

The failures were sporadically seen in GitHub Actions jobs as well. For example:

  • https://github.com/trufflesecurity/trufflehog/actions/runs/17478249783/job/49643183088
  • https://github.com/trufflesecurity/trufflehog/actions/runs/17459343592/job/49580167352
  • https://github.com/trufflesecurity/trufflehog/actions/runs/17459412883/job/49580376149
  • https://github.com/trufflesecurity/trufflehog/actions/runs/17466455637/job/49603364126

bradlarsen avatar Sep 05 '25 14:09 bradlarsen

Note, there is an earlier PR (#4438) to address the test flakiness here. The approach taken there is different: it explicitly allows the two permutations of the test, rather than changing the detector to eliminate the nondeterminism.

bradlarsen avatar Sep 05 '25 15:09 bradlarsen

Also, I'll point out here that the fix for the nondeterminism in this PR (eliminating it from the detector) will fix the test flakiness, making the order of the Raw result output consistent from run to run. However, the consistent order will still be rather unpredictable to users, as it's based on the matched content, and not based on something like the name of the custom regexes supplied.

bradlarsen avatar Sep 05 '25 15:09 bradlarsen