Add generic JWT detection and verification

[bradlarsen]

Description:

This adds a generic detector and verifier for generic JWTs.

Detection is simple using regular expressions. However, this will produce many false positives. JWT verification is performed using the github.com/golang-jwt/jwt/v5 package, performing the usual checks (proper decoding, checking timestamp-related claims, checking for an allow-list of supported algorithms, etc). Only public key cryptography algorithms are supported. Additionally, OIDC Discovery is attempted against the issuer to fetch the public key for signature verification.

Testing:

A few unit tests demonstrate the detector working. Further testing of verification is needed.

TODO:

• [ ] thoroughly test verification
• [ ] add integration tests
• [ ] think about and write up the security implications of using OIDC Discovery with an attacker-controlled URL
• [ ] estimate the impact on increased finding volume when verification is disabled

[bradlarsen]

Note: the custom detector test failed in the test-community job above:

--- FAIL: TestDetectorValidations (0.00s)
    --- FAIL: TestDetectorValidations/custom_validation_-_multiple_regex_validations (0.00s)
        custom_detectors_test.go:556: CustomDetector.FromData() custom validation - multiple regex validations diff: (-got +want)
              []detectors.Result{
              	{
              		... // 2 identical fields
              		Verified:              false,
              		VerificationFromCache: false,
              		Raw: bytes.Join({
            + 			"MyStrongP@ssword",
              			"c392c9837d69b44c764cbf260b-e6184",
            - 			"MyStrongP@ssword",
              		}, ""),
              		RawV2:    nil,
              		Redacted: "",
              		... // 3 ignored and 2 identical fields
              	},
              }
FAIL
FAIL	github.com/trufflesecurity/trufflehog/v3/pkg/custom_detectors	0.043s

I have seen this sporadic test failure a few times now. I'm pretty sure it's caused by iterating over a map (the order of which is unspecified in Go) in the custom detector code:

https://github.com/bradlarsen/trufflehog/blob/1e8671c6de6d721636e672b3066c6773e4ab6d0b/pkg/custom_detectors/custom_detectors.go#L118

[bradlarsen]

I have seen this sporadic test failure a few times now. I'm pretty sure it's caused by iterating over a map (the order of which is unspecified in Go) in the custom detector code

Fixed and merged in #4446.

[bradlarsen]

A TODO item from the description:

estimate the impact on increased finding volume when verification is disabled

I ran with this new JWT detection over 250k recent commits from GitHub. From there, about 4k distinct JWTs were found, and several dozen of them verified completely.

The newly detected JWTs will be notable, but less volume than, say, Github V1.

[joeleonjr]

@bradlarsen given clarification on the unknown status in #4477, I think usage of that status in the JWT detector should be removed. I think it's worth just commenting out those code paths (instead of fully deleting), since they would be useful in the future if we introduce any additional statuses. Your call. And if you have any other ideas here, I'm sure the team would be open to discussing.

[bradlarsen]

I ran over ~200GB of recently-pushed content to public GitHub with this branch. In that experiment, the new JWT detector is not even in the top 10 of all detectors for finding volume, with 777 unverified, 1 verified (!), and 1 unknown finding.

For comparison, the most voluminous detector was GitHub OAuth2, with 6961 unverified and 3 verified findings.

It's unlikely that this new detector is going to blow up TruffleHog's output.