AWS secret scanning with or without --max-depth 1 appears broken

[dinvlad]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

TruffleHog Version

trufflehog dev (freshly installed from Git)

Trace Output

https://gist.github.com/dinvlad/892fa1e435f38fbcc58a936084e1d7ca

Expected Behavior

AWS Secret should be detected both with and without --max-depth 1 option.

Actual Behavior

AWS Secret is not detected at all.

Steps to Reproduce

1. trufflehog git https://github.com/dinvlad/test-secrets-detection.git --trace --no-verification
2. trufflehog git https://github.com/dinvlad/test-secrets-detection.git --trace --no-verification --max-depth 1

Environment

• OS: Ubuntu (WSL)
• Version: 22.04 LTS

Additional Context

The AWS secret in question appears in the resolved URLs of GitHub Asset downloads. I've committed it several times, including in a clean final commit, with the following diff in this repo: https://github.com/dinvlad/test-secrets-detection.git

--- a/README.md
+++ b/README.md
@@ -3,2 +3 @@
-this is a test using a known AWS secret
-
+this is a test using a known publicized AWS secret
@@ -5,0 +5 @@ this is a test using a known AWS secret
+AKIAIWNJYAX4CSVEH53A

Hence, I was hoping the secret would be detected with --max-depth 1 option (from a shallow clone), but it's not detected even from the full history.

References

[dustin-decker]

It appears that there is only a key id. Both a key id and a secret value are required for AWS credentials.

[dinvlad]

Unfortunately, it doesn't work even when specifying both id and secret:

trufflehog git https://github.com/dinvlad/test-secrets-detection --branch test2

(this is a real but deactivated/deleted pair of credentials btw)

[dinvlad]

FWIW, when I run this latest example with a filesystem scan locally, it does detect them. But not so with Git: https://gist.github.com/dinvlad/892fa1e435f38fbcc58a936084e1d7ca#file-filesystem_vs_git_local-txt

[dinvlad]

Any updates on this?

[dustin-decker]

I am still not seeing the secret value in the repository, but you're probably seeing a limitation where the key id and the secret were added in different commits. Right now each diff is scanned individually. It's not common in the wild for parts of a credential to be commited separatedly.

[dinvlad]

They have actually been added in a single diff, please take a look here: https://github.com/dinvlad/test-secrets-detection/commit/c3c247d10eef0bef6a28c981bd4da64529ed816a

Please note this is on branch test2, which is why maybe it wasn't showing up for you. So to reproduce this, please use

trufflehog git https://github.com/dinvlad/test-secrets-detection.git --trace --no-verification --since-commit e96517f0a8f29ef88d9b9c21ecb0d1d37bc90756

or

trufflehog git https://github.com/dinvlad/test-secrets-detection.git --trace --no-verification --max-depth 1

[bill-rich]

@dinvlad Thank you for all the help in troubleshooting this. I've tracked down what is causing the secrets in c3c247d to be missed. It is an issue with the context around changes and how those are handled. I'm testing a fix now.

[dinvlad]

@bill-rich awesome, thanks so much for troubleshooting it!