Scan GitHub and GitLab refs that aren't pulled by default

[rgmz]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Description

This article featured on HackerNews reminded me that both GitHub and GitLab have refs for (merge|pull) requests that — to my knowledge — aren't pulled by default. This is especially juicy for TruffleHog because PRs may contain commits or alternate history with valid secrets that technically no longer exist in the repository.

GitHub

$  git clone https://github.com/trufflesecurity/trufflehog.git
$ cd trufflehog
$ git ls-remote
From https://github.com/trufflesecurity/trufflehog.git
69021f59c57b6a49bafcc8d1827e355405330014        HEAD
aa3ba817175e1bcdf852f11ffa327c1d134ecb26        refs/heads/0x1/add-on-prem-verification-flag
ade5d91d5ca94e996377b1909a802dd9dbc51b6b        refs/heads/1560-error-during-git-scan-using-pre-commit
...
c3644ccaf1bb0799ef919a5897b7737ebff428db        refs/pull/1/head
180b9c288bfc2b0f0543f91dfd74740f4f1d410e        refs/pull/100/head
c36defae7818b738389815dd32a64db398e2d833        refs/pull/100/merge
...
3048c6429d80094b42fd44919e3d6768536178ce        refs/tags/2.0.97
6d2dc5e965f9d5c23acea0c0c6377b06a8ae6c7c        refs/tags/v0.1.0-alpha
fda044631b344997a4556f52aadbd7c8275d0802        refs/tags/v3.0.0

GitLab

$ git clone [email protected]:pdftk-java/pdftk.git
$ cd pdftk
$ git ls-remote
From [email protected]:pdftk-java/pdftk.git
be326bcca502dcdcbfd6dc63f31976ea5aa7ffc9        HEAD
0d30918c940cae11e48aeb9c86857bb1455fd52a        refs/heads/ci
6c6fccca904b9234fa20e5c372e4573c0442156d        refs/heads/ci_native
...
53680180fef65bc6cd9fedc90b345ef354ab8c2c        refs/merge-requests/1/head
a86eca79300f9b0db0d2af2b6cb411168e8df16a        refs/merge-requests/12/head
696c8d99b25552123344bda752d3df6b8929adab        refs/merge-requests/12/merge
...
e80603a9130240019417eec30d7edad6cecba325        refs/tags/v3.0.0
4e20310b472b52cca12ad94252f90cff6cfebad6        refs/tags/v3.0.1

Others

• [ ] Bitbucket
• [ ] Gitea
• [ ] Gogs
• [ ] Sourcehut
• [ ] Forgejo
• [ ] ?

Problem to be Addressed

Find valid secrets that may be hidden in old PR histories.

Description of the Preferred Solution

Off the top of my head, I do not have any ideas on how to tackle this. It seems that you need to manually query for (git ls-remote) and checkout each ref in a detached state. The history will not show up via git log otherwise.

Scanning refs could result in the same commits being scanned twice. I am not sure if this would produce duplicate results, or if we already deduplicate results.

Additional Context

N/A

References

N/A

[rgmz]

This seems trivial to implement:

# assuming the origin is GitHub, which can be checked with `git ls-remote`
git fetch origin "+refs/pull/*:refs/heads/pull/*"

[0x736E]

can this behaviour be confirmed to be present in TruffleHog?