gitleaks icon indicating copy to clipboard operation
gitleaks copied to clipboard

Add URL & JSON sources

Open bplaxco opened this issue 1 month ago • 3 comments

Add support for scanning URLs directly and JSON data.

Description:

  • If the URL returns an application/json type then treat the response as JSON data to scan.
  • Support scanning URLs included in the JSON data where:
    • The full value of the field is the URL
    • AND the path to the URL is in the FetchURLPatterns list.

FetchURLPatterns

This is a list of glob and recursive glob (i.e. **) patterns that can be passed in when fetching content

Paths and Path Separators

The path returned for items in a JSON scan look like file paths:

For example, given:

{
   "some": [{}, {"target": "secret"}]
}

The path to "secret" would be:

some/1/target

If a URL returns JSON or a URL is fetched from JSON the ! inner path separator is used to indicate it crossed the bound from one source to another.

Extra

I also added an import target to the Makefile to run goimports to auto import and tidy in the project as needed to make it easier for folks with really minimal editor setups ( :raised_hand_with_fingers_splayed: ) ^_^

To-do

  • [x] Add the sources
  • [x] Add json and url commands
  • [ ] Test/discuss changes (WIP)
  • [ ] Update docs

Checklist:

  • [x] Does your PR pass tests?
  • [x] Have you written new tests for your changes?
  • [x] Have you lint your code locally prior to submission?

bplaxco avatar Nov 19 '25 17:11 bplaxco