trufflehog icon indicating copy to clipboard operation
trufflehog copied to clipboard

Published 20 hours ago • verified publisher icon trufflesecurity

Docker Image Identification in Tar Files

Open joeleonjr opened this issue 2 years ago • 7 comments

Description:

The scanner will currently untar a file and look through the archive for secrets; however, this approach misses secrets hidden in Docker image layers. (The docker save command turns docker images into tar files.) This PR adds docker image scanning inside all archived files identified as the application/x-tar mimetype. A use-case for this feature would be scanning a filesystem, identifying tar files, and then checking if any of them are docker images.

Checklist:

  • [x] Tests passing (make test-community)?
  • [x] Lint passing (make lint this requires golangci-lint)?

joeleonjr avatar Aug 18 '23 20:08 joeleonjr

@ahrav I moved the integration test into a separate file (and made the updates you suggested), but it's throwing an error. Locally it worked great. This is what I'm seeing:

panic: test timed out after 5m0s
running tests:
	TestFoundKeyInDockerTar (5m0s)

goroutine 62 [running]:
testing.(*M).startAlarm.func1()
	/opt/hostedtoolcache/go/1.21.0/x64/src/testing/testing.go:2[259](https://github.com/trufflesecurity/trufflehog/actions/runs/5963890907/job/16177983237?pr=1643#step:5:260) +0x3b9
created by time.goFunc
	/opt/hostedtoolcache/go/1.21.0/x64/src/time/sleep.go:176 +0x2d

goroutine 1 [chan receive, 5 minutes]:
testing.(*T).Run(0xc00050d860, {0xeaba6e?, 0x52bdfc?}, 0xef8168)
	/opt/hostedtoolcache/go/1.21.0/x64/src/testing/testing.go:1649 +0x3c8
testing.runTests.func1(0x1798fe0?)
	/opt/hostedtoolcache/go/1.21.0/x64/src/testing/testing.go:2054 +0x3e
testing.tRunner(0xc00050d860, 0xc0005bfc48)
	/opt/hostedtoolcache/go/1.21.0/x64/src/testing/testing.go:1595 +0xff
testing.runTests(0xc000555180?, {0x176ed40, 0x6, 0x6}, {0xc?, 0xc0005bfd08?, 0x17983e0?})
	/opt/hostedtoolcache/go/1.21.0/x64/src/testing/testing.go:2052 +0x445
testing.(*M).Run(0xc000555180)
	/opt/hostedtoolcache/go/1.21.0/x64/src/testing/testing.go:1925 +0x636
main.main()
	_testmain.go:57 +0x19c

goroutine 20 [select]:
go.opencensus.io/stats/view.(*worker).start(0xc000506100)
	/home/runner/go/pkg/mod/[email protected]/stats/view/worker.go:292 +0x9f
created by go.opencensus.io/stats/view.init.0 in goroutine 1
	/home/runner/go/pkg/mod/[email protected]/stats/view/worker.go:34 +0x8d

goroutine 8 [running]:
	goroutine running on other thread; stack unavailable
created by testing.(*T).Run in goroutine 1
	/opt/hostedtoolcache/go/1.21.0/x64/src/testing/testing.go:1648 +0x3ad

goroutine 40 [select]:
github.com/trufflesecurity/trufflehog/v3/pkg/sources/docker.ScanDockerImg({0xfef4b0, 0xc000879d70}, {0xff3c78, 0xc000634040}, 0xc000133380, 0xc00007e370)
	/home/runner/work/trufflehog/trufflehog/pkg/sources/docker/docker.go:217 +0xdc5
github.com/trufflesecurity/trufflehog/v3/pkg/handlers.HandleFile({0xfef4b0?, 0xc000879d70}, {0xfe06c0, 0xc000879da0}, 0x0?, 0x0?)
	/home/runner/work/trufflehog/trufflehog/pkg/handlers/handlers.go:65 +0x6e5
github.com/trufflesecurity/trufflehog/v3/pkg/handlers.TestFoundKeyInDockerTar.func1()
	/home/runner/work/trufflehog/trufflehog/pkg/handlers/archive_integration_test.go:88 +0x65
created by github.com/trufflesecurity/trufflehog/v3/pkg/handlers.TestFoundKeyInDockerTar in goroutine 8
	/home/runner/work/trufflehog/trufflehog/pkg/handlers/archive_integration_test.go:86 +0x8d8
FAIL	github.com/trufflesecurity/trufflehog/v3/pkg/handlers	[300](https://github.com/trufflesecurity/trufflehog/actions/runs/5963890907/job/16177983237?pr=1643#step:5:301).061s

joeleonjr avatar Aug 24 '23 13:08 joeleonjr

@ahrav I moved the integration test into a separate file (and made the updates you suggested), but it's throwing an error. Locally it worked great. This is what I'm seeing:

panic: test timed out after 5m0s
running tests:
	TestFoundKeyInDockerTar (5m0s)

goroutine 62 [running]:
testing.(*M).startAlarm.func1()
	/opt/hostedtoolcache/go/1.21.0/x64/src/testing/testing.go:2[259](https://github.com/trufflesecurity/trufflehog/actions/runs/5963890907/job/16177983237?pr=1643#step:5:260) +0x3b9
created by time.goFunc
	/opt/hostedtoolcache/go/1.21.0/x64/src/time/sleep.go:176 +0x2d

goroutine 1 [chan receive, 5 minutes]:
testing.(*T).Run(0xc00050d860, {0xeaba6e?, 0x52bdfc?}, 0xef8168)
	/opt/hostedtoolcache/go/1.21.0/x64/src/testing/testing.go:1649 +0x3c8
testing.runTests.func1(0x1798fe0?)
	/opt/hostedtoolcache/go/1.21.0/x64/src/testing/testing.go:2054 +0x3e
testing.tRunner(0xc00050d860, 0xc0005bfc48)
	/opt/hostedtoolcache/go/1.21.0/x64/src/testing/testing.go:1595 +0xff
testing.runTests(0xc000555180?, {0x176ed40, 0x6, 0x6}, {0xc?, 0xc0005bfd08?, 0x17983e0?})
	/opt/hostedtoolcache/go/1.21.0/x64/src/testing/testing.go:2052 +0x445
testing.(*M).Run(0xc000555180)
	/opt/hostedtoolcache/go/1.21.0/x64/src/testing/testing.go:1925 +0x636
main.main()
	_testmain.go:57 +0x19c

goroutine 20 [select]:
go.opencensus.io/stats/view.(*worker).start(0xc000506100)
	/home/runner/go/pkg/mod/[email protected]/stats/view/worker.go:292 +0x9f
created by go.opencensus.io/stats/view.init.0 in goroutine 1
	/home/runner/go/pkg/mod/[email protected]/stats/view/worker.go:34 +0x8d

goroutine 8 [running]:
	goroutine running on other thread; stack unavailable
created by testing.(*T).Run in goroutine 1
	/opt/hostedtoolcache/go/1.21.0/x64/src/testing/testing.go:1648 +0x3ad

goroutine 40 [select]:
github.com/trufflesecurity/trufflehog/v3/pkg/sources/docker.ScanDockerImg({0xfef4b0, 0xc000879d70}, {0xff3c78, 0xc000634040}, 0xc000133380, 0xc00007e370)
	/home/runner/work/trufflehog/trufflehog/pkg/sources/docker/docker.go:217 +0xdc5
github.com/trufflesecurity/trufflehog/v3/pkg/handlers.HandleFile({0xfef4b0?, 0xc000879d70}, {0xfe06c0, 0xc000879da0}, 0x0?, 0x0?)
	/home/runner/work/trufflehog/trufflehog/pkg/handlers/handlers.go:65 +0x6e5
github.com/trufflesecurity/trufflehog/v3/pkg/handlers.TestFoundKeyInDockerTar.func1()
	/home/runner/work/trufflehog/trufflehog/pkg/handlers/archive_integration_test.go:88 +0x65
created by github.com/trufflesecurity/trufflehog/v3/pkg/handlers.TestFoundKeyInDockerTar in goroutine 8
	/home/runner/work/trufflehog/trufflehog/pkg/handlers/archive_integration_test.go:86 +0x8d8
FAIL	github.com/trufflesecurity/trufflehog/v3/pkg/handlers	[300](https://github.com/trufflesecurity/trufflehog/actions/runs/5963890907/job/16177983237?pr=1643#step:5:301).061s

Oh i wonder if the runner timed out after the 5 min interval. We can try to increase the timeout from 5m to test.

ahrav avatar Aug 24 '23 14:08 ahrav

@joeleonjr would this solve https://github.com/trufflesecurity/trufflehog/issues/674 ?

zricethezav avatar Aug 28 '23 16:08 zricethezav

@joeleonjr would this solve #674 ?

Good catch. Yes, this closes #674

joeleonjr avatar Aug 28 '23 17:08 joeleonjr

CLA assistant check
All committers have signed the CLA.

CLAassistant avatar Aug 17 '24 04:08 CLAassistant

Is there a specific reason why this has not been merged after being approved for more than 1,5 years?

john-funk avatar Apr 04 '25 17:04 john-funk

Is there a specific reason why this has not been merged after being approved for more than 1,5 years?

+1

kashifkhan0771 avatar Aug 06 '25 10:08 kashifkhan0771