guac icon indicating copy to clipboard operation
guac copied to clipboard

[ingestion bug] File collector fails with "operation not permitted"

Open pxp928 opened this issue 1 year ago • 0 comments

Describe the bug Running the file collector on a large dataset (100,000+) causes it to fail with the following error:

{"level":"error","ts":1716911281.6153402,"caller":"cmd/files.go:162","msg":"collector ended with error: error walking path: <private>, err: error reading file: <private>, err: <private>: operation not permitted","guac-version":"v0.6.0","documentHash":"sha256:6d9cb98ae7d595ed48b4b2156000c521019972d1d9f8119467470c0ebd6782a1","stacktrace":"github.com/guacsec/guac/cmd/guaccollect/cmd.initializeNATsandCollector.func2\n\t/home/runner/work/guac/guac/cmd/guaccollect/cmd/files.go:162\ngithub.com/guacsec/guac/pkg/handler/collector.Collect\n\t/home/runner/work/guac/guac/pkg/handler/collector/collector.go:111\ngithub.com/guacsec/guac/cmd/guaccollect/cmd.initializeNATsandCollector.func3\n\t/home/runner/work/guac/guac/cmd/guaccollect/cmd/files.go:173"}

Running it single-threaded via:

find . -name "*.json" -exec guaccollect .... "{}" \;

does not cause this issue.

Expected behavior File collector should not fail when ingesting a large dataset

GUAC version v0.6.0

pxp928 avatar May 29 '24 12:05 pxp928