dftimewolf icon indicating copy to clipboard operation
dftimewolf copied to clipboard

gcp_logging_collect recipe fails silently when it hits Cloud Logging API limits

Open bskeggs opened this issue 1 year ago • 0 comments

When running the gcp_logging_collect recipe, the collection will fail silently if it exceeds the cloud logging API quota limit.

Based on my testing, this will first issue a 'google_api_exceptions.TooManyRequests' exception. Following this, all subsequent queries from the same logging client will be returned with a 'google_api_exceptions.StopIteration' exception. This occurs even when I implemented a 2 minute delay between requests after the first 'TooManyRequests' exception.

Due to this, I believe the previous logging retries fix from https://github.com/log2timeline/dftimewolf/commit/32e39780d808cc506c19580aea68e3b1c1810ede may not achieve any successful retries, as a new logging client is not generated after hitting the API limit.

bskeggs avatar Dec 02 '22 08:12 bskeggs