nccgroup
GCP scan does not cache requests to Service Usage API
Describe the bug
When scanning GCP organization with approx. 40 projects, I received 429 status responses from the Service Usage API (mentioned in #940) . I observe a high number of requests to that API in the GCP console.
To Reproduce
Run a scan on a GCP project, with the default settings. Multiple calls to Service Usage API will be recorded in the GCP console.
Additional context
In "is_api_enabled", force_new=true is passed when creating the service usage client: https://github.com/nccgroup/ScoutSuite/blob/4aaf8eaf4acb059be4d1918c14d4027203a51baf/ScoutSuite/providers/gcp/facade/base.py#L133
This has the effect of disabling caching. Better might be to move the Service Usage client into its own Facade.
The same likely applies to the Resource Manager v2 call further up in the same file, though this is only called once per resource type, and has a higher default quota: https://github.com/nccgroup/ScoutSuite/blob/4aaf8eaf4acb059be4d1918c14d4027203a51baf/ScoutSuite/providers/gcp/facade/base.py#L92
I'm not able to find a closed issue relating to this, but I believe this was by design, as caching was causing some issues. Have you tried running this with force_new=False?
When scanning GCP organization with approx. 40 projects, I received 429 status responses from the Service Usage API (mentioned in #940) . I observe a high number of requests to that API in the GCP console.
I fail to see how this is related to the caching?
I'm not able to find a closed issue relating to this, but I believe this was by design, as caching was causing some issues. Have you tried running this with
force_new=False?
Not yet. (I haven't yet understood which object the "_client_name" attribute would get set on - I get the impression that I would also need to move the call into a separate Facade class in order for the caching not to clash with anything?)
FWIW it looks to me like force_new=True has been there since the line was introduced: https://github.com/nccgroup/ScoutSuite/commit/1a4ee10bb25466bcf449b808f08a34659ad2def8#diff-28dc468de8be8bdcc5a14706c2c2c62c45fc1ce9e9a224cd9ea3647c36c92a60R138
I fail to see how this is related to the caching?
Because projects.py is subclassed once per service, and because it then calls 'is_api_enabled' on every project, I believe the API call is being repeated once per service per project.
In my case, with ~40 projects and ~8 services to check, I've leapt to the assumption that this is making ~320 calls to the API at the start of the scan, and exceeding the 240 requests per minute limit - therefore explaining the errors I see and the high usage.
Ideally, I would expect to see ~40 calls to the Service Usage API per scan, which would be less than the default limit.
Because projects.py is subclassed once per service, and because it then calls 'is_api_enabled' on every project, I believe the API call is being repeated once per service per project.
Right I get what you mean. It's not related to the caching, but as you're saying we'd really want to call it once per project (moving it to the base facade as you mentioned would allow this).