GoogleWorkspaceReports - getting API error, Quota exceeded for quota metric

[dmkolfl]

Hello!

I'm getting the error below for my FunctionApp. The API quota in GCP is standard - 2600 per minute. All settings for the FunctionApp are by default.

Google does not accept my request to increase the quota.

Something wrong while getting the results. Exception error text: <HttpError 403 when requesting https://admin.googleapis.com/admin/reports/v1/activity/users/all/applications/access_transparency?maxResults=1000&startTime=2024-03-11T05%3A20%3A00.000Z&endTime=2024-03-11T05%3A23%3A00.000Z&alt=json returned 'Quota exceeded for quota metric Queries and limit Queries per minute per user of service admin.googleapis.com for consumer project_number:227460265128.'. Details: '[{message: 'Quota exceeded for quota metric Queries and limit Queries per minute per user of service admin.googleapis.com for consumer project_number:227460265128.', domain: usageLimits, reason: rateLimitExceeded}]'>

Why does it request the logs from March 2024 if the current date is April 25, 2024?

[ingest0x]

We have started to get this error also suddenly

[ingest0x]

@dmkolfl do you have a significant log-gap starting around 05:20 on March 11th? In our experience, we had an issue arise on April 16th related to the account that generated the Pickle string lost it's privileges and we believe this caused log ingestion to stop working. We did not realise it until our Azure AppInsights quota limit approached 100% (the function app was producing a large amount of exception log data). Upon investigation we found the Function Apps were exceeding hourly Google API limits. We then realised it all started on April 16th. It appears the Function Apps are trying to back-fill the logging gap but is exceeding API quotas doing so. It seems like the function app does not have the ability to back-off when receiving an API limit excess response

[dmkolfl]

I verified and our account still has required permissions but at first time we got the error about the quota first and then we were flooded by logs for AppInsights.

[dmkolfl]

Today I started the app again today, and it did more than 21k requests in a minute and, of course, it started failing because it reached the quota. Why does it generate so many requests?

[v-sudkharat]

Hi @dmkolfl, Thanks for flagging this issue, we will investigate this issue and get back to you with some updates by 03-05-2024. Thanks!

[sc-roberts]

Keen to hear on this as we ran into the same problem and overnight it logged over 100GB or data into Sentinel through App traces and metrics, and $50 charge for the function app.

[dmkolfl]

Hi @dmkolfl, Thanks for flagging this issue, we will investigate this issue and get back to you with some updates by 03-05-2024. Thanks!

Do you have any updates about this topic?

[v-sudkharat]

Hi @ingest0x, We are still checking on this issue, and need more time investigate this issue. We will update you once repro done from our end. Thanks!

[v-sudkharat]

Hi @dmkolfl, Could you please check the function app by updating the WEBSITE_RUN_FROM_PACKAGE with below shared URL in the function app. and let us know the response.

Link :- https://github.com/Azure/Azure-Sentinel/raw/v-mchatla/GoogleWorkspace-HandlingAPIRateLimit/Solutions/GoogleWorkspaceReports/Data%20Connectors/GWorkspaceReportsAPISentinelConn.zip

And once it gets updated, please restart the function app.

Thanks!

[dmkolfl]

Hi @v-sudkharat I replaced the link for this setting and am monitoring the status. Give me 2-3 days to be able to verify it.

[v-sudkharat]

@dmkolfl, Sure. Please let us know once done

[ingest0x]

Hi @dmkolfl, Could you please check the function app by updating the WEBSITE_RUN_FROM_PACKAGE with below shared URL in the function app. and let us know the response.

Link :- https://github.com/Azure/Azure-Sentinel/raw/v-mchatla/GoogleWorkspace-HandlingAPIRateLimit/Solutions/GoogleWorkspaceReports/Data%20Connectors/GWorkspaceReportsAPISentinelConn.zip

And once it gets updated, please restart the function app.

Thanks!

We are testing this now

[v-sudkharat]

@ingest0x, Noted. Please let us know the result. Thanks!

[v-sudkharat]

Hey @dmkolfl, Are we good with the connector? waiting for your response.

[dmkolfl]

I think we are good for now. Thank you for your support

[v-sudkharat]

@dmkolfl, Thanks for the confirmation. So, closing this issue. If you still need support for this issue, feel free to re-open it any time. Thank you for your co-operation.