Lot of ResourceRequestsThrottled in scraper on metricDefinitions/read and not actual retrieval of values

[ahurtaud]

Discussed in https://github.com/tomkerkhove/promitor/discussions/1886

^{Originally posted by ahurtaud December 14, 2021} Hello,

On some big subscriptions we are seeing in promitor scrapers a lot of:

[15:57:55 FTL] ResourceRequestsThrottled: Number of requests for action 'microsoft.insights/metricDefinitions/read' exceeded the limit of '400' for time interval '00:05:00'. Please try again after '309' seconds.
[15:57:55 FTL] ResourceRequestsThrottled: Number of requests for action 'microsoft.insights/metricDefinitions/read' exceeded the limit of '400' for time interval '00:05:00'. Please try again after '309' seconds.
[15:57:55 FTL] ResourceRequestsThrottled: Number of requests for action 'microsoft.insights/metricDefinitions/read' exceeded the limit of '400' for time interval '00:05:00'. Please try again after '309' seconds.

What this microsoft.insights/metricDefinitions/read is used for? and how could we reduce it if possible?

Thank you

[tomkerkhove]

This action is for getting the available metric definitions for a given resource so I'll see what I can do.

[ahurtaud]

it looks like it is doing it for every resource instances and not "per resource type", to be confirmed.

If its the case, maybe we can do it only once per type (e.g. NetworkInterface)? Otherwise, maybe try to get metrics as per the configuration, and if it does not exists, Azure API will shout 4xx? and we can log this as well?

WDYT?

[tomkerkhove]

Those are valuable options; the tricky thing however is that it often depends on the SKU for a given resource as well but I'll investigate it asap.

[tomkerkhove]

Potential workaround would be to partition the amount of resources across multiple deployments.

It's not nice, but it should work.

[tomkerkhove]

Do you still see this in 2.7.x @ahurtaud?

[ahurtaud]

Do you still see this in 2.7.x @ahurtaud?

This is quite difficult to answer @tomkerkhove :/. Here is the status:

• This still happen on scraper 2.6.0.

• However if I load scraper 2.6.1, it seems to not work with resource-discovery 0.7.0 and I cant load 0.7.1 because of https://github.com/tomkerkhove/promitor/issues/2018 I see some 404 on discovery with exact same config, I believe because of the pagination (?currentPage=1):

[09:06:47 INF] Request starting HTTP/1.1 GET http://promitor-agent-resource-discovery-prd.argos-external.svc:8080/api/v2/resources/groups/storage-account-landscape/discover?currentPage=1 - -
[09:06:47 INF] Request finished HTTP/1.1 GET http://promitor-agent-resource-discovery-prd.argos-external.svc:8080/api/v2/resources/groups/storage-account-landscape/discover?currentPage=1 - - - 404 0 - 0.0967ms

• AFAIK there no scraper 2.7.x released ?

[tomkerkhove]

Yes, my bad - I was confusing it with KEDA v2.7 which I co-maintain as well :)

Ok then I will have a look at it, sorry for the dela