azure-webjobs-sdk icon indicating copy to clipboard operation
azure-webjobs-sdk copied to clipboard

WebJobsTelemetryInitializer causes throttling of Application Insights custom metrics

Open olegsych opened this issue 3 years ago • 7 comments

Default instrumentation of Azure Functions produces metrics with unbounded cardinality, which leads to throttling of custom metrics in Azure Monitor.

Repro steps

Enable integration with Application Insights and add the following code to your function.

var telemetry = new MetricTelemetry("MyNamespace, "MyMetric", count, sum, min, max);
telemetryClient.TrackMetric(telemetry);

Expected behavior

MyMetric is tracked without high-cardinality custom dimensions.

Actual behavior

MyMetric is tracked with the InvocationId dimension, an unbounded cardinality value, which at high enough volume get's the custom metrics throttled for the entire subscription.

Known workarounds

Override default functions configuration?

Related information

https://docs.microsoft.com/en-us/azure/azure-monitor/essentials/metrics-custom-overview#quotas-and-limits

  • Package version (latest used by functions)
  • Links to source (n/a)

olegsych avatar Mar 03 '21 17:03 olegsych

Hi @brettsam, Could you please suggest the repo to transfer this issue

v-bbalaiagar avatar Apr 01 '21 17:04 v-bbalaiagar

@v-bbalaiagar -- this is the correct repo.

brettsam avatar Apr 08 '21 13:04 brettsam

Hi @brettsam, Can you suggest how to further proceed to triage this issue. Do we require more information for further investigation

v-bbalaiagar avatar Apr 14 '21 08:04 v-bbalaiagar

Hi @olegsych, Can you please give us more information regarding the scenario to reproduce the issue.

v-bbalaiagar avatar May 07 '21 14:05 v-bbalaiagar

@v-bbalaiagar I have provided this information in the issue. Do you have specific questions?

olegsych avatar May 07 '21 21:05 olegsych

Hi @brettsam, Could you please look into this issue.

v-bbalaiagar avatar Jun 11 '21 17:06 v-bbalaiagar

Any progress on this issue?

nilsgaeq avatar Mar 01 '22 08:03 nilsgaeq