azure-docs
azure-docs copied to clipboard
MicrosoftDocs
Azure Monitor - InsightsMetrics - populate custom tag, or pod name
Azure Monitor - InsightsMetrics
Hello,
We need some way to filter metrics scraped from our services in Azure Kubernetes that are injected into InsightsMetrics. It seems that labels/annotations from pods are droped, we did not find a way to populate pod name with metrics...
How should we plot a graph of requests per second per pod, in Azure Monitor, if we cannot filter out by a pod?
https://learn.microsoft.com/en-us/azure/azure-monitor/essentials/prometheus-metrics-scrape-configuration https://github.com/Azure/iotedge/issues/6141
Example record from InsightsMetrics
12/3/2022, 10:32:00.000 AM aks-default-12500777-vmss000000 container.azm.ms/telegraf prometheus keycloak_response_errors 3 {"code":"500","container.azm.ms/clusterId":"/subscriptions/xxxxxx-5fa9-493e-9256-xxxxxx/resourceGroups/RG-dev/providers/Microsoft.ContainerService/managedClusters/aks-dev","container.azm.ms/clusterName":"aks-dev","hostName":"aks-default-12500777-vmss000000","method":"GET","resource":"realms,realms/center/protocol/openid-connect","scrapeUrl":"https://identity.dev.net/realms/center/metrics"} xxxxxx-9e53-4d70-9df1-xxxxxxxxxx
Document Details
⚠ Do not edit this section. It is required for learn.microsoft.com ➟ GitHub issue linking.
- ID: 9f2f9b8e-6345-7fdc-602f-a728b446e3e9
- Version Independent ID: 16bc2a0f-b03e-b411-ff0c-def649ad463d
- Content: Customize scraping of Prometheus metrics in Azure Monitor (preview) - Azure Monitor
- Content Source: articles/azure-monitor/essentials/prometheus-metrics-scrape-configuration.md
- Service: azure-monitor
- Sub-service: essentials
- GitHub Login: @EdB-MSFT
- Microsoft Alias: edbaynash
@sirkubax Thanks for your feedback! We will investigate and update as appropriate.
as I get it, Insights Metrics have limited Fields by default, I think a Computer 'field' could be a place to inject POD or container, or it could be placed into Tags
https://github.com/MicrosoftDocs/azure-reference-other/blob/main/azure-monitor-ref/tables/insightsmetrics.md
Maybe it is possible to write prometheus scrapper to inject such labels?
eh found it
colleague definend URL scraping in Configmap in the section of prometheus_data_collection_settings.node
so just use pod annotations and it would be picked up by prometheus_data_collection_settings.cluster