dotnet-monitor icon indicating copy to clipboard operation
dotnet-monitor copied to clipboard

Managed Prometheus scraping in Aks has timestamp error

Open wiktork opened this issue 1 year ago • 1 comments

  1. Enable Managed Prometheus integration in an Aks Cluster.
  2. Follow https://learn.microsoft.com/en-us/azure/azure-monitor/essentials/prometheus-metrics-scrape-configuration#configmaps to create a pod level scraper against dotnet-monitor's /metrics endpoint.
  3. Check the prometheus targets using https://learn.microsoft.com/en-us/azure/azure-monitor/essentials/prometheus-metrics-troubleshoot#prometheus-interface

Notice that the dotnet-monitor target has the following error:

  • inconsistent timestamps on metric points for metric

Note that the error repros with a 30s or 1m scraping interval and a 5-second metrics interval.

Workaround: Configure Metrics__Count to be 1.

wiktork avatar May 23 '23 05:05 wiktork

Having same issue. I had to set Metrics__Count to 1 and scrape interval to 5s

ajay-sainy avatar Jun 26 '23 15:06 ajay-sainy