Metric counters don't report 0 to Prometheus

[risinger]

NTH does not report 0 when a metric has not been observed during runtime. Prometheus expects counters to report 0 when they are 0. If they are simply not reported, rate will report 0 when the metric is first reported, no matter the value. Because the first value is an observation, any alerting or visualization in Prometheus will either miss all errors/actions in the first scrape that has that metric or the alert/graphing needs to use the raw total number of observations during an NTH pod's lifetime.

I realize this may well be a bug in OpenTelemetry, but it also seems possible the counters are not properly initialized in NTH.

Here's a grafana graph to visualize the problem (increase is syntactic sugar for rate times the range selector):

[haugenj]

From reading around, it looks like ideally we would initialize these metrics at the beginning when we set up the metrics server. We can do that for the error metrics, but the node metrics are trickier because they use labels with the node name which we don't know ahead of time.

Perhaps we can init the error metrics early, and for the node metrics do a lookup to see if they're initialized before the first increment?

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. If you want this issue to never become stale, please ask a maintainer to apply the "stalebot-ignore" label.

[drstevens]

Bump here per @bwagner5's suggestion in slack

[stevehipwell]

@bwagner5 we're using the (rate(actions_node{job="aws-node-termination-handler",node_status!="success"}[5m])) query to trigger alerts and are getting our alerts replayed every 12 hours.

[stevehipwell]

@bwagner5 have you been able to look into this?

[cjerad]

This has been released in v1.20.0, chart version 0.22.0