Timer for http.server.connection.lifetime grows larger than default

[DieBauer]

In my application I'm using Kamon to have metrics for my akka http server. Now my application warns a few times per minute the following

Failed to record value [5941012601004] on [http.server.connection.lifetime,{port=8080,component=akka.http.server,interface=myserver}] because the value is outside of the configured range. The recorded value was adjusted to the highest trackable value [3600000000000]. You might need to change your dynamic range configuration for this metric

I see that I can change the https://github.com/kamon-io/Kamon/blob/master/kamon-core/src/main/resources/reference.conf#L104 for all timers in kamon. Or I can also change kamon.metric.factory.custom-settings { "http.server.connection.lifetime" { highest-trackable-value = ... } }

However, I'd rather see the connection lifetime be reported in seconds, or millis, rather than nanos.

[alexmnyc]

Same here

[pnerg]

A vote from me too

[nikhilaroratgo]

May I ask why do the value exceeds the max value at the very first place ?

[Dichotomia]

I have the same problem here on multiple applications.

[antoine-morisseau]

We encounter the same issues in our play app with some values that goes up to 47 hours. Do we know what this metric http.server.connection.lifetime is attached to ?

I searched in Kamon repository and found this : https://github.com/kamon-io/Kamon/blob/master/instrumentation/kamon-instrumentation-common/src/main/scala/kamon/instrumentation/http/HttpServerMetrics.scala#L52

"Tracks the time elapsed between connection creation and connection close"

But what is a connection creation and close for akka http ? the creation and destruction of an instance of akka-http server ? But it can be open more than one hour up to several hours if there is always calls on the server that need the akka http connection to be open.

Does @SimunKaracic or @ivantopo can give use some clues about what this metric is for within an akka http instrumentation ?

[Dichotomia]

We encounter the same issues in our play app with some values that goes up to 47 hours. Do we know what this metric http.server.connection.lifetime is attached to ?

I searched in Kamon repository and found this : https://github.com/kamon-io/Kamon/blob/master/instrumentation/kamon-instrumentation-common/src/main/scala/kamon/instrumentation/http/HttpServerMetrics.scala#L52

"Tracks the time elapsed between connection creation and connection close"

But what is a connection creation and close for akka http ? the creation and destruction of an instance of akka-http server ? But it can be open more than one hour up to several hours if there is always calls on the server that need the akka http connection to be open.

Does @SimunKaracic or @ivantopo can give use some clues about what this metric is for within an akka http instrumentation ?

I think it had been fixed in Kamon 2.5.9 by this PR https://github.com/kamon-io/Kamon/pull/1204

[antoine-morisseau]

oh yes ! I'll try that and post an update !

[ivantopo]

Thanks for the update @Dichotomia! Indeed, it should be roughly fixed since Kamon Telemetry v2.5.9. I say "roughly" because we increased the default settings for the metric to a week, but if your connections are open for longer than that you might have the same issue again. Very unlikely to happen, though.

Going back to @antoine-morisseau's question here:

But what is a connection creation and close for akka http ? the creation and destruction of an instance of akka-http server ? But it can be open more than one hour up to several hours if there is always calls on the server that need the akka http connection to be open.

It refers to open and closing of a HTTP connection to the HTTP server. It gets paired with the http.server.connection.usage metric that tracks the distribution of how many requests were handled by a single connection. The idea here is that we should be able to detect whether connections are being opened/closed all the time by seeing small connection lifetimes and always 1 request per connection and be able to optimize that, since you will most likely want to keep connections open from your load balancer to your application to minimize latency.

Hope this comment helps clarify the question. Cheers!