HttpUrlConnectionSender opens a new connection on every send

[fkoehler]

After the metrics have been send with the http client the connection is disconnected which means according to javadoc it is ok to close the socket and that no further connections are done via it:

https://github.com/micrometer-metrics/micrometer/blob/99d151bf20d2e724f7f4cc11e326b745e71540dc/micrometer-core/src/main/java/io/micrometer/core/ipc/http/HttpUrlConnectionSender.java#L117-L119

Shouldn't we reuse the opened connection as long as possible to avoid constant new connections? But I am not fully sure how to implement this and how the internals of HttpUrlConnection function.

As far as I understand it keep-alive is enabled by default and connections will be reused because we fully consume the inputstreams. It might be enough to remove the conn.disconnect() at the end to reuse more of the connections?

[jm-g]

+1 Since, we have TLS handshakes all the time. Which is quiet expensive and adds latency.

[shakuzen]

I looked through the official documentation on Persistent Connections (I couldn't find the equivalent for a more recent version of Java than Java 8). I also verified the behavior via Wireshark on an Ubuntu VM running on my MacBook.

As far as I understand it keep-alive is enabled by default and connections will be reused because we fully consume the inputstreams.

This is true. The piece of information that I was missing and is inconveniently not mentioned in the documentation I linked above, is that the idle timeout for a persistent connection is 5 seconds. If a request to the same host is made within 5 seconds, the same underlying TCP connection will be reused.

That seems reasonable, and I don't know how we can customize it. MeterRegistrys using HttpUrlConnectionSender default to sending once every minute. At that time scale, I don't understand why the overhead of establishing a new connection is significant. Could anyone wanting this elaborate on that point?

Given the above, we may not be able to anything with the HttpUrlConnectionSender. I'd bet other senders are more configurable with the underlying HTTP client.

[shakuzen]

It might be enough to remove the conn.disconnect() at the end to reuse more of the connections?

I forgot to mention in my previous comment, I tried with an implementation that did not call disconnect(). I did not see any difference in behavior. Requests within 5 seconds reused the underlying connection, requests more than 5 seconds later established a new connection.

[karniemi]

At that time scale, I don't understand why the overhead of establishing a new connection is significant. Could anyone wanting this elaborate on that point?

because of micro-services(or alike), tens/hundreds/houtsands of them:

• This causes extra load on the server side, and connection setup latencies that exceed the 1s default connection setup/read timeout is met easily
• also this causes peak loads on client side, when a huge set of micro-services(or alike), running on the same virtual machine happen to start a new connection at the same time. This is more like minor issue, agree.
• also this brings up security borders of VMs and firewalls - which nowadays tend to block bursts of new connection setups
• ...and these are not made up, we are actually seeing frequent connection timeouts from micrometer towards influxb. Though, probably this could be prevented also by throwing more money(=resources) on our infra. Also, there seems to be a bug which leaks connections on the server side, which is supposed to be corrected on new versions of influxdb.

Basically, in the client side with one jvm opening a new connection every one minute - I truely agree with you, the overhead is not significant. But in larger scale, things add up :-).

[shakuzen]

I would be willing to update documentation or make a change so users can configure the idle timeout, but I don't know how to do that with HttpURLConnection. If someone knows, please do share or propose a change.

[shakuzen]

This is where the 5000 millisecond lifetime is hard-coded, so I don't think there is a way to configure it.

[karniemi]

In the new versions, in master, there seems to be an implemented feature which would allow configuring the value ( jvm-wide, if I had to guess): https://github.com/openjdk/jdk/commit/d8f44aa39e921594505864e6270f42b745265293 .... but otherwise, the http-client would seem to respect a value returned by the server in http-headers. ( So at least for us, this means we now need to be looking at our influxdb configs, and if some version supports setting the "Keep-alive: timeout=N"-header)

[shakuzen]

Thanks for sharing. JDK-8278067 is the corresponding JDK issue. The property is only available in Java 19 EA builds for now, as far as I can tell.

[karniemi]

Does micrometer already handle the connection like this?: https://awmanoj.github.io/tech/2016/12/16/keep-alive-http-requests-in-golang/ Briefly, the client should read the response body for keep-alive to work, becuase reading the response signals the lower layers that the connection is not idle. It's an old article, though.

[shakuzen]

@karniemi That article is about the Golang HTTP client. As I've written in my previous comments, the keepalive appears to be working as expected for HttpUrlConnection.

If a request to the same host is made within 5 seconds, the same underlying TCP connection will be reused.

The new system properties you previously found have since been backported and are scheduled to be in the next patch releases of Java 8, 11, 17. Once those are out, you should be able to use the system property http.keepAlive.time.server to customize the default keepalive timeout for HttpUrlConnection.

As I mentioned in https://github.com/micrometer-metrics/micrometer/issues/1679#issuecomment-1145639674, I don't think there is anything we can change in Micrometer to better support this, but suggestions are welcome if someone has ideas. JDK-8278067 will soon make it possible to customize the default keepalive timeout.

[karniemi]

@shakuzen OK, thanks! ( And sorry for getting mixed up about the golang http client. )