akka
HTTP Client API - TCP Negotiation Timeout Doesn't Work as Expected
Hello,
We've recently started to develop a feature that a part of it involves sending HTTP requests to endpoints URLs that might not be active. Meaning, we could have a URL to an IP of a machine that wouldn't receive HTTP requests, and of course, no TCP connections at all.
We've tried configuring the HTTP Client API with akka.http.client.connecting-timeout=3s and also akka.http.host-connection-pool.max-retries=0 retries.
While doing that, we've saw in Wireshark that the TCP Syn message that was sent by the sending endpoint, had a corresponding RST message on the same TCP connection after 10s.
The issue is having a response in the future of the Http().singleRequest call only after ~7s.
When trying to configure the Http().singleRequest with retires of 1, we did saw another attempt of TCP Syn to the IP configured in the call, but the overall time in the PCAP was 20s, and additional 3s to actually receive the future response of the Http().singleRequest call.
In both described tests, when trying to add the configuration of akka.http.client.idle-timeout=3s, which of courses expected to destroy the TCP connection after 3s, we do see in the PCAP that every TCP syn has a corresponding RST message which has ~4s interval between them(The SYN and RST).
In the case of having retries configured to 0, we have the response time of the future in ~7s and with 1 retry we get the future response after ~11s.
In conclusion, what we want to achieve is having the HTTP client either one of the following if possible of course:
- Send more TCP syn messages in a shorter window of time, meaning more retries of TCP syn in a short period of time.
- Be able to configure the total TCP negotiation time, and have the future response in that somewhat close period of time.
- Have another possibility to access the entity, or alternatively know if we can establish a TCP connection or not before doing the HTTP request itself, which I guess can be the TCP Sockets in Akka but asking in the case maybe there's another unknown option.
General details:
- Scala version 2.12
- Akka Actor version 2.6.18
- Akka Stream version 2.6.18
- Akka HTTP version 10.2.8
I hope there might be a mistake in my configurations and it can be solved in a mere configuration but if not, please advise when you can of course if you have any way of possible actions that can be taken on the scenario.
If anything else is needed to understand the issue or analyze it please let me know.
Thanks!
Hello @jrudolph sup? :)
Hope it's OK to tag you here. I saw your comments on other issues so I just wanted to make sure someone from the Akka project has seen this issue or not by any chance.
Is the described issue is a known one? If it does, is there a way to overcome it, or alternatively if there are previous related issues to read about, that would be great as well?
Sincerely yours, Ido
Sorry, forget to answer here. I had a try at reproducing this but for me it works as expected (when running requests against a host that just drops SYN packets to certain ports). Maybe you can share debug logs and a tcpdump?
Hi @idoMajor41,
you can look at the documentation of the base-connection-backoff setting in the reference.conf. It shows our handling of connection attempts. In particular, after a failure we do exponential backoff up to the setting of max-connection-backoff until calling connect again.
If you want to probe an endpoint, instead of using the pool, try using a dedicated connection with Http().outgoingConnection and add a timeout stage to get a quicker result.
Btw.
that might not be active.
"not be active" can mean many things:
- cannot be routed (as reported by an ICMP packet)
- refuses connection (as reported by an ICMP packet)
- returns RST packet
- drops SYN packets (because of firewall or because a routeable endpoint has suddenly gone away)
The first three will probably quickly end up in a failed connection attempt in akka-http. The last one will trigger automatic retries (in Linux) which might take minutes with default settings until it times out on that level (though the connecting-timeout will speed up reports in akka-http).