How the HttpClient to select the connection for multiple HttpDestinations with DuplexConnectionPool? (HTTPS)

[jingke]

Jetty version 9.4.7.v20170914

Java version

java -version

openjdk version "1.8.0_232" OpenJDK Runtime Environment (build 1.8.0_232-b09) OpenJDK 64-Bit Server VM (build 25.232-b09, mixed mode)

Question

How the HttpClient to select the connection for multiple HttpDestinations with DuplexConnectionPool? (HTTPS)

From our understanding, one HttpClient could serve multiple HttpDestinations, each HttpDestination contaions one DuplexConnectionPool and each pool maintains their own connection to same destination.

But from our testing, it seems the Connections could be used among HttpDestinations. Scenario is below.

One HttpClient, Two Destinations [scheme: https, hosts: tpas3.ims.fs5000lab.com, tpas.ims.fs5000lab.com, port: 8443], start call load to send HTTPS request to both destination. then stop one destination, we found the request to this stopped destination was sent to non-stopped one with the connection to non-stopped destination.

That's strange to us since our newRequest give the stopped destination but finally the connection was found in non-stopped destination's connection pool.

Does the PoolingHttpDestination select the different detination's connection?

Below is our log: (no detail jetty log as some unknown issue to print them)

==== before sending request to stopped destination, print all destinations ============ 99976 PRS0: TRACE sendRequest, https request, http clients: =============== [HttpDestination[https://tpas3.ims.fs5000lab.com:8443]@1b8c909,queue=0,pool=DuplexConnectionPool@3c29bfad[c=5/1200,a=0,i=5], HttpDestination[https://tpas .ims.fs5000lab.com:8443]@68560557,queue=0,pool=DuplexConnectionPool@1a7736f7[c=1/1200,a=1,i=0]] 99977 ^M

==== before sending request to stopped destination, print get stopped destination result ============ 99978 +++ 2022/08/04 01:23:18.646 IMS HIGH java:29219 E:60378 S:42738 (httpsClient-89:HttpsRequestHandler.java 145 V-0002 169.254.64.2 PRS0M0) 99979 99980 PRS0: TRACE sendRequest, https request, http clients: --------------- HttpDestination[https://tpas.ims.fs5000lab.com:8443]@68560557,queue=0,pool=DuplexConnectionPool@1a7736f7[c=1/1200,a=1,i=0] https tpas.ims.fs5000lab.com 8 443 99981 ^M 99982 +++ 2022/08/04 01:23:18.646 IMS HIGH java:29219 E:60379 S:42739 (httpsClient-89:HttpsRequestHandler.java 149 V-0002 169.254.64.2 PRS0M0) 99983

==== before sending request to stopped destination, print each connections activate connections and idle connection per destination ============ 99984 PRS0: TRACE sendRequest, https request, http desination: ######## HttpDestination[https://tpas3.ims.fs5000lab.com:8443]@1b8c909,queue=0,pool=DuplexConnectionPool@3c29bfad[c=5/1200,a=0,i=5] 99985 ^M 99986 +++ 2022/08/04 01:23:18.646 IMS HIGH java:29219 E:60380 S:42740 (httpsClient-89:HttpsRequestHandler.java 149 V-0002 169.254.64.2 PRS0M0) 99987 99988 PRS0: TRACE sendRequest, https request, http desination: ######## HttpDestination[https://tpas.ims.fs5000lab.com:8443]@68560557,queue=0,pool=DuplexConnectionPool@1a7736f7[c=1/1200,a=1,i=0] 99989 ^M 99990 +++ 2022/08/04 01:23:18.646 IMS HIGH java:29219 E:60381 S:42741 (httpsClient-89:HttpsRequestHandler.java 154 V-0002 169.254.64.2 PRS0M0) 99991 99992 PRS0: TRACE sendRequest: connection: HttpConnectionOverHTTP@79a946a8(l:/c10d:2:193:0:0:0:0:1:49132 <-> r:/2000:0:0:0:57:8443,closed=false)=>HttpChannelOverHTTP@2a7ae7d3(exchange=HttpExchange@aa92a92 req=TERMINAT ED/null@null res=PENDING/null@null)[send=HttpSenderOverHTTP@2e7d127d(req=QUEUED,snd=COMPLETED,failure=null)[HttpGenerator@283d1d1e{s=START}],recv=HttpReceiverOverHTTP@550423b5(rsp=IDLE,failure=null)[HttpParser{s=START,0 of -1}]]

we are sending the request to FQDN https://tpas.ims.fs5000lab.com:8443, ip dns query result is https://[2000:0:0:0:41]:8443,

PRS0: TRACE sendRequest, https request, target url: https://tpas.ims.fs5000lab.com:8443/assim/1.0/notifications/callDirection, final url: https://[2000:0:0:0:41]:8443/assim/1.0/notifications/callDirection, sub na me: 4512195813-33655-SUSPENDED-CALL-RECIPIENT

but finally HttpClient select one connection to IP 2000:0:0:0:57, this IP for sure belong to non-stopped destination (tpas3.ims.fs5000lab.com)

[lisay-yan]

Hi, Jetty Expert

For @jingke 's question, suspect it maybe caused by set resolver per request, and over write maybe happened during traffic load.

Per request call below func, it may mess connect target in Jetty. client.setSocketAddressResolver(new SocketAddressResolver() ...

My question, is it possible set preferred IP per request without rewrite resolver? Application keep a failure IP list, and will prevent target to such IP for following request. Thanks.

[sbordet]

Sorry, but I don't understand what you're doing.

Can you write a reproducible test case that we can run and that shows the problem?

It's quite unlikely that connections end up in the wrong connection pool without messing quite heavily with the HttpClient internals.

[github-actions[bot]]

This issue has been automatically marked as stale because it has been a full year without activity. It will be closed if no further activity occurs. Thank you for your contributions.

[sbordet]

No feedback, closing.