Jetty Http/2 Client send operation getting stuck in performance run

[kdtech1001]

trafficstars

Jetty version(s) 11.0.20 Java version/vendor (use: java -version) 17.0.11 Spring boot version(s) 2.7.21 OS type/version Linux

I know Jetty 11 is out of support and we are migrating to 12. but in one of our old release we are observing that Jetty send is getting stuck for a good amount of time while running performance (4K TPS) on Ipv6 using Jetty HTTP/2 Client. Everything works if we use Ipv4. We are using jetty with reactor core to make asynchronous send.

Thread Stacktrace

[email protected]/jdk.internal.misc.Unsafe.park(Native Method), 
[email protected]/java.util.concurrent.locks.LockSupport.park(LockSupport.java:211), 
[email protected]/java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:715), [email protected]/java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:938), 
[email protected]/java.util.concurrent.locks.ReentrantLock$Sync.lock(ReentrantLock.java:153), 
[email protected]/java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:322), 
[email protected]/java.net.InMemoryCookieStore.get(InMemoryCookieStore.java:120), 
org.eclipse.jetty.client.HttpConnection.normalizeRequest(HttpConnection.java:224), 
org.eclipse.jetty.http2.client.http.HttpConnectionOverHTTP2.normalizeRequest(HttpConnectionOverHTTP2.java:158), 
org.eclipse.jetty.http2.client.http.HttpConnectionOverHTTP2.send(HttpConnectionOverHTTP2.java:111), 
org.eclipse.jetty.client.HttpDestination.send(HttpDestination.java:440), 
org.eclipse.jetty.client.HttpDestination.process(HttpDestination.java:416), 
org.eclipse.jetty.client.HttpDestination.process(HttpDestination.java:371), 
org.eclipse.jetty.client.HttpDestination.send(HttpDestination.java:354), 
org.eclipse.jetty.client.HttpDestination.send(HttpDestination.java:348), 
org.eclipse.jetty.client.HttpDestination.send(HttpDestination.java:325), 
org.eclipse.jetty.client.HttpDestination.send(HttpDestination.java:304), 
org.eclipse.jetty.client.HttpClient.send(HttpClient.java:501), 
org.eclipse.jetty.client.HttpRequest$$Lambda$1511/0x0000000800c886b8.accept(Unknown Source), 
org.eclipse.jetty.client.HttpRequest.sendAsync(HttpRequest.java:879), 
org.eclipse.jetty.client.HttpRequest.send(HttpRequest.java:867), 
org.eclipse.jetty.reactive.client.internal.ResponseListenerProcessor.send(ResponseListenerProcessor.java:169), 
org.eclipse.jetty.reactive.client.internal.ResponseListenerProcessor.onRequest(ResponseListenerProcessor.java:155), 
org.eclipse.jetty.reactive.client.internal.AbstractSinglePublisher.request(AbstractSinglePublisher.java:90), 
reactor.core.publisher.MonoIgnoreThen$ThenIgnoreMain.onSubscribe(MonoIgnoreThen.java:134), 
org.eclipse.jetty.reactive.client.internal.AbstractSinglePublisher.subscribe(AbstractSinglePublisher.java:62), 
reactor.core.publisher.MonoSource.subscribe(MonoSource.java:71), 
reactor.core.publisher.MonoIgnoreThen$ThenIgnoreMain.subscribeNext(MonoIgnoreThen.java:240), 
reactor.core.publisher.MonoIgnoreThen$ThenIgnoreMain.onComplete(MonoIgnoreThen.java:203), 
reactor.core.publisher.FluxContextWrite$ContextWriteSubscriber.onComplete(FluxContextWrite.java:126), 
reactor.core.publisher.MonoFlatMap$FlatMapMain.secondComplete(MonoFlatMap.java:250), 
reactor.core.publisher.MonoFlatMap$FlatMapInner.onComplete(MonoFlatMap.java:324), 
reactor.core.publisher.MonoIgnoreThen$ThenIgnoreMain.onComplete(MonoIgnoreThen.java:209), 
reactor.core.publisher.MonoIgnoreElements$IgnoreElementsSubscriber.onComplete(MonoIgnoreElements.java:89), 
reactor.core.publisher.FluxConcatIterable$ConcatIterableSubscriber.onComplete(FluxConcatIterable.java:121), 
reactor.core.publisher.FluxConcatIterable.subscribe(FluxConcatIterable.java:60), 
reactor.core.publisher.MonoFromFluxOperator.subscribe(MonoFromFluxOperator.java:81), 
reactor.core.publisher.MonoIgnoreThen$ThenIgnoreMain.subscribeNext(MonoIgnoreThen.java:240), 
reactor.core.publisher.MonoIgnoreThen$ThenIgnoreMain.onComplete(MonoIgnoreThen.java:203), 
reactor.core.publisher.MonoCreate$DefaultMonoSink.success(MonoCreate.java:140), 
org.springframework.http.client.reactive.JettyClientHttpRequest.lambda$writeWith$2(JettyClientHttpRequest.java:98), 
org.springframework.http.client.reactive.JettyClientHttpRequest$$Lambda$2062/0x0000000800ebd3c0.accept(Unknown Source), 
reactor.core.publisher.MonoCreate.subscribe(MonoCreate.java:58), 
reactor.core.publisher.Mono.subscribe(Mono.java:4495), 
reactor.core.publisher.MonoIgnoreThen$ThenIgnoreMain.subscribeNext(MonoIgnoreThen.java:263), 
reactor.core.publisher.MonoIgnoreThen.subscribe(MonoIgnoreThen.java:51), 
reactor.core.publisher.MonoFlatMap$FlatMapMain.onNext(MonoFlatMap.java:165), 
reactor.core.publisher.FluxSwitchIfEmpty$SwitchIfEmptySubscriber.onNext(FluxSwitchIfEmpty.java:74), 
reactor.core.publisher.Operators$MonoInnerProducerBase.complete(Operators.java:2811), 
reactor.core.publisher.MonoSingle$SingleSubscriber.onComplete(MonoSingle.java:180), 
reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.onComplete(FluxMapFuseable.java:152), 
reactor.core.publisher.Operators$ScalarSubscription.request(Operators.java:2547), 
reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.request(FluxMapFuseable.java:171), 
reactor.core.publisher.MonoSingle$SingleSubscriber.doOnRequest(MonoSingle.java:103), 
reactor.core.publisher.Operators$MonoInnerProducerBase.request(Operators.java:2878), 
reactor.core.publisher.Operators$MultiSubscriptionSubscriber.set(Operators.java:2341), 
reactor.core.publisher.Operators$MultiSubscriptionSubscriber.onSubscribe(Operators.java:2215), 
reactor.core.publisher.MonoSingle$SingleSubscriber.onSubscribe(MonoSingle.java:115), 
reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.onSubscribe(FluxMapFuseable.java:96), 
reactor.core.publisher.FluxJust.subscribe(FluxJust.java:68), 
reactor.core.publisher.Mono.subscribe(Mono.java:4495), 
reactor.core.publisher.MonoIgnoreThen$ThenIgnoreMain.subscribeNext(MonoIgnoreThen.java:263), 
reactor.core.publisher.MonoIgnoreThen.subscribe(MonoIgnoreThen.java:51), 
reactor.core.publisher.InternalMonoOperator.subscribe(InternalMonoOperator.java:64), 
reactor.core.publisher.MonoDefer.subscribe(MonoDefer.java:53), 
reactor.core.publisher.Mono.subscribe(Mono.java:4495), 
reactor.core.publisher.Mono.subscribeWith(Mono.java:4561), 
reactor.core.publisher.Mono.subscribe(Mono.java:4462), 
reactor.core.publisher.Mono.subscribe(Mono.java:4398), 
reactor.core.publisher.Mono.subscribe(Mono.java:4370), 
Test.Application.asynclients.SendHttpRequest.buildRequest(SendHttpRequest.java:543)

Trying to understand what could be the reason behind InMemoryCookieStore.get() getting stuck.

Any pointers would be helpful. Thanks.

[sbordet]

Trying to understand what could be the reason behind InMemoryCookieStore.get() getting stuck.

No idea. We did not see this issue in other IPv6 deployments we know.

This stack trace shows the thread that is waiting for the lock, so there must be another thread that has the lock -- you should look at what that thread is doing.

[github-actions[bot]]

This issue has been automatically marked as stale because it has been a full year without activity. It will be closed if no further activity occurs. Thank you for your contributions.