Jetty Client EOF Exception on server to server communication

[ginccc]

Jetty version(s) 9.4.42.v20210604

Java version/vendor (use: java -version) 11

OS type/version linux

Description On our production servers we sometimes experience an EOFException. It is the jetty client consuming an api. Usually it works fine, but sometimes it throws this exception below.. I haven't been able to reproduce, but it keeps happening..

I thought I reach out to see if this is maybe an edge case or if something can be done about it. Please advise :-)

(...)

Caused by: java.util.concurrent.ExecutionException java.io.EOFException: HttpConnectionOverHTTP@e0c13a4::DecryptedEndPoint@1708bd0c{l=/IP:PORT,r=HOST/IP:443,OPEN,fill=-,flush=-,to=118679/0} 
    FutureResponseListener.java:118 org.eclipse.jetty.client.util.FutureResponseListener.getResult
    FutureResponseListener.java:101 org.eclipse.jetty.client.util.FutureResponseListener.get
    HttpClientWrapper.java:120 ai.labs.httpclient.impl.HttpClientWrapper$RequestWrapper.send
    HttpCallsTask.java:288 ai.labs.restapi.connector.impl.HttpCallsTask.executeRequest
    HttpCallsTask.java:194 ai.labs.restapi.connector.impl.HttpCallsTask.executeAndMeasureRequest
    HttpCallsTask.java:136 ai.labs.restapi.connector.impl.HttpCallsTask.executeTask
    LifecycleManager.java:38 ai.labs.lifecycle.LifecycleManager.executeLifecycle
    Conversation.java:302 ai.labs.runtime.internal.Conversation.executePackages
    Conversation.java:233 ai.labs.runtime.internal.Conversation.executeConversationStep
    Conversation.java:82 ai.labs.runtime.internal.Conversation.init
    Bot.java:52 ai.labs.runtime.internal.Bot.startConversation
    RestBotEngine.java:140 ai.labs.core.rest.internal.RestBotEngine.startConversationWithContext
    RestBotManagement.java:211 ai.labs.core.rest.internal.RestBotManagement.createNewConversation
    RestBotManagement.java:128 ai.labs.core.rest.internal.RestBotManagement.initUserConversation
    RestBotManagement.java:59 ai.labs.core.rest.internal.RestBotManagement.loadConversationMemory
    Unknown:-1 jdk.internal.reflect.GeneratedMethodAccessor240.invoke
    Unknown:-1 jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke
    Unknown:-1 java.lang.reflect.Method.invoke
    MethodInjectorImpl.java:170 org.jboss.resteasy.core.MethodInjectorImpl.invoke
    MethodInjectorImpl.java:130 org.jboss.resteasy.core.MethodInjectorImpl.invoke
    ResourceMethodInvoker.java:660 org.jboss.resteasy.core.ResourceMethodInvoker.internalInvokeOnTarget
    ResourceMethodInvoker.java:524 org.jboss.resteasy.core.ResourceMethodInvoker.invokeOnTargetAfterFilter
    ResourceMethodInvoker.java:474 org.jboss.resteasy.core.ResourceMethodInvoker.lambda$invokeOnTarget$2
    PreMatchContainerRequestContext.java:364 org.jboss.resteasy.core.interception.jaxrs.PreMatchContainerRequestContext.filter
    ResourceMethodInvoker.java:476 org.jboss.resteasy.core.ResourceMethodInvoker.invokeOnTarget
    ResourceMethodInvoker.java:434 org.jboss.resteasy.core.ResourceMethodInvoker.invoke
    ResourceMethodInvoker.java:408 org.jboss.resteasy.core.ResourceMethodInvoker.invoke
    ResourceMethodInvoker.java:69 org.jboss.resteasy.core.ResourceMethodInvoker.invoke
    SynchronousDispatcher.java:492 org.jboss.resteasy.core.SynchronousDispatcher.invoke
    SynchronousDispatcher.java:261 org.jboss.resteasy.core.SynchronousDispatcher.lambda$invoke$4
    SynchronousDispatcher.java:161 org.jboss.resteasy.core.SynchronousDispatcher.lambda$preprocess$0
    PreMatchContainerRequestContext.java:364 org.jboss.resteasy.core.interception.jaxrs.PreMatchContainerRequestContext.filter
    SynchronousDispatcher.java:164 org.jboss.resteasy.core.SynchronousDispatcher.preprocess
    SynchronousDispatcher.java:247 org.jboss.resteasy.core.SynchronousDispatcher.invoke
    ServletContainerDispatcher.java:249 org.jboss.resteasy.plugins.server.servlet.ServletContainerDispatcher.service
    HttpServletDispatcher.java:60 org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service
    HttpServletDispatcher.java:55 org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service
    HttpServlet.java:790 javax.servlet.http.HttpServlet.service
    ServletHolder.java:799 org.eclipse.jetty.servlet.ServletHolder.handle
    ServletHandler.java:1626 org.eclipse.jetty.servlet.ServletHandler$ChainEnd.doFilter
    ServerRuntime.java:342 ai.labs.server.ServerRuntime$4.doFilter
    FilterHolder.java:193 org.eclipse.jetty.servlet.FilterHolder.doFilter
    ServletHandler.java:1601 org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter
    ServerRuntime.java:283 ai.labs.server.ServerRuntime$3.doFilter
    FilterHolder.java:193 org.eclipse.jetty.servlet.FilterHolder.doFilter
    ServletHandler.java:1601 org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter
    ServletHandler.java:548 org.eclipse.jetty.servlet.ServletHandler.doHandle
    ScopedHandler.java:233 org.eclipse.jetty.server.handler.ScopedHandler.nextHandle
    SessionHandler.java:1624 org.eclipse.jetty.server.session.SessionHandler.doHandle
    ScopedHandler.java:233 org.eclipse.jetty.server.handler.ScopedHandler.nextHandle
    ContextHandler.java:1435 org.eclipse.jetty.server.handler.ContextHandler.doHandle
    ScopedHandler.java:188 org.eclipse.jetty.server.handler.ScopedHandler.nextScope
    ServletHandler.java:501 org.eclipse.jetty.servlet.ServletHandler.doScope
    SessionHandler.java:1594 org.eclipse.jetty.server.session.SessionHandler.doScope
    ScopedHandler.java:186 org.eclipse.jetty.server.handler.ScopedHandler.nextScope
    ContextHandler.java:1350 org.eclipse.jetty.server.handler.ContextHandler.doScope
    ScopedHandler.java:141 org.eclipse.jetty.server.handler.ScopedHandler.handle
    HandlerList.java:59 org.eclipse.jetty.server.handler.HandlerList.handle
    StatisticsHandler.java:179 org.eclipse.jetty.server.handler.StatisticsHandler.handle
    HandlerWrapper.java:127 org.eclipse.jetty.server.handler.HandlerWrapper.handle
    Server.java:516 org.eclipse.jetty.server.Server.handle
    HttpChannel.java:388 org.eclipse.jetty.server.HttpChannel.lambda$handle$1
    HttpChannel.java:633 org.eclipse.jetty.server.HttpChannel.dispatch
    HttpChannel.java:380 org.eclipse.jetty.server.HttpChannel.handle
    HttpConnection.java:277 org.eclipse.jetty.server.HttpConnection.onFillable
    AbstractConnection.java:311 org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded
    FillInterest.java:105 org.eclipse.jetty.io.FillInterest.fillable
    ChannelEndPoint.java:104 org.eclipse.jetty.io.ChannelEndPoint$1.run
    EatWhatYouKill.java:338 org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask
    EatWhatYouKill.java:315 org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce
    EatWhatYouKill.java:173 org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce
    EatWhatYouKill.java:131 org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run
    ReservedThreadExecutor.java:383 org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run
    Unknown:-1 java.util.concurrent.Executors$RunnableAdapter.call
    Unknown:-1 java.util.concurrent.FutureTask.run
    Unknown:-1 java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run
    Unknown:-1 java.util.concurrent.ThreadPoolExecutor.runWorker
    Unknown:-1 java.util.concurrent.ThreadPoolExecutor$Worker.run
    Unknown:-1 java.lang.Thread.run

Caused by: java.io.EOFException 
HttpConnectionOverHTTP@e0c13a4::DecryptedEndPoint@1708bd0c{l=/IP:PORT,r=HOST/IP:443,OPEN,fill=-,flush=-,to=118679/0} 
    HttpReceiverOverHTTP.java:376 org.eclipse.jetty.client.http.HttpReceiverOverHTTP.earlyEOF
    HttpParser.java:1614 org.eclipse.jetty.http.HttpParser.parseNext
    HttpReceiverOverHTTP.java:259 org.eclipse.jetty.client.http.HttpReceiverOverHTTP.shutdown
    HttpReceiverOverHTTP.java:181 org.eclipse.jetty.client.http.HttpReceiverOverHTTP.process
    HttpReceiverOverHTTP.java:79 org.eclipse.jetty.client.http.HttpReceiverOverHTTP.receive
    HttpChannelOverHTTP.java:131 org.eclipse.jetty.client.http.HttpChannelOverHTTP.receive
    HttpConnectionOverHTTP.java:172 org.eclipse.jetty.client.http.HttpConnectionOverHTTP.onFillable
    AbstractConnection.java:311 org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded
    FillInterest.java:105 org.eclipse.jetty.io.FillInterest.fillable
    SslConnection.java:555 org.eclipse.jetty.io.ssl.SslConnection$DecryptedEndPoint.onFillable
    SslConnection.java:410 org.eclipse.jetty.io.ssl.SslConnection.onFillable
    SslConnection.java:164 org.eclipse.jetty.io.ssl.SslConnection$2.succeeded
    FillInterest.java:105 org.eclipse.jetty.io.FillInterest.fillable
    ChannelEndPoint.java:104 org.eclipse.jetty.io.ChannelEndPoint$1.run
    Unknown:-1 java.util.concurrent.Executors$RunnableAdapter.call
    Unknown:-1 java.util.concurrent.FutureTask.run
    Unknown:-1 java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run
    Unknown:-1 java.util.concurrent.ThreadPoolExecutor.runWorker
    Unknown:-1 java.util.concurrent.ThreadPoolExecutor$Worker.run
    Unknown:-1 java.lang.Thread.run

How to reproduce? N/A

[sbordet]

The call to HttpReceiverOverHTTP.shutdown() in the stack trace means that the other end closed the connection prematurely.

You should look at the other side and see why it closed the connection.

[github-actions[bot]]

This issue has been automatically marked as stale because it has been a full year without activity. It will be closed if no further activity occurs. Thank you for your contributions.