Unexpected ClosedChannelException

[n-miles]

We observed the following error when making an HTTP request on AHC version 2.11.0. I don't have any other relevant information about this since we don't have debug logs turned on in our production environment. It has only happened once, and I cannot reproduce it.

java.util.concurrent.CompletionException: java.nio.channels.ClosedChannelException
	at java.base/java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:331)
	at java.base/java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:346)
	at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:870)
	at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:837)
	at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506)
	at java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2159)
	at org.asynchttpclient.netty.NettyResponseFuture.abort(NettyResponseFuture.java:273)
	at org.asynchttpclient.netty.request.WriteListener.abortOnThrowable(WriteListener.java:50)
	at org.asynchttpclient.netty.request.WriteListener.operationComplete(WriteListener.java:61)
	at org.asynchttpclient.netty.request.WriteCompleteListener.operationComplete(WriteCompleteListener.java:28)
	at org.asynchttpclient.netty.request.WriteCompleteListener.operationComplete(WriteCompleteListener.java:20)
	at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:500)
	at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:474)
	at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:413)
	at io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:538)
	at io.netty.util.concurrent.DefaultPromise.setFailure0(DefaultPromise.java:531)
	at io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:111)
	at io.netty.channel.AbstractChannel$AbstractUnsafe.safeSetFailure(AbstractChannel.java:991)
	at io.netty.channel.AbstractChannel$AbstractUnsafe.write(AbstractChannel.java:863)
	at io.netty.channel.DefaultChannelPipeline$HeadContext.write(DefaultChannelPipeline.java:1378)
	at io.netty.channel.AbstractChannelHandlerContext.invokeWrite0(AbstractChannelHandlerContext.java:716)
	at io.netty.channel.AbstractChannelHandlerContext.invokeWrite(AbstractChannelHandlerContext.java:708)
	at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:791)
	at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:701)
	at io.netty.channel.AbstractChannelHandlerContext.invokeWrite(AbstractChannelHandlerContext.java:710)
	at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:791)
	at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:701)
	at io.netty.channel.AbstractChannelHandlerContext.invokeWrite(AbstractChannelHandlerContext.java:710)
	at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:791)
	at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:701)
	at io.netty.channel.AbstractChannelHandlerContext.invokeWrite(AbstractChannelHandlerContext.java:710)
	at io.netty.channel.AbstractChannelHandlerContext.access$1700(AbstractChannelHandlerContext.java:56)
	at io.netty.channel.AbstractChannelHandlerContext$AbstractWriteTask.write(AbstractChannelHandlerContext.java:1102)
	at io.netty.channel.AbstractChannelHandlerContext$WriteAndFlushTask.write(AbstractChannelHandlerContext.java:1149)
	at io.netty.channel.AbstractChannelHandlerContext$AbstractWriteTask.run(AbstractChannelHandlerContext.java:1073)
	at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163)
	at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:416)
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:515)
	at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:918)
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	at java.base/java.lang.Thread.run(Thread.java:835)
Caused by: java.nio.channels.ClosedChannelException: null
	at io.netty.channel.AbstractChannel$AbstractUnsafe.newClosedChannelException(AbstractChannel.java:955)
	... 24 common frames omitted

It sounds like something on the client side is closing the Netty channel early and then the client is trying to use it again. I don't know enough about how the networking portions of AHC work to offer a guess as to a solution.

The error happened about 16ms after it was submitted to the AsyncHttpClient, so that might rule out timeouts.

[TomGranot]

@n-miles Have you seen this happening since?

[n-miles]

I haven't noticed it if it has. We have retry logic on everything so small blips usually get absorbed without issue.

[TomGranot]

@n-miles I thought about closing this, but I'm going to keep it in a list of defects Id like to eventually get to, and perhaps jump off of for the roadmap (for example, add a task for writing tests that check this type of behavior). If you've got any other examples of this type of shenanigans please continue writing them here, it's great.