IllegalStateException after ArrayByteBufferPool$Buffer.release called from HttpConnection.releaseRequestBuffer

[zbigniew-dt]

Jetty version(s) 12.0.16

Jetty Environment ee8

Java version/vendor (use: java -version) OpenJDK Runtime Environment Temurin-17.0.13+11

OS type/version Linux

Description It is similar to #12662 and #12082 but .ArrayByteBufferPool$Buffer.release is called from another method. It happened 4 times. I cannot reproduce it.

java.lang.IllegalStateException: already released ReferenceCounter@5ac3b52d[r=0]
	at org.eclipse.jetty.io.Retainable$ReferenceCounter.lambda$release$2(Retainable.java:192)
	at java.base/java.util.concurrent.atomic.AtomicInteger.updateAndGet(AtomicInteger.java:281)
	at org.eclipse.jetty.io.Retainable$ReferenceCounter.release(Retainable.java:189)
	at org.eclipse.jetty.io.AbstractRetainableByteBuffer.release(AbstractRetainableByteBuffer.java:58)
	at org.eclipse.jetty.io.ArrayByteBufferPool$Buffer.release(ArrayByteBufferPool.java:651)
	at org.eclipse.jetty.server.internal.HttpConnection.releaseRequestBuffer(HttpConnection.java:345)
	at org.eclipse.jetty.server.internal.HttpConnection.parseAndFillForContent(HttpConnection.java:534)
	at org.eclipse.jetty.server.internal.HttpConnection$HttpStreamOverHTTP1.read(HttpConnection.java:1350)
	at org.eclipse.jetty.server.HttpStream$Wrapper.read(HttpStream.java:167)
	at org.eclipse.jetty.server.internal.HttpChannelState$ChannelRequest.read(HttpChannelState.java:931)
	at org.eclipse.jetty.server.Request$Wrapper.read(Request.java:896)
	at org.eclipse.jetty.server.handler.EventsHandler$EventsRequest.read(EventsHandler.java:376)
	at org.eclipse.jetty.server.Request$Wrapper.read(Request.java:896)
	at org.eclipse.jetty.ee8.nested.HttpChannel.produceContent(HttpChannel.java:179)
	at org.eclipse.jetty.ee8.nested.AsyncContentProducer.produceRawContent(AsyncContentProducer.java:388)
	at org.eclipse.jetty.ee8.nested.AsyncContentProducer.nextTransformedContent(AsyncContentProducer.java:282)
	at org.eclipse.jetty.ee8.nested.AsyncContentProducer.nextContent(AsyncContentProducer.java:216)
	at org.eclipse.jetty.ee8.nested.BlockingContentProducer.nextContent(BlockingContentProducer.java:91)
	at org.eclipse.jetty.ee8.nested.HttpInput.read(HttpInput.java:258)
	at org.eclipse.jetty.ee8.nested.HttpInput.read(HttpInput.java:244)

How to reproduce? I do not know.

[gregw]

What connector/protocol are you using? HTTP/1.1? H2? H3? TLS? What kind of content? How large? chunked? what reads it? Is it always fully read? Gzipped?

[lorban]

Unfortunately, a single stack trace isn't nearly enough information to track down a byte buffer lifecycle bug as retains/releases can be done at many different places for many different reasons.

Answering @gregw's question would help narrowing the scope of where to look; on top of that reproducing the problem with the ArrayByteBufferPool.Tracking configured & calling dumpLeaks() on it afterward would also tremendously help figuring out where the problem lies.

[zbigniew-dt]

The stack trace is from jetty configured on HTTPS and HTTP, The stack trace comes from the HTTP connection. HTTPS is used only for diagnostic access - almost not used.
| What kind of content?

• binary | How large? chunked? what reads it? Is it always fully read? Unfortunately, I cannot help here. So far the issue did not repeat. I can try @lorban suggestion if it backs

[jmbeach]

I'm also seeing this after upgrading to Jetty 12

[lorban]

@jmbeach what exact version of Jetty are you using? Can you easily reproduce that? Can you afford modifying your environment to collect some data we could use to help us track down that issue?

Thanks!

[jmbeach]

Using 12.0.15. Not sure what causes it.

Also, my errors bubble up from org.eclipse.jetty.util.thread.QueuedThreadPool:

Job failed java.lang.IllegalStateException: already released ReferenceCounter@3701de74[r=0]
	at org.eclipse.jetty.io.Retainable$ReferenceCounter.lambda$release$2(Retainable.java:192)
	at java.base/java.util.concurrent.atomic.AtomicInteger.updateAndGet(Unknown Source)
	at org.eclipse.jetty.io.Retainable$ReferenceCounter.release(Retainable.java:189)
	at org.eclipse.jetty.io.AbstractRetainableByteBuffer.release(AbstractRetainableByteBuffer.java:58)
	at org.eclipse.jetty.io.ArrayByteBufferPool$Buffer.release(ArrayByteBufferPool.java:651)
	at org.eclipse.jetty.server.internal.HttpConnection.releaseRequestBuffer(HttpConnection.java:345)
	at org.eclipse.jetty.server.internal.HttpConnection.onFillable(HttpConnection.java:481)
	at org.eclipse.jetty.server.internal.HttpConnection.run(HttpConnection.java:640)
	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:979)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.doRunJob(QueuedThreadPool.java:1209)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1164)
	at java.base/java.lang.Thread.run(Unknown Source) 

I don't see anything interesting coinciding with these errors. They're very infrequent as well.

I possibly can help modify my environment to collect data. What would that entail?

[joakime]

Using 12.0.15. Not sure what causes it.

Please use 12.0.19 to test this. You are missing many improvements to the ByteBufferPool. And there's been many additions to the various ByteBufferPool implementations to collect statistics and information about how it is operating that your old version of Jetty does not have.

[lorban]

Please upgrade to the latest version like @joakime suggests as the leak tracker recently was seriously improved, then arrange your code to pass a ArrayByteBufferPool.Tracking instance to the Server constructor, e.g.:

ArrayByteBufferPool.Tracking bufferPool = new ArrayByteBufferPool.Tracking();
Server server = new Server(null, null, bufferPool);

Then you can call dumpLeaks() on that buffer pool when the server is at rest (meaning you know there are no in-flight requests) to get a report of all buffer lifecycle anomalies. This should help understanding what's going on.