jetty.project icon indicating copy to clipboard operation
jetty.project copied to clipboard

Published 20 hours ago • verified publisher icon jetty

Potential memory leak with ArrayRetainableByteBufferPool$RetainedBucket

Open nirvanaemmone opened this issue 2 years ago • 8 comments
trafficstars

Jetty version(s) 10.0.12

Java version/vendor (use: java -version) open jdk 11 OS type/version linux / amd 64 Description We have customer reported the server stoped the working due to OOM. Looked at the dump it seems like related to ArrayRetainableByteBufferPool: Screenshot 2023-09-13 at 11 16 51 AM

How to reproduce? Currently unknown

nirvanaemmone avatar Sep 13 '23 15:09 nirvanaemmone

What version of Jetty?

The version you wrote, "10.12", doesn't exist.

If you are using Jetty 10.x, please upgrade to Jetty 10.0.16 (there are fixes you are missing for various pooling bugs filed in the past)

Past issues:

  • #8266
  • #8810

joakime avatar Sep 13 '23 16:09 joakime

On top of what @joakime recommended, I would advise you to collect a server dump as that will help understand the buffer pool's internal state.

To mitigate the problem, you could try explicitly configuring a buffer pool with a memory limit by adding a specific pool to the server before starting it. Here's a quick example that configures a pool limited to 1 GB of heap buffers and 1 GB of off-heap buffers:

long oneGb = 1024L * 1024L * 1024L;
LogarithmicArrayByteBufferPool pool = new LogarithmicArrayByteBufferPool(-1, -1, -1, oneGb, oneGb);
server.addBean(pool);

lorban avatar Sep 13 '23 16:09 lorban

Hi @joakime thanks for the quick update and sry for my dismiss. the version is 10.0.12. Also I think this bug may be unrelated to some of recent fix for websockets. And thanks @lorban for the work around, I will take a look for this. Unfortunately it's customer's server so we might not easy for us to collect the server dump

nirvanaemmone avatar Sep 13 '23 21:09 nirvanaemmone

Buffer leaks can happen pretty much anywhere in the code, so we will need some help from your side to reduce the scope of where it might be. Without a way to narrow down where the problem lies, the chances of finding the leak are unfortunately pretty slim.

Getting a server dump is a good first step, knowing which protocol (HTTP? HTTPS? H2? websockets?) triggers the problem would help too.

lorban avatar Sep 13 '23 21:09 lorban

I'm observing this even after upgrading to jetty 10.0.17 using https.

The environment in question does have errors like the following in the logs quite a bit.

2023-12-24T00:02:47.831+0000 [JettyHttpPool-710] ERROR org.glassfish.jersey.server.ServerRuntime$Responder [ServerRuntime.java:writeResponse:662] - An I/O error has occurred while writing a response message entity to the container output stream.
org.glassfish.jersey.server.internal.process.MappableException: org.eclipse.jetty.io.EofException
	at org.glassfish.jersey.server.internal.MappableExceptionWrapperInterceptor.aroundWriteTo(MappableExceptionWrapperInterceptor.java:67)
	at org.glassfish.jersey.message.internal.WriterInterceptorExecutor.proceed(WriterInterceptorExecutor.java:139)
	at org.glassfish.jersey.message.internal.MessageBodyFactory.writeTo(MessageBodyFactory.java:1116)
	at org.glassfish.jersey.server.ServerRuntime$Responder.writeResponse(ServerRuntime.java:635)
	at org.glassfish.jersey.server.ServerRuntime$Responder.processResponse(ServerRuntime.java:373)
	at org.glassfish.jersey.server.ServerRuntime$Responder.process(ServerRuntime.java:363)
	at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:258)
	at org.glassfish.jersey.internal.Errors$1.call(Errors.java:248)
	at org.glassfish.jersey.internal.Errors$1.call(Errors.java:244)
	at org.glassfish.jersey.internal.Errors.process(Errors.java:292)
	at org.glassfish.jersey.internal.Errors.process(Errors.java:274)
	at org.glassfish.jersey.internal.Errors.process(Errors.java:244)
	at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:265)
	at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:234)
	at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:684)
	at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:394)
	at org.glassfish.jersey.servlet.ServletContainer.serviceImpl(ServletContainer.java:386)
	at org.glassfish.jersey.servlet.ServletContainer.doFilter(ServletContainer.java:561)
	at org.glassfish.jersey.servlet.ServletContainer.doFilter(ServletContainer.java:502)
	at org.glassfish.jersey.servlet.ServletContainer.doFilter(ServletContainer.java:439)
	at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
	at com.xgen.svc.brs.slurp.IncorrectContentTypeFilter.doFilter(IncorrectContentTypeFilter.java:30)
	at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
	at com.xgen.svc.mms.res.module.PromJettyPathFilter.doFilter(PromJettyPathFilter.java:70)
	at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
	at com.xgen.svc.mms.res.filter.XContentTypeOptionsFilter.doFilter(XContentTypeOptionsFilter.java:33)
	at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
	at com.xgen.svc.mms.res.filter.XPermittedCrossDomainPoliciesFilter.doFilter(XPermittedCrossDomainPoliciesFilter.java:32)
	at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
	at com.xgen.svc.mms.res.filter.ReferrerPolicyFilter.doFilter(ReferrerPolicyFilter.java:33)
	at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
	at com.xgen.svc.mms.res.filter.HttpStrictTransportSecurityFilter.doFilter(HttpStrictTransportSecurityFilter.java:39)
	at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
	at com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:121)
	at com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:133)
	at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202)
	at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635)
	at com.xgen.svc.mms.res.filter.GzipDecompressFilter.doFilter(GzipDecompressFilter.java:49)
	at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202)
	at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635)
	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:527)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:131)
	at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:598)
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:223)
	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1570)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:131)
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122)
	at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:822)
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:223)
	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1384)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:176)
	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:484)
	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1543)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:174)
	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1306)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:129)
	at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:141)
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122)
	at org.eclipse.jetty.server.Server.handle(Server.java:563)
	at org.eclipse.jetty.server.HttpChannel$RequestDispatchable.dispatch(HttpChannel.java:1598)
	at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:753)
	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:501)
	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:287)
	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:314)
	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:100)
	at org.eclipse.jetty.io.ssl.SslConnection$DecryptedEndPoint.onFillable(SslConnection.java:558)
	at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:379)
	at org.eclipse.jetty.io.ssl.SslConnection$2.succeeded(SslConnection.java:146)
	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:100)
	at org.eclipse.jetty.io.SelectableChannelEndPoint$1.run(SelectableChannelEndPoint.java:53)
	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:969)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.doRunJob(QueuedThreadPool.java:1194)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1149)
	at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: org.eclipse.jetty.io.EofException: null
	at org.eclipse.jetty.io.SocketChannelEndPoint.flush(SocketChannelEndPoint.java:116)
	at org.eclipse.jetty.io.ssl.SslConnection.networkFlush(SslConnection.java:486)
	at org.eclipse.jetty.io.ssl.SslConnection$DecryptedEndPoint.flush(SslConnection.java:1115)
	at org.eclipse.jetty.io.WriteFlusher.flush(WriteFlusher.java:422)
	at org.eclipse.jetty.io.WriteFlusher.write(WriteFlusher.java:275)
	at org.eclipse.jetty.io.WriteFlusher.write(WriteFlusher.java:254)
	at org.eclipse.jetty.io.AbstractEndPoint.write(AbstractEndPoint.java:386)
	at org.eclipse.jetty.server.HttpConnection$SendCallback.process(HttpConnection.java:826)
	at org.eclipse.jetty.util.IteratingCallback.processing(IteratingCallback.java:243)
	at org.eclipse.jetty.util.IteratingCallback.iterate(IteratingCallback.java:224)
	at org.eclipse.jetty.server.HttpConnection.send(HttpConnection.java:589)
	at org.eclipse.jetty.server.HttpChannel.sendResponse(HttpChannel.java:1046)
	at org.eclipse.jetty.server.HttpChannel.write(HttpChannel.java:1123)
	at org.eclipse.jetty.server.handler.gzip.GzipHttpOutputInterceptor.commit(GzipHttpOutputInterceptor.java:168)
	at org.eclipse.jetty.server.handler.gzip.GzipHttpOutputInterceptor.write(GzipHttpOutputInterceptor.java:96)
	at org.eclipse.jetty.server.HttpOutput.channelWrite(HttpOutput.java:270)
	at org.eclipse.jetty.server.HttpOutput.channelWrite(HttpOutput.java:254)
	at org.eclipse.jetty.server.HttpOutput.write(HttpOutput.java:886)
	at java.base/java.io.OutputStream.write(OutputStream.java:122)
	at org.glassfish.jersey.servlet.internal.ResponseWriter$NonCloseableOutputStreamWrapper.write(ResponseWriter.java:296)
	at org.glassfish.jersey.message.internal.CommittingOutputStream.write(CommittingOutputStream.java:189)
	at org.glassfish.jersey.message.internal.WriterInterceptorExecutor$UnCloseableOutputStream.write(WriterInterceptorExecutor.java:271)
	at org.glassfish.jersey.message.internal.ByteArrayProvider.writeTo(ByteArrayProvider.java:73)
	at org.glassfish.jersey.message.internal.ByteArrayProvider.writeTo(ByteArrayProvider.java:37)
	at org.glassfish.jersey.message.internal.WriterInterceptorExecutor$TerminalWriterInterceptor.invokeWriteTo(WriterInterceptorExecutor.java:242)
	at org.glassfish.jersey.message.internal.WriterInterceptorExecutor$TerminalWriterInterceptor.aroundWriteTo(WriterInterceptorExecutor.java:227)
	at org.glassfish.jersey.message.internal.WriterInterceptorExecutor.proceed(WriterInterceptorExecutor.java:139)
	at org.glassfish.jersey.server.internal.JsonWithPaddingInterceptor.aroundWriteTo(JsonWithPaddingInterceptor.java:85)
	at org.glassfish.jersey.message.internal.WriterInterceptorExecutor.proceed(WriterInterceptorExecutor.java:139)
	at org.glassfish.jersey.server.internal.MappableExceptionWrapperInterceptor.aroundWriteTo(MappableExceptionWrapperInterceptor.java:61)
	... 75 common frames omitted
	Suppressed: org.eclipse.jetty.io.EofException: null
		at org.eclipse.jetty.io.SocketChannelEndPoint.flush(SocketChannelEndPoint.java:116)
		at org.eclipse.jetty.io.ssl.SslConnection.networkFlush(SslConnection.java:486)
		at org.eclipse.jetty.io.ssl.SslConnection$DecryptedEndPoint.flush(SslConnection.java:1115)
		at org.eclipse.jetty.io.ssl.SslConnection$DecryptedEndPoint.doShutdownOutput(SslConnection.java:1338)
		at org.eclipse.jetty.io.ssl.SslConnection$DecryptedEndPoint.doClose(SslConnection.java:1434)
		at org.eclipse.jetty.io.AbstractEndPoint.doOnClose(AbstractEndPoint.java:258)
		at org.eclipse.jetty.io.AbstractEndPoint.close(AbstractEndPoint.java:227)
		at org.eclipse.jetty.io.ssl.SslConnection$DecryptedEndPoint.flush(SslConnection.java:1204)
		... 102 common frames omitted
	Caused by: java.io.IOException: Broken pipe
		at java.base/sun.nio.ch.FileDispatcherImpl.writev0(Native Method)
		at java.base/sun.nio.ch.SocketDispatcher.writev(SocketDispatcher.java:51)
		at java.base/sun.nio.ch.IOUtil.write(IOUtil.java:182)
		at java.base/sun.nio.ch.IOUtil.write(IOUtil.java:130)
		at java.base/sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:493)
		at java.base/java.nio.channels.SocketChannel.write(SocketChannel.java:507)
		at org.eclipse.jetty.io.SocketChannelEndPoint.flush(SocketChannelEndPoint.java:110)
		... 109 common frames omitted
Caused by: java.io.IOException: Broken pipe
	at java.base/sun.nio.ch.FileDispatcherImpl.writev0(Native Method)
	at java.base/sun.nio.ch.SocketDispatcher.writev(SocketDispatcher.java:51)
	at java.base/sun.nio.ch.IOUtil.write(IOUtil.java:182)
	at java.base/sun.nio.ch.IOUtil.write(IOUtil.java:130)
	at java.base/sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:493)
	at java.base/java.nio.channels.SocketChannel.write(SocketChannel.java:507)
	at org.eclipse.jetty.io.SocketChannelEndPoint.flush(SocketChannelEndPoint.java:110)
	... 104 common frames omitted

pvertenten avatar Jan 13 '24 01:01 pvertenten

@pvertenten Can you collect more details about the leaks you observed? Any idea if the GZip handler could be related, when it is or it is not inflating/deflating for instance? Unfortunately this stack trace references a too broad context to get a reasonable chance to track down a buffer leak.

As I suggested earlier, a Jetty server dump would help to understand the state of the buffer pool, and a heap dump could help track down where the leak actually is.

Thanks!

lorban avatar Jan 16 '24 15:01 lorban

@pvertenten any update on this? Thanks!

lorban avatar May 21 '24 13:05 lorban

@lorban I'm experiencing a similar issue. I created a separate issue, I suspect my problem could be related: https://github.com/jetty/jetty.project/issues/11858

KGOH avatar May 30 '24 12:05 KGOH

This issue has been automatically marked as stale because it has been a full year without activity. It will be closed if no further activity occurs. Thank you for your contributions.

github-actions[bot] avatar Jun 23 '25 00:06 github-actions[bot]

A buffer leak was found and fixed in 10.0.x/11.0.x, see: https://github.com/jetty/jetty.project/pull/12896

lorban avatar Jun 23 '25 07:06 lorban