OutOfMemoryError is getting during ProxyToServerConnection

[slipwalker]

Getting this error for one of the build with Selenium tests. Tests are running in 35 parallel threads, each test is started in separate docker container, each test has its own proxy. After some time after run the tests, they started failing with this error. Could you please help on this one, why this could happen? Thanks in advance.

build	28-Sep-2017 14:45:21	14:45:21.952 [00192] ERROR [:::] o.l.p.impl.ProxyToServerConnection - (AWAITING_CHUNK) [id: 0x3743b57a, L:/127.0.0.1:33726 - R:localhost/127.0.0.1:8080]: Caught an exception on ProxyToServerConnection
build	28-Sep-2017 14:45:21	java.lang.OutOfMemoryError: Java heap space
build	28-Sep-2017 14:45:21		at java.util.Arrays.copyOf(Arrays.java:3236)
build	28-Sep-2017 14:45:21		at java.io.ByteArrayOutputStream.toByteArray(ByteArrayOutputStream.java:191)
build	28-Sep-2017 14:45:21		at net.lightbody.bmp.util.BrowserMobHttpUtil.decompressContents(BrowserMobHttpUtil.java:99)
build	28-Sep-2017 14:45:21		at net.lightbody.bmp.filters.ServerResponseCaptureFilter.decompressContents(ServerResponseCaptureFilter.java:131)
build	28-Sep-2017 14:45:21		at net.lightbody.bmp.filters.ServerResponseCaptureFilter.captureFullResponseContents(ServerResponseCaptureFilter.java:118)
build	28-Sep-2017 14:45:21		at net.lightbody.bmp.filters.ServerResponseCaptureFilter.serverToProxyResponse(ServerResponseCaptureFilter.java:99)
build	28-Sep-2017 14:45:21		at net.lightbody.bmp.filters.HarCaptureFilter.serverToProxyResponse(HarCaptureFilter.java:250)
build	28-Sep-2017 14:45:21		at net.lightbody.bmp.filters.BrowserMobHttpFilterChain.serverToProxyResponse(BrowserMobHttpFilterChain.java:123)
build	28-Sep-2017 14:45:21		at org.littleshoot.proxy.impl.ClientToProxyConnection.respond(ClientToProxyConnection.java:434)
build	28-Sep-2017 14:45:21		at org.littleshoot.proxy.impl.ProxyToServerConnection.respondWith(ProxyToServerConnection.java:519)
build	28-Sep-2017 14:45:21		at org.littleshoot.proxy.impl.ProxyToServerConnection.readHTTPChunk(ProxyToServerConnection.java:251)
build	28-Sep-2017 14:45:21		at org.littleshoot.proxy.impl.ProxyConnection.readHTTP(ProxyConnection.java:146)
build	28-Sep-2017 14:45:21		at org.littleshoot.proxy.impl.ProxyConnection.read(ProxyConnection.java:120)
build	28-Sep-2017 14:45:21		at org.littleshoot.proxy.impl.ProxyToServerConnection.read(ProxyToServerConnection.java:214)
build	28-Sep-2017 14:45:21		at org.littleshoot.proxy.impl.ProxyConnection.channelRead0(ProxyConnection.java:587)
build	28-Sep-2017 14:45:21		at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105)
build	28-Sep-2017 14:45:21		at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:367)
build	28-Sep-2017 14:45:21		at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:353)
build	28-Sep-2017 14:45:21		at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:346)
build	28-Sep-2017 14:45:21		at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:266)
build	28-Sep-2017 14:45:21		at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:367)
build	28-Sep-2017 14:45:21		at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:353)
build	28-Sep-2017 14:45:21		at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:346)
build	28-Sep-2017 14:45:21		at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86)
build	28-Sep-2017 14:45:21		at org.littleshoot.proxy.impl.ProxyConnection$ResponseReadMonitor.channelRead(ProxyConnection.java:738)
build	28-Sep-2017 14:45:21		at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:367)
build	28-Sep-2017 14:45:21		at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:353)
build	28-Sep-2017 14:45:21		at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:346)
build	28-Sep-2017 14:45:21		at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:293)
build	28-Sep-2017 14:45:21		at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:267)
build	28-Sep-2017 14:45:21		at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:367)
build	28-Sep-2017 14:45:21		at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:353)

[slipwalker]

And similar error:

build	28-Sep-2017 16:19:47	16:19:47.973 [00302] ERROR [:::] o.l.p.impl.ProxyToServerConnection - (AWAITING_CHUNK) [id: 0xe295e6ad, L:/127.0.0.1:41142 - R:localhost/127.0.0.1:8080]: Caught an exception on ProxyToServerConnection
build	28-Sep-2017 16:19:47	java.lang.OutOfMemoryError: Java heap space
build	28-Sep-2017 16:19:47		at java.util.Arrays.copyOf(Arrays.java:3332)
build	28-Sep-2017 16:19:47		at java.lang.StringCoding.safeTrim(StringCoding.java:89)
build	28-Sep-2017 16:19:47		at java.lang.StringCoding.decode(StringCoding.java:230)
build	28-Sep-2017 16:19:47		at java.lang.String.<init>(String.java:463)
build	28-Sep-2017 16:19:47		at java.lang.String.<init>(String.java:515)
build	28-Sep-2017 16:19:47		at net.lightbody.bmp.util.BrowserMobHttpUtil.getContentAsString(BrowserMobHttpUtil.java:169)
build	28-Sep-2017 16:19:47		at net.lightbody.bmp.filters.HarCaptureFilter.captureResponseContent(HarCaptureFilter.java:477)
build	28-Sep-2017 16:19:47		at net.lightbody.bmp.filters.HarCaptureFilter.serverToProxyResponse(HarCaptureFilter.java:267)
build	28-Sep-2017 16:19:47		at net.lightbody.bmp.filters.BrowserMobHttpFilterChain.serverToProxyResponse(BrowserMobHttpFilterChain.java:123)
build	28-Sep-2017 16:19:47		at org.littleshoot.proxy.impl.ClientToProxyConnection.respond(ClientToProxyConnection.java:434)
build	28-Sep-2017 16:19:47		at org.littleshoot.proxy.impl.ProxyToServerConnection.respondWith(ProxyToServerConnection.java:519)
build	28-Sep-2017 16:19:47		at org.littleshoot.proxy.impl.ProxyToServerConnection.readHTTPChunk(ProxyToServerConnection.java:251)
build	28-Sep-2017 16:19:47		at org.littleshoot.proxy.impl.ProxyConnection.readHTTP(ProxyConnection.java:146)
build	28-Sep-2017 16:19:47		at org.littleshoot.proxy.impl.ProxyConnection.read(ProxyConnection.java:120)
build	28-Sep-2017 16:19:47		at org.littleshoot.proxy.impl.ProxyToServerConnection.read(ProxyToServerConnection.java:214)
build	28-Sep-2017 16:19:47		at org.littleshoot.proxy.impl.ProxyConnection.channelRead0(ProxyConnection.java:587)
build	28-Sep-2017 16:19:47		at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105)
build	28-Sep-2017 16:19:47		at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356)
build	28-Sep-2017 16:19:47		at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)
build	28-Sep-2017 16:19:47		at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:335)
build	28-Sep-2017 16:19:47		at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286)
build	28-Sep-2017 16:19:47		at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356)
build	28-Sep-2017 16:19:47		at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)
build	28-Sep-2017 16:19:47		at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:335)
build	28-Sep-2017 16:19:47		at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86)
build	28-Sep-2017 16:19:47		at org.littleshoot.proxy.impl.ProxyConnection$ResponseReadMonitor.channelRead(ProxyConnection.java:738)
build	28-Sep-2017 16:19:47		at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356)
build	28-Sep-2017 16:19:47		at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)
build	28-Sep-2017 16:19:47		at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:335)
build	28-Sep-2017 16:19:47		at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:312)
build	28-Sep-2017 16:19:47		at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:286)
build	28-Sep-2017 16:19:47		at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356)

[eldem]

any solution for this?

[joshbruning]

The decompressed contents of your network traffic does not fit in memory. There are at least a couple of ways to deal with this:

1. Stop capturing response contents for these tests if you don't need/use them.
2. Set -Xmx higher for the java process that is running the proxy. I find -Xmx2g to be sufficient for my own tests.

(It is possible that fetching the har multiple times rather than once at the end would result in lower memory usage; however, I haven't tried that or looked into the implementation.)

If BMP cannot already do it, I would personally like to see an option to skip the decompression step or relying on disk ... or something that could reduce the memory growth/spikes when har capture is enabled.

[Pareshdanej]

Facing the same issue? Do we have any solution for this?