async-http-client icon indicating copy to clipboard operation
async-http-client copied to clipboard

The code seems to spam org.asynchttpclient.netty.channel.NettyChannelConnector$1.operationComplete()

Open couldbejake opened this issue 2 years ago • 4 comments

I am opening a large amount of requests using AsyncHttpClient, and seem to run into a large amount of console spam whilst the program is running.

I have noticed that this occurs on Windows, but not on Linux.

Version: Async-http-client version '3.0.0.Beta2'

Preview:

[AsyncHttpClient-9-6] WARN io.netty.util.concurrent.DefaultPromise - An exception was thrown by org.asynchttpclient.netty.channel.NettyChannelConnector$1.operationComplete()
java.lang.NullPointerException: Cannot invoke "org.asynchttpclient.netty.timeout.TimeoutsHolder.setResolvedRemoteAddress(java.net.InetSocketAddress)" because "timeoutsHolder" is null
	at org.asynchttpclient.netty.channel.NettyConnectListener.onSuccess(NettyConnectListener.java:100)
	at org.asynchttpclient.netty.channel.NettyChannelConnector$1.onSuccess(NettyChannelConnector.java:91)
	at org.asynchttpclient.netty.SimpleChannelFutureListener.operationComplete(SimpleChannelFutureListener.java:28)
	at org.asynchttpclient.netty.SimpleChannelFutureListener.operationComplete(SimpleChannelFutureListener.java:22)
	at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590)
	at io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:583)
	at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:559)
	at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492)
	at io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:636)
	at io.netty.util.concurrent.DefaultPromise.setSuccess0(DefaultPromise.java:625)
	at io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:105)
	at io.netty.channel.DefaultChannelPromise.trySuccess(DefaultChannelPromise.java:84)
	at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.fulfillConnectPromise(AbstractNioChannel.java:300)
	at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:335)
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:776)
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724)
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650)
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562)
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	at java.base/java.lang.Thread.run(Thread.java:1589)
[AsyncHttpClient-17-14] WARN io.netty.util.concurrent.DefaultPromise - An exception was thrown by org.asynchttpclient.netty.channel.NettyChannelConnector$1.operationComplete()
java.lang.NullPointerException: Cannot invoke "org.asynchttpclient.netty.timeout.TimeoutsHolder.setResolvedRemoteAddress(java.net.InetSocketAddress)" because "timeoutsHolder" is null
	at org.asynchttpclient.netty.channel.NettyConnectListener.onSuccess(NettyConnectListener.java:100)
	at org.asynchttpclient.netty.channel.NettyChannelConnector$1.onSuccess(NettyChannelConnector.java:91)
	at org.asynchttpclient.netty.SimpleChannelFutureListener.operationComplete(SimpleChannelFutureListener.java:28)
	at org.asynchttpclient.netty.SimpleChannelFutureListener.operationComplete(SimpleChannelFutureListener.java:22)
	at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590)
	at io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:583)
	at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:559)
	at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492)
	at io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:636)
	at io.netty.util.concurrent.DefaultPromise.setSuccess0(DefaultPromise.java:625)
	at io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:105)
	at io.netty.channel.DefaultChannelPromise.trySuccess(DefaultChannelPromise.java:84)
	at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.fulfillConnectPromise(AbstractNioChannel.java:300)
	at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:335)
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:776)
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724)
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650)
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562)
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	at java.base/java.lang.Thread.run(Thread.java:1589)

couldbejake avatar Jun 27 '23 05:06 couldbejake

Is this happening under heavy load?

hyperxpro avatar Jun 27 '23 18:06 hyperxpro

Is this happening under heavy load?

This is indeed happening under heavy load (800+ requests per second). I chose this library as it's performant, but it seems to also causes memory leaks when running continuously. The code https://github.com/Scrapium/scrape-tweets-dev-2 (The memory leak might be caused by the uncaught exception)

couldbejake avatar Jun 28 '23 14:06 couldbejake

It's seems to be due to the way AsyncHttpClient is interacting with the netty framework This library is my only option, as it allows me to set the proxy of the request, rather than the client.

couldbejake avatar Jun 28 '23 16:06 couldbejake

Can you share basic reproducer?

hyperxpro avatar Jun 29 '23 10:06 hyperxpro