kop [BUG] lookup topic failed (Failed to open a socket...)

[BUG] lookup topic failed (Failed to open a socket...)

Open zbye opened this issue 3 years ago • 3 comments

Describe the bug

Pulsar 2.9.2 + KoP 2.9.2

the broker's log shows some exceptions as follow：

2022-06-28T11:14:19,719+0800 [globalEventExecutor-5-1211] ERROR io.streamnative.pulsar.handlers.kop.KafkaRequestHandler - [[id: 0x19aa8f59, L:/10.187.128.67:9092 - R:/10.189.111.220:58966]] Failed to get partitioned topic persistent://public/default/BD_BD_FUNDFLOW_ORDERS1_JH
org.apache.pulsar.client.admin.PulsarAdminException: java.util.concurrent.CompletionException: org.apache.pulsar.client.admin.internal.http.AsyncHttpConnector$RetryException: Could not complete the operation. Number of retries has been exhausted. Failed reason: Failed to open a socket.
        at org.apache.pulsar.client.admin.internal.BaseResource.getApiException(BaseResource.java:247) ~[org.apache.pulsar-pulsar-client-admin-original-2.9.2.jar:2.9.2]
        at org.apache.pulsar.client.admin.internal.TopicsImpl$7.failed(TopicsImpl.java:529) ~[org.apache.pulsar-pulsar-client-admin-original-2.9.2.jar:2.9.2]
        at org.glassfish.jersey.client.JerseyInvocation$1.failed(JerseyInvocation.java:882) ~[org.glassfish.jersey.core-jersey-client-2.34.jar:?]
        at org.glassfish.jersey.client.ClientRuntime.processFailure(ClientRuntime.java:247) ~[org.glassfish.jersey.core-jersey-client-2.34.jar:?]
        at org.glassfish.jersey.client.ClientRuntime.processFailure(ClientRuntime.java:242) ~[org.glassfish.jersey.core-jersey-client-2.34.jar:?]
        at org.glassfish.jersey.client.ClientRuntime.access$100(ClientRuntime.java:62) ~[org.glassfish.jersey.core-jersey-client-2.34.jar:?]
        at org.glassfish.jersey.client.ClientRuntime$2.lambda$failure$1(ClientRuntime.java:178) ~[org.glassfish.jersey.core-jersey-client-2.34.jar:?]
        at org.glassfish.jersey.internal.Errors$1.call(Errors.java:248) ~[org.glassfish.jersey.core-jersey-common-2.34.jar:?]
        at org.glassfish.jersey.internal.Errors$1.call(Errors.java:244) ~[org.glassfish.jersey.core-jersey-common-2.34.jar:?]
        at org.glassfish.jersey.internal.Errors.process(Errors.java:292) ~[org.glassfish.jersey.core-jersey-common-2.34.jar:?]
        at org.glassfish.jersey.internal.Errors.process(Errors.java:274) ~[org.glassfish.jersey.core-jersey-common-2.34.jar:?]
        at org.glassfish.jersey.internal.Errors.process(Errors.java:244) ~[org.glassfish.jersey.core-jersey-common-2.34.jar:?]
        at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:288) ~[org.glassfish.jersey.core-jersey-common-2.34.jar:?]
        at org.glassfish.jersey.client.ClientRuntime$2.failure(ClientRuntime.java:178) ~[org.glassfish.jersey.core-jersey-client-2.34.jar:?]
        at org.apache.pulsar.client.admin.internal.http.AsyncHttpConnector.lambda$apply$1(AsyncHttpConnector.java:204) ~[org.apache.pulsar-pulsar-client-admin-original-2.9.2.jar:2.9.2]
        at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) ~[?:1.8.0_131]
        at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) ~[?:1.8.0_131]
        at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) ~[?:1.8.0_131]
        at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1977) ~[?:1.8.0_131]
        at org.apache.pulsar.client.admin.internal.http.AsyncHttpConnector.lambda$retryOperation$4(AsyncHttpConnector.java:263) ~[org.apache.pulsar-pulsar-client-admin-original-2.9.2.jar:2.9.2]
        at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) ~[?:1.8.0_131]
        at java.util.concurrent.CompletableFuture.uniWhenCompleteStage(CompletableFuture.java:778) ~[?:1.8.0_131]
        at java.util.concurrent.CompletableFuture.whenComplete(CompletableFuture.java:2140) ~[?:1.8.0_131]
        at org.apache.pulsar.client.admin.internal.http.AsyncHttpConnector.retryOperation(AsyncHttpConnector.java:250) ~[org.apache.pulsar-pulsar-client-admin-original-2.9.2.jar:2.9.2]
        at org.apache.pulsar.client.admin.internal.http.AsyncHttpConnector.lambda$retryOperation$4(AsyncHttpConnector.java:258) ~[org.apache.pulsar-pulsar-client-admin-original-2.9.2.jar:2.9.2]
        at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) ~[?:1.8.0_131]
        at java.util.concurrent.CompletableFuture.uniWhenCompleteStage(CompletableFuture.java:778) ~[?:1.8.0_131]
        at java.util.concurrent.CompletableFuture.whenComplete(CompletableFuture.java:2140) ~[?:1.8.0_131]
        at org.apache.pulsar.client.admin.internal.http.AsyncHttpConnector.retryOperation(AsyncHttpConnector.java:250) ~[org.apache.pulsar-pulsar-client-admin-original-2.9.2.jar:2.9.2]
        at org.apache.pulsar.client.admin.internal.http.AsyncHttpConnector.lambda$retryOperation$4(AsyncHttpConnector.java:258) ~[org.apache.pulsar-pulsar-client-admin-original-2.9.2.jar:2.9.2]
        at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) ~[?:1.8.0_131]
        at java.util.concurrent.CompletableFuture.uniWhenCompleteStage(CompletableFuture.java:778) ~[?:1.8.0_131]
        at java.util.concurrent.CompletableFuture.whenComplete(CompletableFuture.java:2140) ~[?:1.8.0_131]
        at org.apache.pulsar.client.admin.internal.http.AsyncHttpConnector.retryOperation(AsyncHttpConnector.java:250) ~[org.apache.pulsar-pulsar-client-admin-original-2.9.2.jar:2.9.2]
        at org.apache.pulsar.client.admin.internal.http.AsyncHttpConnector.lambda$retryOperation$4(AsyncHttpConnector.java:258) ~[org.apache.pulsar-pulsar-client-admin-original-2.9.2.jar:2.9.2]
        at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) ~[?:1.8.0_131]
        at java.util.concurrent.CompletableFuture.uniWhenCompleteStage(CompletableFuture.java:778) ~[?:1.8.0_131]
        at java.util.concurrent.CompletableFuture.whenComplete(CompletableFuture.java:2140) ~[?:1.8.0_131]
        at org.apache.pulsar.client.admin.internal.http.AsyncHttpConnector.retryOperation(AsyncHttpConnector.java:250) ~[org.apache.pulsar-pulsar-client-admin-original-2.9.2.jar:2.9.2]
        at org.apache.pulsar.client.admin.internal.http.AsyncHttpConnector.lambda$retryOperation$4(AsyncHttpConnector.java:258) ~[org.apache.pulsar-pulsar-client-admin-original-2.9.2.jar:2.9.2]
        at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) ~[?:1.8.0_131]
        at java.util.concurrent.CompletableFuture.uniWhenCompleteStage(CompletableFuture.java:778) ~[?:1.8.0_131]
        at java.util.concurrent.CompletableFuture.whenComplete(CompletableFuture.java:2140) ~[?:1.8.0_131]
        at org.apache.pulsar.client.admin.internal.http.AsyncHttpConnector.retryOperation(AsyncHttpConnector.java:250) ~[org.apache.pulsar-pulsar-client-admin-original-2.9.2.jar:2.9.2]
        at org.apache.pulsar.client.admin.internal.http.AsyncHttpConnector.lambda$retryOperation$4(AsyncHttpConnector.java:258) ~[org.apache.pulsar-pulsar-client-admin-original-2.9.2.jar:2.9.2]
        at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) [?:1.8.0_131]
        at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) [?:1.8.0_131]
        at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) [?:1.8.0_131]
        at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1977) [?:1.8.0_131]
        at org.asynchttpclient.netty.NettyResponseFuture.abort(NettyResponseFuture.java:273) [org.asynchttpclient-async-http-client-2.12.1.jar:?]
        at org.asynchttpclient.netty.channel.NettyConnectListener.onFailure(NettyConnectListener.java:181) [org.asynchttpclient-async-http-client-2.12.1.jar:?]
        at org.asynchttpclient.netty.channel.NettyChannelConnector$1.onFailure(NettyChannelConnector.java:108) [org.asynchttpclient-async-http-client-2.12.1.jar:?]
        at org.asynchttpclient.netty.SimpleChannelFutureListener.operationComplete(SimpleChannelFutureListener.java:28) [org.asynchttpclient-async-http-client-2.12.1.jar:?]
        at org.asynchttpclient.netty.SimpleChannelFutureListener.operationComplete(SimpleChannelFutureListener.java:20) [org.asynchttpclient-async-http-client-2.12.1.jar:?]
        at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:578) [io.netty-netty-common-4.1.74.Final.jar:4.1.74.Final]
        at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:552) [io.netty-netty-common-4.1.74.Final.jar:4.1.74.Final]
        at io.netty.util.concurrent.DefaultPromise.access$200(DefaultPromise.java:35) [io.netty-netty-common-4.1.74.Final.jar:4.1.74.Final]
        at io.netty.util.concurrent.DefaultPromise$1.run(DefaultPromise.java:502) [io.netty-netty-common-4.1.74.Final.jar:4.1.74.Final]
        at io.netty.util.concurrent.GlobalEventExecutor$TaskRunner.run(GlobalEventExecutor.java:240) [io.netty-netty-common-4.1.74.Final.jar:4.1.74.Final]
        at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [io.netty-netty-common-4.1.74.Final.jar:4.1.74.Final]
        at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [io.netty-netty-common-4.1.74.Final.jar:4.1.74.Final]
        at java.lang.Thread.run(Thread.java:748) [?:1.8.0_131]
Caused by: java.util.concurrent.CompletionException: org.apache.pulsar.client.admin.internal.http.AsyncHttpConnector$RetryException: Could not complete the operation. Number of retries has been exhausted. Failed reason: Failed to open a socket.
        at java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:292) ~[?:1.8.0_131]
        at java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:308) ~[?:1.8.0_131]
        at java.util.concurrent.CompletableFuture.orApply(CompletableFuture.java:1371) ~[?:1.8.0_131]
        at java.util.concurrent.CompletableFuture$OrApply.tryFire(CompletableFuture.java:1350) ~[?:1.8.0_131]
        ... 45 more
Caused by: org.apache.pulsar.client.admin.internal.http.AsyncHttpConnector$RetryException: Could not complete the operation. Number of retries has been exhausted. Failed reason: Failed to open a socket.
        at org.apache.pulsar.client.admin.internal.http.AsyncHttpConnector.lambda$retryOperation$4(AsyncHttpConnector.java:265) ~[org.apache.pulsar-pulsar-client-admin-original-2.9.2.jar:2.9.2]
        ... 42 more
Caused by: java.net.ConnectException: Failed to open a socket.
        at org.asynchttpclient.netty.channel.NettyConnectListener.onFailure(NettyConnectListener.java:179) ~[org.asynchttpclient-async-http-client-2.12.1.jar:?]
        at org.asynchttpclient.netty.channel.NettyChannelConnector$1.onFailure(NettyChannelConnector.java:108) ~[org.asynchttpclient-async-http-client-2.12.1.jar:?]
        at org.asynchttpclient.netty.SimpleChannelFutureListener.operationComplete(SimpleChannelFutureListener.java:28) ~[org.asynchttpclient-async-http-client-2.12.1.jar:?]
        at org.asynchttpclient.netty.SimpleChannelFutureListener.operationComplete(SimpleChannelFutureListener.java:20) ~[org.asynchttpclient-async-http-client-2.12.1.jar:?]
        at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:578) ~[io.netty-netty-common-4.1.74.Final.jar:4.1.74.Final]
        at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:552) ~[io.netty-netty-common-4.1.74.Final.jar:4.1.74.Final]
        at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:491) ~[io.netty-netty-common-4.1.74.Final.jar:4.1.74.Final]
        at io.netty.util.concurrent.DefaultPromise.addListener(DefaultPromise.java:184) ~[io.netty-netty-common-4.1.74.Final.jar:4.1.74.Final]
        at io.netty.channel.DefaultChannelPromise.addListener(DefaultChannelPromise.java:95) ~[io.netty-netty-transport-4.1.74.Final.jar:4.1.74.Final]
        at io.netty.channel.DefaultChannelPromise.addListener(DefaultChannelPromise.java:30) ~[io.netty-netty-transport-4.1.74.Final.jar:4.1.74.Final]
        at org.asynchttpclient.netty.channel.NettyChannelConnector.connect0(NettyChannelConnector.java:82) ~[org.asynchttpclient-async-http-client-2.12.1.jar:?]
        at org.asynchttpclient.netty.channel.NettyChannelConnector.connect(NettyChannelConnector.java:69) ~[org.asynchttpclient-async-http-client-2.12.1.jar:?]
        at org.asynchttpclient.netty.request.NettyRequestSender$1.lambda$onSuccess$0(NettyRequestSender.java:315) ~[org.asynchttpclient-async-http-client-2.12.1.jar:?]
        at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:578) ~[io.netty-netty-common-4.1.74.Final.jar:4.1.74.Final]
        at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:552) ~[io.netty-netty-common-4.1.74.Final.jar:4.1.74.Final]
        at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:491) ~[io.netty-netty-common-4.1.74.Final.jar:4.1.74.Final]
        at io.netty.util.concurrent.DefaultPromise.addListener(DefaultPromise.java:184) ~[io.netty-netty-common-4.1.74.Final.jar:4.1.74.Final]
        at io.netty.util.concurrent.DefaultPromise.addListener(DefaultPromise.java:35) ~[io.netty-netty-common-4.1.74.Final.jar:4.1.74.Final]
        at org.asynchttpclient.netty.request.NettyRequestSender$1.onSuccess(NettyRequestSender.java:313) ~[org.asynchttpclient-async-http-client-2.12.1.jar:?]
        at org.asynchttpclient.netty.request.NettyRequestSender$1.onSuccess(NettyRequestSender.java:301) ~[org.asynchttpclient-async-http-client-2.12.1.jar:?]
        at org.asynchttpclient.netty.SimpleFutureListener.operationComplete(SimpleFutureListener.java:24) ~[org.asynchttpclient-async-http-client-2.12.1.jar:?]
        at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:578) ~[io.netty-netty-common-4.1.74.Final.jar:4.1.74.Final]
        at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:552) ~[io.netty-netty-common-4.1.74.Final.jar:4.1.74.Final]
        at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:491) ~[io.netty-netty-common-4.1.74.Final.jar:4.1.74.Final]
        at io.netty.util.concurrent.DefaultPromise.addListener(DefaultPromise.java:184) ~[io.netty-netty-common-4.1.74.Final.jar:4.1.74.Final]
        at io.netty.util.concurrent.DefaultPromise.addListener(DefaultPromise.java:35) ~[io.netty-netty-common-4.1.74.Final.jar:4.1.74.Final]
        at org.asynchttpclient.netty.request.NettyRequestSender.sendRequestWithNewChannel(NettyRequestSender.java:301) ~[org.asynchttpclient-async-http-client-2.12.1.jar:?]
        at org.asynchttpclient.netty.request.NettyRequestSender.sendRequestWithCertainForceConnect(NettyRequestSender.java:142) ~[org.asynchttpclient-async-http-client-2.12.1.jar:?]
        at org.asynchttpclient.netty.request.NettyRequestSender.sendRequest(NettyRequestSender.java:113) ~[org.asynchttpclient-async-http-client-2.12.1.jar:?]
        at org.asynchttpclient.DefaultAsyncHttpClient.execute(DefaultAsyncHttpClient.java:259) ~[org.asynchttpclient-async-http-client-2.12.1.jar:?]
        at org.asynchttpclient.DefaultAsyncHttpClient.executeRequest(DefaultAsyncHttpClient.java:228) ~[org.asynchttpclient-async-http-client-2.12.1.jar:?]
        at org.asynchttpclient.BoundRequestBuilder.execute(BoundRequestBuilder.java:39) ~[org.asynchttpclient-async-http-client-2.12.1.jar:?]
        at org.apache.pulsar.client.admin.internal.http.AsyncHttpConnector.oneShot(AsyncHttpConnector.java:316) ~[org.apache.pulsar-pulsar-client-admin-original-2.9.2.jar:2.9.2]
        at org.apache.pulsar.client.admin.internal.http.AsyncHttpConnector.lambda$retryOrTimeOut$2(AsyncHttpConnector.java:236) ~[org.apache.pulsar-pulsar-client-admin-original-2.9.2.jar:2.9.2]
        at org.apache.pulsar.client.admin.internal.http.AsyncHttpConnector.retryOperation(AsyncHttpConnector.java:248) ~[org.apache.pulsar-pulsar-client-admin-original-2.9.2.jar:2.9.2]
        ... 38 more
Caused by: io.netty.channel.ChannelException: Failed to open a socket.
        at io.netty.channel.socket.nio.NioSocketChannel.newSocket(NioSocketChannel.java:71) ~[io.netty-netty-transport-4.1.74.Final.jar:4.1.74.Final]
        at io.netty.channel.socket.nio.NioSocketChannel.<init>(NioSocketChannel.java:88) ~[io.netty-netty-transport-4.1.74.Final.jar:4.1.74.Final]
        at io.netty.channel.socket.nio.NioSocketChannel.<init>(NioSocketChannel.java:81) ~[io.netty-netty-transport-4.1.74.Final.jar:4.1.74.Final]
        at org.asynchttpclient.netty.channel.NioTransportFactory.newChannel(NioTransportFactory.java:27) ~[org.asynchttpclient-async-http-client-2.12.1.jar:?]
        at org.asynchttpclient.netty.channel.NioTransportFactory.newChannel(NioTransportFactory.java:21) ~[org.asynchttpclient-async-http-client-2.12.1.jar:?]
        at io.netty.bootstrap.AbstractBootstrap.initAndRegister(AbstractBootstrap.java:310) ~[io.netty-netty-transport-4.1.74.Final.jar:4.1.74.Final]
        at io.netty.bootstrap.Bootstrap.doResolveAndConnect(Bootstrap.java:155) ~[io.netty-netty-transport-4.1.74.Final.jar:4.1.74.Final]
        at io.netty.bootstrap.Bootstrap.connect(Bootstrap.java:148) ~[io.netty-netty-transport-4.1.74.Final.jar:4.1.74.Final]
        at org.asynchttpclient.netty.channel.NettyChannelConnector.connect0(NettyChannelConnector.java:81) ~[org.asynchttpclient-async-http-client-2.12.1.jar:?]
        at org.asynchttpclient.netty.channel.NettyChannelConnector.connect(NettyChannelConnector.java:69) ~[org.asynchttpclient-async-http-client-2.12.1.jar:?]
        at org.asynchttpclient.netty.request.NettyRequestSender$1.lambda$onSuccess$0(NettyRequestSender.java:315) ~[org.asynchttpclient-async-http-client-2.12.1.jar:?]
        at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:578) ~[io.netty-netty-common-4.1.74.Final.jar:4.1.74.Final]
        at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:552) ~[io.netty-netty-common-4.1.74.Final.jar:4.1.74.Final]
        at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:491) ~[io.netty-netty-common-4.1.74.Final.jar:4.1.74.Final]
        at io.netty.util.concurrent.DefaultPromise.addListener(DefaultPromise.java:184) ~[io.netty-netty-common-4.1.74.Final.jar:4.1.74.Final]
        at io.netty.util.concurrent.DefaultPromise.addListener(DefaultPromise.java:35) ~[io.netty-netty-common-4.1.74.Final.jar:4.1.74.Final]
        at org.asynchttpclient.netty.request.NettyRequestSender$1.onSuccess(NettyRequestSender.java:313) ~[org.asynchttpclient-async-http-client-2.12.1.jar:?]
        at org.asynchttpclient.netty.request.NettyRequestSender$1.onSuccess(NettyRequestSender.java:301) ~[org.asynchttpclient-async-http-client-2.12.1.jar:?]
        at org.asynchttpclient.netty.SimpleFutureListener.operationComplete(SimpleFutureListener.java:24) ~[org.asynchttpclient-async-http-client-2.12.1.jar:?]
        at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:578) ~[io.netty-netty-common-4.1.74.Final.jar:4.1.74.Final]
        at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:552) ~[io.netty-netty-common-4.1.74.Final.jar:4.1.74.Final]
        at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:491) ~[io.netty-netty-common-4.1.74.Final.jar:4.1.74.Final]
        at io.netty.util.concurrent.DefaultPromise.addListener(DefaultPromise.java:184) ~[io.netty-netty-common-4.1.74.Final.jar:4.1.74.Final]
        at io.netty.util.concurrent.DefaultPromise.addListener(DefaultPromise.java:35) ~[io.netty-netty-common-4.1.74.Final.jar:4.1.74.Final]
        at org.asynchttpclient.netty.request.NettyRequestSender.sendRequestWithNewChannel(NettyRequestSender.java:301) ~[org.asynchttpclient-async-http-client-2.12.1.jar:?]
        at org.asynchttpclient.netty.request.NettyRequestSender.sendRequestWithCertainForceConnect(NettyRequestSender.java:142) ~[org.asynchttpclient-async-http-client-2.12.1.jar:?]
        at org.asynchttpclient.netty.request.NettyRequestSender.sendRequest(NettyRequestSender.java:113) ~[org.asynchttpclient-async-http-client-2.12.1.jar:?]
        at org.asynchttpclient.DefaultAsyncHttpClient.execute(DefaultAsyncHttpClient.java:259) ~[org.asynchttpclient-async-http-client-2.12.1.jar:?]
        at org.asynchttpclient.DefaultAsyncHttpClient.executeRequest(DefaultAsyncHttpClient.java:228) ~[org.asynchttpclient-async-http-client-2.12.1.jar:?]
        at org.asynchttpclient.BoundRequestBuilder.execute(BoundRequestBuilder.java:39) ~[org.asynchttpclient-async-http-client-2.12.1.jar:?]
        at org.apache.pulsar.client.admin.internal.http.AsyncHttpConnector.oneShot(AsyncHttpConnector.java:316) ~[org.apache.pulsar-pulsar-client-admin-original-2.9.2.jar:2.9.2]
        at org.apache.pulsar.client.admin.internal.http.AsyncHttpConnector.lambda$retryOrTimeOut$2(AsyncHttpConnector.java:236) ~[org.apache.pulsar-pulsar-client-admin-original-2.9.2.jar:2.9.2]
        at org.apache.pulsar.client.admin.internal.http.AsyncHttpConnector.retryOperation(AsyncHttpConnector.java:248) ~[org.apache.pulsar-pulsar-client-admin-original-2.9.2.jar:2.9.2]
        ... 38 more
Caused by: java.net.SocketException: 打开的文件过多
        at sun.nio.ch.Net.socket0(Native Method) ~[?:1.8.0_131]
        at sun.nio.ch.Net.socket(Net.java:411) ~[?:1.8.0_131]
        at sun.nio.ch.Net.socket(Net.java:404) ~[?:1.8.0_131]
        at sun.nio.ch.SocketChannelImpl.<init>(SocketChannelImpl.java:105) ~[?:1.8.0_131]
        at sun.nio.ch.SelectorProviderImpl.openSocketChannel(SelectorProviderImpl.java:60) ~[?:1.8.0_131]
        at io.netty.channel.socket.nio.NioSocketChannel.newSocket(NioSocketChannel.java:69) ~[io.netty-netty-transport-4.1.74.Final.jar:4.1.74.Final]
        at io.netty.channel.socket.nio.NioSocketChannel.<init>(NioSocketChannel.java:88) ~[io.netty-netty-transport-4.1.74.Final.jar:4.1.74.Final]
        at io.netty.channel.socket.nio.NioSocketChannel.<init>(NioSocketChannel.java:81) ~[io.netty-netty-transport-4.1.74.Final.jar:4.1.74.Final]
        at org.asynchttpclient.netty.channel.NioTransportFactory.newChannel(NioTransportFactory.java:27) ~[org.asynchttpclient-async-http-client-2.12.1.jar:?]
        at org.asynchttpclient.netty.channel.NioTransportFactory.newChannel(NioTransportFactory.java:21) ~[org.asynchttpclient-async-http-client-2.12.1.jar:?]
        at io.netty.bootstrap.AbstractBootstrap.initAndRegister(AbstractBootstrap.java:310) ~[io.netty-netty-transport-4.1.74.Final.jar:4.1.74.Final]
        at io.netty.bootstrap.Bootstrap.doResolveAndConnect(Bootstrap.java:155) ~[io.netty-netty-transport-4.1.74.Final.jar:4.1.74.Final]
        at io.netty.bootstrap.Bootstrap.connect(Bootstrap.java:148) ~[io.netty-netty-transport-4.1.74.Final.jar:4.1.74.Final]
        at org.asynchttpclient.netty.channel.NettyChannelConnector.connect0(NettyChannelConnector.java:81) ~[org.asynchttpclient-async-http-client-2.12.1.jar:?]
        at org.asynchttpclient.netty.channel.NettyChannelConnector.connect(NettyChannelConnector.java:69) ~[org.asynchttpclient-async-http-client-2.12.1.jar:?]
        at org.asynchttpclient.netty.request.NettyRequestSender$1.lambda$onSuccess$0(NettyRequestSender.java:315) ~[org.asynchttpclient-async-http-client-2.12.1.jar:?]
        at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:578) ~[io.netty-netty-common-4.1.74.Final.jar:4.1.74.Final]
        at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:552) ~[io.netty-netty-common-4.1.74.Final.jar:4.1.74.Final]
        at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:491) ~[io.netty-netty-common-4.1.74.Final.jar:4.1.74.Final]
        at io.netty.util.concurrent.DefaultPromise.addListener(DefaultPromise.java:184) ~[io.netty-netty-common-4.1.74.Final.jar:4.1.74.Final]
        at io.netty.util.concurrent.DefaultPromise.addListener(DefaultPromise.java:35) ~[io.netty-netty-common-4.1.74.Final.jar:4.1.74.Final]
        at org.asynchttpclient.netty.request.NettyRequestSender$1.onSuccess(NettyRequestSender.java:313) ~[org.asynchttpclient-async-http-client-2.12.1.jar:?]
        at org.asynchttpclient.netty.request.NettyRequestSender$1.onSuccess(NettyRequestSender.java:301) ~[org.asynchttpclient-async-http-client-2.12.1.jar:?]
        at org.asynchttpclient.netty.SimpleFutureListener.operationComplete(SimpleFutureListener.java:24) ~[org.asynchttpclient-async-http-client-2.12.1.jar:?]
        at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:578) ~[io.netty-netty-common-4.1.74.Final.jar:4.1.74.Final]
        at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:552) ~[io.netty-netty-common-4.1.74.Final.jar:4.1.74.Final]
        at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:491) ~[io.netty-netty-common-4.1.74.Final.jar:4.1.74.Final]
        at io.netty.util.concurrent.DefaultPromise.addListener(DefaultPromise.java:184) ~[io.netty-netty-common-4.1.74.Final.jar:4.1.74.Final]
        at io.netty.util.concurrent.DefaultPromise.addListener(DefaultPromise.java:35) ~[io.netty-netty-common-4.1.74.Final.jar:4.1.74.Final]
        at org.asynchttpclient.netty.request.NettyRequestSender.sendRequestWithNewChannel(NettyRequestSender.java:301) ~[org.asynchttpclient-async-http-client-2.12.1.jar:?]
        at org.asynchttpclient.netty.request.NettyRequestSender.sendRequestWithCertainForceConnect(NettyRequestSender.java:142) ~[org.asynchttpclient-async-http-client-2.12.1.jar:?]
        at org.asynchttpclient.netty.request.NettyRequestSender.sendRequest(NettyRequestSender.java:113) ~[org.asynchttpclient-async-http-client-2.12.1.jar:?]
        at org.asynchttpclient.DefaultAsyncHttpClient.execute(DefaultAsyncHttpClient.java:259) ~[org.asynchttpclient-async-http-client-2.12.1.jar:?]
        at org.asynchttpclient.DefaultAsyncHttpClient.executeRequest(DefaultAsyncHttpClient.java:228) ~[org.asynchttpclient-async-http-client-2.12.1.jar:?]
        at org.asynchttpclient.BoundRequestBuilder.execute(BoundRequestBuilder.java:39) ~[org.asynchttpclient-async-http-client-2.12.1.jar:?]
        at org.apache.pulsar.client.admin.internal.http.AsyncHttpConnector.oneShot(AsyncHttpConnector.java:316) ~[org.apache.pulsar-pulsar-client-admin-original-2.9.2.jar:2.9.2]
        at org.apache.pulsar.client.admin.internal.http.AsyncHttpConnector.lambda$retryOrTimeOut$2(AsyncHttpConnector.java:236) ~[org.apache.pulsar-pulsar-client-admin-original-2.9.2.jar:2.9.2]
        at org.apache.pulsar.client.admin.internal.http.AsyncHttpConnector.retryOperation(AsyncHttpConnector.java:248) ~[org.apache.pulsar-pulsar-client-admin-original-2.9.2.jar:2.9.2]
        ... 38 more
2022-06-28T11:14:19,810+0800 [pulsar-acceptor-21-1] WARN  io.netty.channel.DefaultChannelPipeline - An exceptionCaught() event was fired, and it reached at the tail of the pipeline. It usually means the last handler in the pipeline did not handle the exception.
io.netty.channel.unix.Errors$NativeIoException: accept(..) failed: 打开的文件过多
2022-06-28T11:14:19,825+0800 [globalEventExecutor-5-1211] ERROR io.streamnative.pulsar.handlers.kop.KafkaRequestHandler - [[id: 0x19aa8f59, L:/10.187.128.67:9092 - R:/10.189.111.220:58966]] Failed to get partitioned topic persistent://public/default/BD_BD_FUNDFLOW_ORDERS1_JH
org.apache.pulsar.client.admin.PulsarAdminException: java.util.concurrent.CompletionException: org.apache.pulsar.client.admin.internal.http.AsyncHttpConnector$RetryException: Could not complete the operation. Number of retries has been exhausted. Failed reason: Failed to open a socket.
        at org.apache.pulsar.client.admin.internal.BaseResource.getApiException(BaseResource.java:247) ~[org.apache.pulsar-pulsar-client-admin-original-2.9.2.jar:2.9.2]
        at org.apache.pulsar.client.admin.internal.TopicsImpl$7.failed(TopicsImpl.java:529) ~[org.apache.pulsar-pulsar-client-admin-original-2.9.2.jar:2.9.2]
        at org.glassfish.jersey.client.JerseyInvocation$1.failed(JerseyInvocation.java:882) ~[org.glassfish.jersey.core-jersey-client-2.34.jar:?]
        at org.glassfish.jersey.client.ClientRuntime.processFailure(ClientRuntime.java:247) ~[org.glassfish.jersey.core-jersey-client-2.34.jar:?]
        at org.glassfish.jersey.client.ClientRuntime.processFailure(ClientRuntime.java:242) ~[org.glassfish.jersey.core-jersey-client-2.34.jar:?]
        at org.glassfish.jersey.client.ClientRuntime.access$100(ClientRuntime.java:62) ~[org.glassfish.jersey.core-jersey-client-2.34.jar:?]
        at org.glassfish.jersey.client.ClientRuntime$2.lambda$failure$1(ClientRuntime.java:178) ~[org.glassfish.jersey.core-jersey-client-2.34.jar:?]
        at org.glassfish.jersey.internal.Errors$1.call(Errors.java:248) ~[org.glassfish.jersey.core-jersey-common-2.34.jar:?]
        at org.glassfish.jersey.internal.Errors$1.call(Errors.java:244) ~[org.glassfish.jersey.core-jersey-common-2.34.jar:?]
        at org.glassfish.jersey.internal.Errors.process(Errors.java:292) ~[org.glassfish.jersey.core-jersey-common-2.34.jar:?]
        at org.glassfish.jersey.internal.Errors.process(Errors.java:274) ~[org.glassfish.jersey.core-jersey-common-2.34.jar:?]
        at org.glassfish.jersey.internal.Errors.process(Errors.java:244) ~[org.glassfish.jersey.core-jersey-common-2.34.jar:?]
        at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:288) ~[org.glassfish.jersey.core-jersey-common-2.34.jar:?]
        at org.glassfish.jersey.client.ClientRuntime$2.failure(ClientRuntime.java:178) ~[org.glassfish.jersey.core-jersey-client-2.34.jar:?]
        at org.apache.pulsar.client.admin.internal.http.AsyncHttpConnector.lambda$apply$1(AsyncHttpConnector.java:204) ~[org.apache.pulsar-pulsar-client-admin-original-2.9.2.jar:2.9.2]
        at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) ~[?:1.8.0_131]
        at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) ~[?:1.8.0_131]
        at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) ~[?:1.8.0_131]
        at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1977) ~[?:1.8.0_131]
        at org.apache.pulsar.client.admin.internal.http.AsyncHttpConnector.lambda$retryOperation$4(AsyncHttpConnector.java:263) ~[org.apache.pulsar-pulsar-client-admin-original-2.9.2.jar:2.9.2]
        at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) ~[?:1.8.0_131]
        at java.util.concurrent.CompletableFuture.uniWhenCompleteStage(CompletableFuture.java:778) ~[?:1.8.0_131]
        at java.util.concurrent.CompletableFuture.whenComplete(CompletableFuture.java:2140) ~[?:1.8.0_131]
        at org.apache.pulsar.client.admin.internal.http.AsyncHttpConnector.retryOperation(AsyncHttpConnector.java:250) ~[org.apache.pulsar-pulsar-client-admin-original-2.9.2.jar:2.9.2]
        at org.apache.pulsar.client.admin.internal.http.AsyncHttpConnector.lambda$retryOperation$4(AsyncHttpConnector.java:258) ~[org.apache.pulsar-pulsar-client-admin-original-2.9.2.jar:2.9.2]
        at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) ~[?:1.8.0_131]
        at java.util.concurrent.CompletableFuture.uniWhenCompleteStage(CompletableFuture.java:778) ~[?:1.8.0_131]
        at java.util.concurrent.CompletableFuture.whenComplete(CompletableFuture.java:2140) ~[?:1.8.0_131]
        at org.apache.pulsar.client.admin.internal.http.AsyncHttpConnector.retryOperation(AsyncHttpConnector.java:250) ~[org.apache.pulsar-pulsar-client-admin-original-2.9.2.jar:2.9.2]
        at org.apache.pulsar.client.admin.internal.http.AsyncHttpConnector.lambda$retryOperation$4(AsyncHttpConnector.java:258) ~[org.apache.pulsar-pulsar-client-admin-original-2.9.2.jar:2.9.2]
        at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) ~[?:1.8.0_131]
        at java.util.concurrent.CompletableFuture.uniWhenCompleteStage(CompletableFuture.java:778) ~[?:1.8.0_131]
        at java.util.concurrent.CompletableFuture.whenComplete(CompletableFuture.java:2140) ~[?:1.8.0_131]
        at org.apache.pulsar.client.admin.internal.http.AsyncHttpConnector.retryOperation(AsyncHttpConnector.java:250) ~[org.apache.pulsar-pulsar-client-admin-original-2.9.2.jar:2.9.2]
        at org.apache.pulsar.client.admin.internal.http.AsyncHttpConnector.lambda$retryOperation$4(AsyncHttpConnector.java:258) ~[org.apache.pulsar-pulsar-client-admin-original-2.9.2.jar:2.9.2]
        at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) ~[?:1.8.0_131]
        at java.util.concurrent.CompletableFuture.uniWhenCompleteStage(CompletableFuture.java:778) ~[?:1.8.0_131]
        at java.util.concurrent.CompletableFuture.whenComplete(CompletableFuture.java:2140) ~[?:1.8.0_131]
        at org.apache.pulsar.client.admin.internal.http.AsyncHttpConnector.retryOperation(AsyncHttpConnector.java:250) ~[org.apache.pulsar-pulsar-client-admin-original-2.9.2.jar:2.9.2]
        at org.apache.pulsar.client.admin.internal.http.AsyncHttpConnector.lambda$retryOperation$4(AsyncHttpConnector.java:258) ~[org.apache.pulsar-pulsar-client-admin-original-2.9.2.jar:2.9.2]
        at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) ~[?:1.8.0_131]
        at java.util.concurrent.CompletableFuture.uniWhenCompleteStage(CompletableFuture.java:778) ~[?:1.8.0_131]
        at java.util.concurrent.CompletableFuture.whenComplete(CompletableFuture.java:2140) ~[?:1.8.0_131]
        at org.apache.pulsar.client.admin.internal.http.AsyncHttpConnector.retryOperation(AsyncHttpConnector.java:250) ~[org.apache.pulsar-pulsar-client-admin-original-2.9.2.jar:2.9.2]
        at org.apache.pulsar.client.admin.internal.http.AsyncHttpConnector.lambda$retryOperation$4(AsyncHttpConnector.java:258) ~[org.apache.pulsar-pulsar-client-admin-original-2.9.2.jar:2.9.2]
        at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) [?:1.8.0_131]
        at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) [?:1.8.0_131]
        at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) [?:1.8.0_131]
        at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1977) [?:1.8.0_131]
        at org.asynchttpclient.netty.NettyResponseFuture.abort(NettyResponseFuture.java:273) [org.asynchttpclient-async-http-client-2.12.1.jar:?]
        at org.asynchttpclient.netty.channel.NettyConnectListener.onFailure(NettyConnectListener.java:181) [org.asynchttpclient-async-http-client-2.12.1.jar:?]
        at org.asynchttpclient.netty.channel.NettyChannelConnector$1.onFailure(NettyChannelConnector.java:108) [org.asynchttpclient-async-http-client-2.12.1.jar:?]
        at org.asynchttpclient.netty.SimpleChannelFutureListener.operationComplete(SimpleChannelFutureListener.java:28) [org.asynchttpclient-async-http-client-2.12.1.jar:?]
        at org.asynchttpclient.netty.SimpleChannelFutureListener.operationComplete(SimpleChannelFutureListener.java:20) [org.asynchttpclient-async-http-client-2.12.1.jar:?]
        at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:578) [io.netty-netty-common-4.1.74.Final.jar:4.1.74.Final]
        at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:552) [io.netty-netty-common-4.1.74.Final.jar:4.1.74.Final]
        at io.netty.util.concurrent.DefaultPromise.access$200(DefaultPromise.java:35) [io.netty-netty-common-4.1.74.Final.jar:4.1.74.Final]
        at io.netty.util.concurrent.DefaultPromise$1.run(DefaultPromise.java:502) [io.netty-netty-common-4.1.74.Final.jar:4.1.74.Final]
        at io.netty.util.concurrent.GlobalEventExecutor$TaskRunner.run(GlobalEventExecutor.java:240) [io.netty-netty-common-4.1.74.Final.jar:4.1.74.Final]
        at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [io.netty-netty-common-4.1.74.Final.jar:4.1.74.Final]
        at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [io.netty-netty-common-4.1.74.Final.jar:4.1.74.Final]
        at java.lang.Thread.run(Thread.java:748) [?:1.8.0_131]
Caused by: java.util.concurrent.CompletionException: org.apache.pulsar.client.admin.internal.http.AsyncHttpConnector$RetryException: Could not complete the operation. Number of retries has been exhausted. Failed reason: Failed to open a socket.
        at java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:292) ~[?:1.8.0_131]
        at java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:308) ~[?:1.8.0_131]
        at java.util.concurrent.CompletableFuture.orApply(CompletableFuture.java:1371) ~[?:1.8.0_131]
        at java.util.concurrent.CompletableFuture$OrApply.tryFire(CompletableFuture.java:1350) ~[?:1.8.0_131]
        ... 45 more
Caused by: org.apache.pulsar.client.admin.internal.http.AsyncHttpConnector$RetryException: Could not complete the operation. Number of retries has been exhausted. Failed reason: Failed to open a socket.
        at org.apache.pulsar.client.admin.internal.http.AsyncHttpConnector.lambda$retryOperation$4(AsyncHttpConnector.java:265) ~[org.apache.pulsar-pulsar-client-admin-original-2.9.2.jar:2.9.2]
        ... 42 more
Caused by: java.net.ConnectException: Failed to open a socket.
        at org.asynchttpclient.netty.channel.NettyConnectListener.onFailure(NettyConnectListener.java:179) ~[org.asynchttpclient-async-http-client-2.12.1.jar:?]
        at org.asynchttpclient.netty.channel.NettyChannelConnector$1.onFailure(NettyChannelConnector.java:108) ~[org.asynchttpclient-async-http-client-2.12.1.jar:?]
        at org.asynchttpclient.netty.SimpleChannelFutureListener.operationComplete(SimpleChannelFutureListener.java:28) ~[org.asynchttpclient-async-http-client-2.12.1.jar:?]
        at org.asynchttpclient.netty.SimpleChannelFutureListener.operationComplete(SimpleChannelFutureListener.java:20) ~[org.asynchttpclient-async-http-client-2.12.1.jar:?]
        at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:578) ~[io.netty-netty-common-4.1.74.Final.jar:4.1.74.Final]
        at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:552) ~[io.netty-netty-common-4.1.74.Final.jar:4.1.74.Final]
        at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:491) ~[io.netty-netty-common-4.1.74.Final.jar:4.1.74.Final]
        at io.netty.util.concurrent.DefaultPromise.addListener(DefaultPromise.java:184) ~[io.netty-netty-common-4.1.74.Final.jar:4.1.74.Final]
        at io.netty.channel.DefaultChannelPromise.addListener(DefaultChannelPromise.java:95) ~[io.netty-netty-transport-4.1.74.Final.jar:4.1.74.Final]
        at io.netty.channel.DefaultChannelPromise.addListener(DefaultChannelPromise.java:30) ~[io.netty-netty-transport-4.1.74.Final.jar:4.1.74.Final]
        at org.asynchttpclient.netty.channel.NettyChannelConnector.connect0(NettyChannelConnector.java:82) ~[org.asynchttpclient-async-http-client-2.12.1.jar:?]
        at org.asynchttpclient.netty.channel.NettyChannelConnector.connect(NettyChannelConnector.java:69) ~[org.asynchttpclient-async-http-client-2.12.1.jar:?]
        at org.asynchttpclient.netty.request.NettyRequestSender$1.lambda$onSuccess$0(NettyRequestSender.java:315) ~[org.asynchttpclient-async-http-client-2.12.1.jar:?]
        at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:578) ~[io.netty-netty-common-4.1.74.Final.jar:4.1.74.Final]
        at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:552) ~[io.netty-netty-common-4.1.74.Final.jar:4.1.74.Final]
        at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:491) ~[io.netty-netty-common-4.1.74.Final.jar:4.1.74.Final]
        at io.netty.util.concurrent.DefaultPromise.addListener(DefaultPromise.java:184) ~[io.netty-netty-common-4.1.74.Final.jar:4.1.74.Final]
        at io.netty.util.concurrent.DefaultPromise.addListener(DefaultPromise.java:35) ~[io.netty-netty-common-4.1.74.Final.jar:4.1.74.Final]
        at org.asynchttpclient.netty.request.NettyRequestSender$1.onSuccess(NettyRequestSender.java:313) ~[org.asynchttpclient-async-http-client-2.12.1.jar:?]
        at org.asynchttpclient.netty.request.NettyRequestSender$1.onSuccess(NettyRequestSender.java:301) ~[org.asynchttpclient-async-http-client-2.12.1.jar:?]
        at org.asynchttpclient.netty.SimpleFutureListener.operationComplete(SimpleFutureListener.java:24) ~[org.asynchttpclient-async-http-client-2.12.1.jar:?]
        at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:578) ~[io.netty-netty-common-4.1.74.Final.jar:4.1.74.Final]
        at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:552) ~[io.netty-netty-common-4.1.74.Final.jar:4.1.74.Final]
        at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:491) ~[io.netty-netty-common-4.1.74.Final.jar:4.1.74.Final]
        at io.netty.util.concurrent.DefaultPromise.addListener(DefaultPromise.java:184) ~[io.netty-netty-common-4.1.74.Final.jar:4.1.74.Final]
        at io.netty.util.concurrent.DefaultPromise.addListener(DefaultPromise.java:35) ~[io.netty-netty-common-4.1.74.Final.jar:4.1.74.Final]
        at org.asynchttpclient.netty.request.NettyRequestSender.sendRequestWithNewChannel(NettyRequestSender.java:301) ~[org.asynchttpclient-async-http-client-2.12.1.jar:?]
        at org.asynchttpclient.netty.request.NettyRequestSender.sendRequestWithCertainForceConnect(NettyRequestSender.java:142) ~[org.asynchttpclient-async-http-client-2.12.1.jar:?]
        at org.asynchttpclient.netty.request.NettyRequestSender.sendRequest(NettyRequestSender.java:113) ~[org.asynchttpclient-async-http-client-2.12.1.jar:?]
        at org.asynchttpclient.DefaultAsyncHttpClient.execute(DefaultAsyncHttpClient.java:259) ~[org.asynchttpclient-async-http-client-2.12.1.jar:?]
        at org.asynchttpclient.DefaultAsyncHttpClient.executeRequest(DefaultAsyncHttpClient.java:228) ~[org.asynchttpclient-async-http-client-2.12.1.jar:?]
        at org.asynchttpclient.BoundRequestBuilder.execute(BoundRequestBuilder.java:39) ~[org.asynchttpclient-async-http-client-2.12.1.jar:?]
        at org.apache.pulsar.client.admin.internal.http.AsyncHttpConnector.oneShot(AsyncHttpConnector.java:316) ~[org.apache.pulsar-pulsar-client-admin-original-2.9.2.jar:2.9.2]
        at org.apache.pulsar.client.admin.internal.http.AsyncHttpConnector.lambda$retryOrTimeOut$2(AsyncHttpConnector.java:236) ~[org.apache.pulsar-pulsar-client-admin-original-2.9.2.jar:2.9.2]
        at org.apache.pulsar.client.admin.internal.http.AsyncHttpConnector.retryOperation(AsyncHttpConnector.java:248) ~[org.apache.pulsar-pulsar-client-admin-original-2.9.2.jar:2.9.2]
        ... 38 more
Caused by: io.netty.channel.ChannelException: Failed to open a socket.
        at io.netty.channel.socket.nio.NioSocketChannel.newSocket(NioSocketChannel.java:71) ~[io.netty-netty-transport-4.1.74.Final.jar:4.1.74.Final]
        at io.netty.channel.socket.nio.NioSocketChannel.<init>(NioSocketChannel.java:88) ~[io.netty-netty-transport-4.1.74.Final.jar:4.1.74.Final]
        at io.netty.channel.socket.nio.NioSocketChannel.<init>(NioSocketChannel.java:81) ~[io.netty-netty-transport-4.1.74.Final.jar:4.1.74.Final]
        at org.asynchttpclient.netty.channel.NioTransportFactory.newChannel(NioTransportFactory.java:27) ~[org.asynchttpclient-async-http-client-2.12.1.jar:?]
        at org.asynchttpclient.netty.channel.NioTransportFactory.newChannel(NioTransportFactory.java:21) ~[org.asynchttpclient-async-http-client-2.12.1.jar:?]
        at io.netty.bootstrap.AbstractBootstrap.initAndRegister(AbstractBootstrap.java:310) ~[io.netty-netty-transport-4.1.74.Final.jar:4.1.74.Final]
        at io.netty.bootstrap.Bootstrap.doResolveAndConnect(Bootstrap.java:155) ~[io.netty-netty-transport-4.1.74.Final.jar:4.1.74.Final]
        at io.netty.bootstrap.Bootstrap.connect(Bootstrap.java:148) ~[io.netty-netty-transport-4.1.74.Final.jar:4.1.74.Final]
        at org.asynchttpclient.netty.channel.NettyChannelConnector.connect0(NettyChannelConnector.java:81) ~[org.asynchttpclient-async-http-client-2.12.1.jar:?]
        at org.asynchttpclient.netty.channel.NettyChannelConnector.connect(NettyChannelConnector.java:69) ~[org.asynchttpclient-async-http-client-2.12.1.jar:?]
        at org.asynchttpclient.netty.request.NettyRequestSender$1.lambda$onSuccess$0(NettyRequestSender.java:315) ~[org.asynchttpclient-async-http-client-2.12.1.jar:?]
        at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:578) ~[io.netty-netty-common-4.1.74.Final.jar:4.1.74.Final]
        at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:552) ~[io.netty-netty-common-4.1.74.Final.jar:4.1.74.Final]
        at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:491) ~[io.netty-netty-common-4.1.74.Final.jar:4.1.74.Final]
        at io.netty.util.concurrent.DefaultPromise.addListener(DefaultPromise.java:184) ~[io.netty-netty-common-4.1.74.Final.jar:4.1.74.Final]
        at io.netty.util.concurrent.DefaultPromise.addListener(DefaultPromise.java:35) ~[io.netty-netty-common-4.1.74.Final.jar:4.1.74.Final]
        at org.asynchttpclient.netty.request.NettyRequestSender$1.onSuccess(NettyRequestSender.java:313) ~[org.asynchttpclient-async-http-client-2.12.1.jar:?]
        at org.asynchttpclient.netty.request.NettyRequestSender$1.onSuccess(NettyRequestSender.java:301) ~[org.asynchttpclient-async-http-client-2.12.1.jar:?]
        at org.asynchttpclient.netty.SimpleFutureListener.operationComplete(SimpleFutureListener.java:24) ~[org.asynchttpclient-async-http-client-2.12.1.jar:?]
        at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:578) ~[io.netty-netty-common-4.1.74.Final.jar:4.1.74.Final]
        at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:552) ~[io.netty-netty-common-4.1.74.Final.jar:4.1.74.Final]
        at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:491) ~[io.netty-netty-common-4.1.74.Final.jar:4.1.74.Final]
        at io.netty.util.concurrent.DefaultPromise.addListener(DefaultPromise.java:184) ~[io.netty-netty-common-4.1.74.Final.jar:4.1.74.Final]
        at io.netty.util.concurrent.DefaultPromise.addListener(DefaultPromise.java:35) ~[io.netty-netty-common-4.1.74.Final.jar:4.1.74.Final]
        at org.asynchttpclient.netty.request.NettyRequestSender.sendRequestWithNewChannel(NettyRequestSender.java:301) ~[org.asynchttpclient-async-http-client-2.12.1.jar:?]
        at org.asynchttpclient.netty.request.NettyRequestSender.sendRequestWithCertainForceConnect(NettyRequestSender.java:142) ~[org.asynchttpclient-async-http-client-2.12.1.jar:?]
        at org.asynchttpclient.netty.request.NettyRequestSender.sendRequest(NettyRequestSender.java:113) ~[org.asynchttpclient-async-http-client-2.12.1.jar:?]
        at org.asynchttpclient.DefaultAsyncHttpClient.ex[root@pulsar-broker02 logs]#

To Reproduce

Expected behavior

Screenshots If applicable, add screenshots to help explain your problem.

Additional context Add any other context about the problem here.

Jun 28 '22 03:06 zbye

Can you use the ulimit -a command to find out the maximum number of files that one of your processes can open? Look like your system has already reached the maximum socket file. You can see the following logs.

2022-06-28T11:14:19,810+0800 [pulsar-acceptor-21-1] WARN  io.netty.channel.DefaultChannelPipeline - An exceptionCaught() event was fired, and it reached at the tail of the pipeline. It usually means the last handler in the pipeline did not handle the exception.
io.netty.channel.unix.Errors$NativeIoException: accept(..) failed: 打开的文件过多

Jun 28 '22 03:06 Demogorgon314

@Demogorgon314 the maximum number is 655535, the actual number of all socket files opened is around 4600. [root@pulsar-broker02 broker]# ulimit -a core file size (blocks, -c) 0 data seg size (kbytes, -d) unlimited scheduling priority (-e) 0 file size (blocks, -f) unlimited pending signals (-i) 127923 max locked memory (kbytes, -l) 64 max memory size (kbytes, -m) unlimited open files (-n) 655535 pipe size (512 bytes, -p) 8 POSIX message queues (bytes, -q) 819200 real-time priority (-r) 0 stack size (kbytes, -s) 8192 cpu time (seconds, -t) unlimited max user processes (-u) 655535 virtual memory (kbytes, -v) unlimited file locks (-x) unlimited

Jun 28 '22 06:06 zbye

The broker has works well after restarting the process. thanks.

Jun 28 '22 06:06 zbye

kop kop copied to clipboard

[BUG] lookup topic failed (Failed to open a socket...)

kop
kop copied to clipboard