gRPC message exceeds maximum size

[wangwei-ying]

Describe the bug

Caused by: io.grpc.StatusRuntimeException: RESOURCE_EXHAUSTED: gRPC message exceeds maximum size 10485760: 24329998
	at io.grpc.Status.asRuntimeException(Status.java:533)
	at io.grpc.stub.ClientCalls$UnaryStreamToFuture.onClose(ClientCalls.java:490)
	at io.grpc.PartialForwardingClientCallListener.onClose(PartialForwardingClientCallListener.java:39)
	at io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:23)
	at io.grpc.ForwardingClientCallListener$SimpleForwardingClientCallListener.onClose(ForwardingClientCallListener.java:40)
	at io.grpc.internal.CensusStatsModule$StatsClientInterceptor$1$1.onClose(CensusStatsModule.java:700)
	at io.grpc.PartialForwardingClientCallListener.onClose(PartialForwardingClientCallListener.java:39)
	at io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:23)
	at io.grpc.ForwardingClientCallListener$SimpleForwardingClientCallListener.onClose(ForwardingClientCallListener.java:40)
	at io.grpc.internal.CensusTracingModule$TracingClientInterceptor$1$1.onClose(CensusTracingModule.java:399)
	at io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:510)
	at io.grpc.internal.ClientCallImpl.access$300(ClientCallImpl.java:66)
	at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.close(ClientCallImpl.java:630)
	at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.access$700(ClientCallImpl.java:518)
	at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInternal(ClientCallImpl.java:692)
	at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:681)
	at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
	at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
	... 3 common frames omitted
2021-12-15 17:05:46,692 INFO [DISTRO-INIT] load snapshot Nacos:Naming:v2:ClientData from 10.10.220.46:8848

2021-12-15 17:05:47,217 ERROR [DISTRO-INIT] load snapshot Nacos:Naming:v2:ClientData from 10.10.220.46:8848 failed.

com.alibaba.nacos.core.distributed.distro.exception.DistroException: [DISTRO-EXCEPTION][DISTRO-FAILED] Get distro snapshot failed! 
	at com.alibaba.nacos.naming.consistency.ephemeral.distro.v2.DistroClientTransportAgent.getDatumSnapshot(DistroClientTransportAgent.java:186)
	at com.alibaba.nacos.core.distributed.distro.task.load.DistroLoadDataTask.loadAllDataSnapshotFromRemote(DistroLoadDataTask.java:103)
	at com.alibaba.nacos.core.distributed.distro.task.load.DistroLoadDataTask.load(DistroLoadDataTask.java:87)
	at com.alibaba.nacos.core.distributed.distro.task.load.DistroLoadDataTask.run(DistroLoadDataTask.java:63)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
Caused by: com.alibaba.nacos.api.exception.NacosException: java.util.concurrent.ExecutionException: io.grpc.StatusRuntimeException: RESOURCE_EXHAUSTED: gRPC message exceeds maximum size 10485760: 18983286
	at com.alibaba.nacos.common.remote.client.grpc.GrpcConnection.request(GrpcConnection.java:78)
	at com.alibaba.nacos.common.remote.client.RpcClient.request(RpcClient.java:657)
	at com.alibaba.nacos.core.cluster.remote.ClusterRpcClientProxy.sendRequest(ClusterRpcClientProxy.java:175)
	at com.alibaba.nacos.core.cluster.remote.ClusterRpcClientProxy.sendRequest(ClusterRpcClientProxy.java:161)
	at com.alibaba.nacos.naming.consistency.ephemeral.distro.v2.DistroClientTransportAgent.getDatumSnapshot(DistroClientTransportAgent.java:177)
	... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: io.grpc.StatusRuntimeException: RESOURCE_EXHAUSTED: gRPC message exceeds maximum size 10485760: 18983286
	at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:566)
	at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:445)
	at com.alibaba.nacos.common.remote.client.grpc.GrpcConnection.request(GrpcConnection.java:76)
	... 14 common frames omitted

A clear and concise description of what the bug is.

Expected behavior A clear and concise description of what you expected to happen.

Acutally behavior A clear and concise description of what you actually to happen.

How to Reproduce Steps to reproduce the behavior:

1. Go to '...'
2. Click on '....'
3. Scroll down to '....'
4. See error

Desktop (please complete the following information):

• Version nacos-server 2.03

Additional context Add any other context about the problem here.

[onewe]

How to reproduce it?

[wangwei-ying]

10485760 看了源码可以用过这个参数设置， 重现方式未知，可能是service太多了快照文件太大 nacos.remote.client.grpc.maxinbound.message.size

[whl12345]

gRPC message exceeds maximum size这个报错在高负载下手动宕机一台server，然后重启会出现

11节点2.x server+2.x client 8k服务数，8w注册实例数，宕机1台nacos server后,再重启会出现这个错误

[KomachiSion]

-Dnacos.remote.client.grpc.maxinbound.message.size= -Dnacos.remote.server.grpc.maxinbound.message.size=

[xy37]

nacos是否可以支持自动将快照文件分批传输呢？而不是通过修改-Dnacos.remote.client.grpc.maxinbound.message.size= -Dnacos.remote.server.grpc.maxinbound.message.size=。在扩容场景，用户无法预料自己的快照数据有多大