dubbo Failed to load service metadata, meta type is local

[x] I have searched the issues of this repository and believe that this is not a duplicate.

Environment

Dubbo version: 3.0.3
Operating System version: debian
Java version: 1.8

Steps to reproduce this issue

提供者重新部署时有一定概率出现，会导致所有依赖该提供者的消费者都出现这个异常，调用不了rpc，每次出现了都要把所有的消费者重启才能解决，希望能帮忙排查下，谢谢！

content: 
22-07-15 20:13:46.360 [ERROR] [Dubbo-metadata-retry-thread-1] [] [o.a.d.r.c.e.l.ServiceInstancesChangedListener]  [DUBBO] Failed to load service metadata, meta type is local, dubbo version: 3.0.3, current host: 172.29.75.179
+2
[2022-07-15 20:13:46]
content: 
java.lang.NullPointerException: Cannot invoke "org.apache.dubbo.metadata.MetadataInfo$ServiceInfo.getNumbers()" because the return value of "org.apache.dubbo.registry.client.InstanceAddressURL.getServiceInfo(String)" is null
+3
[2022-07-15 20:13:46]
content: 
	at org.apache.dubbo.registry.client.InstanceAddressURL.getServiceNumbers(InstanceAddressURL.java:448) ~[dubbo-3.0.3.jar!/:3.0.3]
+4
[2022-07-15 20:13:46]
content: 
	at org.apache.dubbo.registry.client.InstanceAddressURL.getNumbers(InstanceAddressURL.java:460) ~[dubbo-3.0.3.jar!/:3.0.3]
+5
[2022-07-15 20:13:46]
content: 
	at org.apache.dubbo.common.URL.getParameter(URL.java:663) ~[dubbo-3.0.3.jar!/:3.0.3]
+6
[2022-07-15 20:13:46]
content: 
	at org.apache.dubbo.common.URL.getPositiveParameter(URL.java:732) ~[dubbo-3.0.3.jar!/:3.0.3]
+7
[2022-07-15 20:13:46]
content: 
	at org.apache.dubbo.remoting.transport.AbstractEndpoint.<init>(AbstractEndpoint.java:46) ~[dubbo-3.0.3.jar!/:3.0.3]
+8
[2022-07-15 20:13:46]
content: 
	at org.apache.dubbo.remoting.transport.AbstractClient.<init>(AbstractClient.java:54) ~[dubbo-3.0.3.jar!/:3.0.3]
+9
[2022-07-15 20:13:46]
content: 
	at org.apache.dubbo.remoting.transport.netty4.NettyClient.<init>(NettyClient.java:83) ~[dubbo-3.0.3.jar!/:3.0.3]
+10
[2022-07-15 20:13:46]
content: 
	at org.apache.dubbo.remoting.transport.netty4.NettyTransporter.connect(NettyTransporter.java:40) ~[dubbo-3.0.3.jar!/:3.0.3]
+11
[2022-07-15 20:13:46]
content: 
	at org.apache.dubbo.remoting.Transporter$Adaptive.connect(Transporter$Adaptive.java) ~[dubbo-3.0.3.jar!/:3.0.3]
+12
[2022-07-15 20:13:46]
content: 
	at org.apache.dubbo.remoting.Transporters.connect(Transporters.java:74) ~[dubbo-3.0.3.jar!/:3.0.3]
+13
[2022-07-15 20:13:46]
content: 
	at org.apache.dubbo.remoting.exchange.support.header.HeaderExchanger.connect(HeaderExchanger.java:39) ~[dubbo-3.0.3.jar!/:3.0.3]
+14
[2022-07-15 20:13:46]
content: 
	at org.apache.dubbo.remoting.exchange.Exchangers.connect(Exchangers.java:108) ~[dubbo-3.0.3.jar!/:3.0.3]
+15
[2022-07-15 20:13:46]
content: 
	at org.apache.dubbo.rpc.protocol.dubbo.LazyConnectExchangeClient.initClient(LazyConnectExchangeClient.java:84) ~[dubbo-3.0.3.jar!/:3.0.3]……
+17
[2022-07-15 20:13:46]
content: 
	at org.apache.dubbo.rpc.protocol.dubbo.ReferenceCountExchangeClient.request(ReferenceCountExchangeClient.java:93) ~[dubbo-3.0.3.jar!/:3.0.3]
+18
[2022-07-15 20:13:46]
content: 
	at org.apache.dubbo.rpc.protocol.dubbo.DubboInvoker.doInvoke(DubboInvoker.java:111) ~[dubbo-3.0.3.jar!/:3.0.3]
+19
[2022-07-15 20:13:46]
content: 
	at org.apache.dubbo.rpc.protocol.AbstractInvoker.doInvokeAndReturn(AbstractInvoker.java:234) ~[dubbo-3.0.3.jar!/:3.0.3]
+20
[2022-07-15 20:13:46]
content: 
	at org.apache.dubbo.rpc.protocol.AbstractInvoker.invoke(AbstractInvoker.java:179) ~[dubbo-3.0.3.jar!/:3.0.3]
+21
[2022-07-15 20:13:46]
content: 
	at org.apache.dubbo.rpc.listener.ListenerInvokerWrapper.invoke(ListenerInvokerWrapper.java:78) ~[dubbo-3.0.3.jar!/:3.0.3]
+22
[2022-07-15 20:13:46]
content: 
	at com.ringle.framework.dubbo.consumer.logging.DubboLogConsumerFilter.invoke(DubboLogConsumerFilter.java:61) ~[ringle-framework-dubbo-0.0.1-SNAPSHOT.jar!/:?]
+23
[2022-07-15 20:13:46]
content: 
	at org.apache.dubbo.rpc.cluster.filter.FilterChainBuilder$FilterChainNode.invoke(FilterChainBuilder.java:84) ~[dubbo-3.0.3.jar!/:3.0.3]
+24
[2022-07-15 20:13:46]
content: 
	at com.ringle.framework.dubbo.consumer.error.DubboExceptionConsumerFilter.invoke(DubboExceptionConsumerFilter.java:35) ~[ringle-framework-dubbo-0.0.1-SNAPSHOT.jar!/:?]
+25
[2022-07-15 20:13:46]
content: 
	at org.apache.dubbo.rpc.cluster.filter.FilterChainBuilder$FilterChainNode.invoke(FilterChainBuilder.java:84) ~[dubbo-3.0.3.jar!/:3.0.3]
+26
[2022-07-15 20:13:46]
content: 
	at com.ringle.framework.dubbo.consumer.context.DubboContextConsumerFilter.invoke(DubboContextConsumerFilter.java:30) ~[ringle-framework-dubbo-0.0.1-SNAPSHOT.jar!/:?]
+27
[2022-07-15 20:13:46]
content: 
	at org.apache.dubbo.rpc.cluster.filter.FilterChainBuilder$FilterChainNode.invoke(FilterChainBuilder.java:84) ~[dubbo-3.0.3.jar!/:3.0.3]
+28
[2022-07-15 20:13:46]
content: 
	at org.apache.dubbo.rpc.proxy.InvokerInvocationHandler.invoke(InvokerInvocationHandler.java:92) ~[dubbo-3.0.3.jar!/:3.0.3]
+29
[2022-07-15 20:13:46]
content: 
	at org.apache.dubbo.common.bytecode.proxy0.getMetadataInfo(proxy0.java) ~[dubbo-3.0.3.jar!/:3.0.3]
+30
[2022-07-15 20:13:46]
content: 
	at org.apache.dubbo.registry.client.event.listener.ServiceInstancesChangedListener.doGetMetadataInfo(ServiceInstancesChangedListener.java:345) ~[dubbo-3.0.3.jar!/:3.0.3]
+30
[2022-07-15 20:13:46]
content: 
	at org.apache.dubbo.registry.client.event.listener.ServiceInstancesChangedListener.doGetMetadataInfo(ServiceInstancesChangedListener.java:345) ~[dubbo-3.0.3.jar!/:3.0.3]
+31
[2022-07-15 20:13:46]
content: 
	at org.apache.dubbo.registry.client.event.listener.ServiceInstancesChangedListener.getRemoteMetadata(ServiceInstancesChangedListener.java:295) ~[dubbo-3.0.3.jar!/:3.0.3]
+32
[2022-07-15 20:13:46]
content: 
	at org.apache.dubbo.registry.client.event.listener.ServiceInstancesChangedListener.onEvent(ServiceInstancesChangedListener.java:138) ~[dubbo-3.0.3.jar!/:3.0.3]
+33
[2022-07-15 20:13:46]
content: 
	at org.apache.dubbo.registry.client.event.listener.ServiceInstancesChangedListener$AddressRefreshRetryTask.run(ServiceInstancesChangedListener.java:456) ~[dubbo-3.0.3.jar!/:3.0.3]
+34
[2022-07-15 20:13:46]
content: 
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) ~[?:?]
+35
[2022-07-15 20:13:46]
content: 
	at java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[?:?]
+36
[2022-07-15 20:13:46]
content: 
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) ~[?:?]
+37
[2022-07-15 20:13:46]
content: 
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) ~[?:?]

Jul 15 '22 12:07 monkeyWie

Related with #10303

Jul 15 '22 13:07 AlbumenJ

可以在报错的时候打下断点，看下 at org.apache.dubbo.registry.client.InstanceAddressURL.getServiceNumbers(InstanceAddressURL.java:448) ~[dubbo-3.0.3.jar!/:3.0.3] 这里的 protocolServiceKey 是什么吗（是新发布的服务吗，或者是服务端还没发布的服务）

Jul 15 '22 14:07 AlbumenJ

另外可以试下 3.0.9 版本是不是也有问题

Jul 15 '22 14:07 AlbumenJ

NPE 的在 3.0.10（下周正式发布）进行容错

Jul 15 '22 14:07 AlbumenJ

@AlbumenJ 现在还没找到稳定重现的步骤，所以也不好进行问题分析，我现在本地服务一直开着看看能不能复现下

Jul 18 '22 02:07 monkeyWie

升级到 3.0.10 发现一个大问题，已经提了个PR修复。

Jul 25 '22 09:07 monkeyWie

[2022-07-15 20:13:46] content: java.lang.NullPointerException: Cannot invoke "org.apache.dubbo.metadata.MetadataInfo$ServiceInfo.getNumbers()" because the return value of "org.apache.dubbo.registry.client.InstanceAddressURL.getServiceInfo(String)" is null +3 [2022-07-15 20:13:46] content: at org.apache.dubbo.registry.client.InstanceAddressURL.getServiceNumbers(InstanceAddressURL.java:448) ~[dubbo-3.0.3.jar!/:3.0.3] +4 [2022-07-15 20:13:46] content: at org.apache.dubbo.registry.client.InstanceAddressURL.getNumbers(InstanceAddressURL.java:460) ~[dubbo-3.0.3.jar!/:3.0.3] +5 [2022-07-15 20:13:46] content: at org.apache.dubbo.common.URL.getParameter(URL.java:663) ~[dubbo-3.0.3.jar!/:3.0.3] +6 [2022-07-15 20:13:46]

这个应该是一个 MetaInfo 初始化的并发问题，在新版本中已经修复。

升级到 3.0.10 发现一个大问题，已经提了个https://github.com/apache/dubbo/pull/10364修复。

DUBBO_PORT_TO_REGISTRY 在应用级服务发现场景下未能进行测试覆盖，结合这个问题，建议升级到最新的 3.0.11 bugfix 版本

Sep 02 '22 02:09 chickenlj