[Bug] The registry uses etcd, but javaagent skywalking fails to start

[glynnwei]

Pre-check

• [X] I am sure that all the content I provide is in English.

Search before asking

• [X] I had searched in the issues and found no similar issues.

Apache Dubbo Component

Java SPI Extensions (apache/dubbo-spi-extensions)

Dubbo Version

dubbo: 3.2.11 jdk: 1.8 skywalking agent: 9.2.0

Steps to reproduce this issue

1、maven import

...
<maven.compiler.source>1.8</maven.compiler.source>
<maven.compiler.target>1.8</maven.compiler.target>
<dubbo.version>3.2.11</dubbo.version>
<spring-boot.version>2.6.4</spring-boot.version>
...
<dependency>
    <groupId>org.apache.dubbo.extensions</groupId>
    <artifactId>dubbo-registry-etcd3</artifactId>
    <version>3.2.0</version>
</dependency>
<dependency>
    <groupId>org.apache.dubbo.extensions</groupId>
    <artifactId>dubbo-metadata-report-etcd</artifactId>
    <version>3.2.0</version>
</dependency>
<dependency>
    <groupId>org.apache.dubbo</groupId>
    <artifactId>dubbo-spring-boot-observability-starter</artifactId>
    <version>3.2.11</version>
</dependency>
<dependency>
    <groupId>org.apache.dubbo</groupId>
    <artifactId>dubbo-metrics-api</artifactId>
    <version>3.2.11</version>
</dependency>
...

2、start javaagent

-javaagent:/skywalking-agent-9.2.0/skywalking-agent.jar
-Dskywalking.agent.service_name=test-project
-Dskywalking.collector.backend_service=127.0.0.1:11800

3、error message

java.lang.IllegalStateException: Failed to unregister grpc://10.63.9.0:8082/proto.helloworld.Greeter?application=test-project&deprecated=false&dubbo=2.0.2&prefer.serialization=fastjson2,hessian2&release=3.2.11&side=provider to registry 127.0.0.1:2379, cause: Failed to unregister grpc://10.63.9.0:8082/proto.helloworld.Greeter?application=test-project&deprecated=false&dubbo=2.0.2&prefer.serialization=fastjson2,hessian2&release=3.2.11&side=provider to etcd etcd3://127.0.0.1:2379/org.apache.dubbo.registry.RegistryService?application=test-project&dubbo=2.0.2&executor-management-mode=isolation&file-cache=true&interface=org.apache.dubbo.registry.RegistryService&logger=slf4j&pid=90951&proxy=nativestub&register=false&release=3.2.11&simplified=true, cause: java.lang.IllegalStateException: Exit span doesn't include meaningful peer information.
	at org.apache.dubbo.registry.support.FailbackRegistry.unregister(FailbackRegistry.java:299)
	at org.apache.dubbo.registry.support.AbstractRegistry.destroy(AbstractRegistry.java:626)
	at org.apache.dubbo.registry.support.FailbackRegistry.destroy(FailbackRegistry.java:473)
	at org.apache.dubbo.registry.etcd.EtcdRegistry.destroy(EtcdRegistry.java:275)
	at org.apache.dubbo.registry.support.RegistryManager.destroyAll(RegistryManager.java:107)
	at org.apache.dubbo.config.deploy.DefaultApplicationDeployer.destroyRegistries(DefaultApplicationDeployer.java:1372)
	at org.apache.dubbo.config.deploy.DefaultApplicationDeployer.postDestroy(DefaultApplicationDeployer.java:1106)
	at org.apache.dubbo.rpc.model.ApplicationModel.onDestroy(ApplicationModel.java:172)
	at org.apache.dubbo.rpc.model.ScopeModel.destroy(ScopeModel.java:122)
	at org.apache.dubbo.rpc.model.ApplicationModel.tryDestroy(ApplicationModel.java:276)
	at org.apache.dubbo.rpc.model.ModuleModel.onDestroy(ModuleModel.java:133)
	at org.apache.dubbo.rpc.model.ScopeModel.destroy(ScopeModel.java:122)
	at org.apache.dubbo.config.spring.context.DubboDeployApplicationListener.onContextClosedEvent(DubboDeployApplicationListener.java:189)
	at org.apache.dubbo.config.spring.context.DubboDeployApplicationListener.onApplicationEvent(DubboDeployApplicationListener.java:145)
	at org.apache.dubbo.config.spring.context.DubboDeployApplicationListener.onApplicationEvent(DubboDeployApplicationListener.java:52)
	at org.springframework.context.event.SimpleApplicationEventMulticaster.doInvokeListener(SimpleApplicationEventMulticaster.java:176)
	at org.springframework.context.event.SimpleApplicationEventMulticaster.invokeListener(SimpleApplicationEventMulticaster.java:169)
	at org.springframework.context.event.SimpleApplicationEventMulticaster.multicastEvent(SimpleApplicationEventMulticaster.java:143)
	at org.springframework.context.support.AbstractApplicationContext.publishEvent(AbstractApplicationContext.java:421)
	at org.springframework.context.support.AbstractApplicationContext.publishEvent(AbstractApplicationContext.java:378)
	at org.springframework.context.support.AbstractApplicationContext.doClose(AbstractApplicationContext.java:1058)
	at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.doClose(ServletWebServerApplicationContext.java:172)
	at org.springframework.context.support.AbstractApplicationContext.close(AbstractApplicationContext.java:1021)
	at org.springframework.boot.SpringApplication.handleRunFailure(SpringApplication.java:796)
	at org.springframework.boot.SpringApplication.run(SpringApplication.java:313)
	at org.springframework.boot.SpringApplication.run(SpringApplication.java:1312)
	at org.springframework.boot.SpringApplication.run(SpringApplication.java:1301)
	at com.test.server.TestApplication.main(TestApplication.java:25)
Caused by: org.apache.dubbo.rpc.RpcException: Failed to unregister grpc://10.63.9.0:8082/proto.helloworld.Greeter?application=test-project&deprecated=false&dubbo=2.0.2&prefer.serialization=fastjson2,hessian2&release=3.2.11&side=provider to etcd etcd3://127.0.0.1:2379/org.apache.dubbo.registry.RegistryService?application=test-project&dubbo=2.0.2&executor-management-mode=isolation&file-cache=true&interface=org.apache.dubbo.registry.RegistryService&logger=slf4j&pid=90951&proxy=nativestub&register=false&release=3.2.11&simplified=true, cause: java.lang.IllegalStateException: Exit span doesn't include meaningful peer information.
	at org.apache.dubbo.registry.etcd.EtcdRegistry.doUnregister(EtcdRegistry.java:130)
	at org.apache.dubbo.registry.support.FailbackRegistry.unregister(FailbackRegistry.java:284)
	... 27 common frames omitted
Caused by: java.lang.IllegalStateException: java.lang.IllegalStateException: Exit span doesn't include meaningful peer information.
	at org.apache.dubbo.remoting.etcd.jetcd.JEtcdClientWrapper.delete(JEtcdClientWrapper.java:481)
	at org.apache.dubbo.remoting.etcd.jetcd.JEtcdClient.delete(JEtcdClient.java:167)
	at org.apache.dubbo.registry.etcd.EtcdRegistry.doUnregister(EtcdRegistry.java:128)
	... 28 common frames omitted
Caused by: java.util.concurrent.ExecutionException: java.lang.IllegalStateException: Exit span doesn't include meaningful peer information.
	at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357)
	at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1928)
	at org.apache.dubbo.remoting.etcd.jetcd.JEtcdClientWrapper.lambda$delete$15(JEtcdClientWrapper.java:476)
	at org.apache.dubbo.remoting.etcd.jetcd.RetryLoops.invokeWithRetry(RetryLoops.java:57)
	at org.apache.dubbo.remoting.etcd.jetcd.JEtcdClientWrapper.delete(JEtcdClientWrapper.java:471)
	... 30 common frames omitted
Caused by: java.lang.IllegalStateException: Exit span doesn't include meaningful peer information.
	at org.apache.skywalking.apm.agent.core.context.TracingContext.inject(TracingContext.java:169)
	at org.apache.skywalking.apm.agent.core.context.TracingContext.inject(TracingContext.java:149)
	at org.apache.skywalking.apm.agent.core.context.ContextManager.inject(ContextManager.java:145)
	at org.apache.skywalking.apm.plugin.grpc.v1.client.TracingClientCall.start(TracingClientCall.java:84)
	at io.grpc.stub.ClientCalls.startCall(ClientCalls.java:341)
	at io.grpc.stub.ClientCalls.asyncUnaryRequestCall(ClientCalls.java:315)
	at io.grpc.stub.ClientCalls.$sw$original$futureUnaryCall$q891cd3(ClientCalls.java:227)
	at io.grpc.stub.ClientCalls.$sw$original$futureUnaryCall$q891cd3$accessor$$sw$i3l8ir2(ClientCalls.java)
	at io.grpc.stub.ClientCalls$$sw$auxiliary$286ppc0.call(Unknown Source)
	at org.apache.skywalking.apm.agent.core.plugin.interceptor.enhance.StaticMethodsInter.intercept(StaticMethodsInter.java:83)
	at io.grpc.stub.ClientCalls.futureUnaryCall(ClientCalls.java)
	at io.etcd.jetcd.api.KVGrpc$KVFutureStub.deleteRange(KVGrpc.java:520)
	at io.etcd.jetcd.KVImpl.lambda$delete$6(KVImpl.java:138)
	at io.etcd.jetcd.ClientConnectionManager.lambda$execute$4(ClientConnectionManager.java:349)
	at net.jodah.failsafe.Functions$1.get(Functions.java:134)
	at net.jodah.failsafe.Functions$1.get(Functions.java:129)
	at net.jodah.failsafe.Functions$2.get(Functions.java:226)
	at net.jodah.failsafe.Functions$2.get(Functions.java:216)
	at net.jodah.failsafe.internal.util.DelegatingScheduler.lambda$schedule$0(DelegatingScheduler.java:141)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:750)

What you expected to happen

Other registries (nacos, zookeeper) can be started successfully, and skywalking data can be collected

Anything else

The /skywalk-agent-9.2.0/plugins directory is loaded properly

Are you willing to submit a pull request to fix on your own?

• [X] Yes I am willing to submit a pull request on my own!

Code of Conduct

• [X] I agree to follow this project's Code of Conduct

[chickenlj]

... <maven.compiler.source>1.8</maven.compiler.source> <maven.compiler.target>1.8</maven.compiler.target> <dubbo.version>3.2.11</dubbo.version> <spring-boot.version>2.6.4</spring-boot.version> ... <groupId>org.apache.dubbo.extensions</groupId> <artifactId>dubbo-registry-etcd3</artifactId> 3.2.0 <groupId>org.apache.dubbo.extensions</groupId> <artifactId>dubbo-metadata-report-etcd</artifactId> 3.2.0 <groupId>org.apache.dubbo</groupId> <artifactId>dubbo-spring-boot-observability-starter</artifactId> 3.2.11 <groupId>org.apache.dubbo</groupId> <artifactId>dubbo-metrics-api</artifactId> 3.2.11 ...

Please try to add dubbo-spring-boot-starter dependency and necessary registry starter dependency. dubbo-metrics-api is not necessary once dubbo-spring-boot-starter is added.

[chickenlj]

Why the url in registry is grpc://10.63.9.0:8082/proto.helloworld.Greeter? Do you set dubbo.protocol.name=tri?

[glynnwei]

... <maven.compiler.source>1.8</maven.compiler.source> <maven.compiler.target>1.8</maven.compiler.target> <dubbo.version>3.2.11</dubbo.version> < spring-boot.version>2.6.4</spring-boot.version> ...

org.apache.dubbo.extensions dubbo-registry-etcd3 3.2.0

org.apache.dubbo.extensions dubbo-metadata-report-etcd 3.2 .0

org.apache.dubbo dubbo-spring-boot-observability-starter 3.2.11

org.apache.dubbo dubbo-metrics-api 3.2.11

...

请尝试添加dubbo-spring-boot-starter依赖项和必要的注册表启动器依赖项。一旦添加dubbo-metrics-api就不需要了。dubbo-spring-boot-starter

<dependency>
            <groupId>org.apache.dubbo</groupId>
            <artifactId>dubbo-spring-boot-starter</artifactId>
            <version>3.2.11</version>
        </dependency>
        <dependency>
            <groupId>org.apache.dubbo</groupId>
            <artifactId>dubbo-spring-boot-observability-starter</artifactId>
            <version>3.2.11</version>
        </dependency>
        <dependency>
            <groupId>org.apache.dubbo.extensions</groupId>
            <artifactId>dubbo-registry-etcd3</artifactId>
            <version>3.2.0</version>
        </dependency>
        <dependency>
            <groupId>org.apache.dubbo.extensions</groupId>
            <artifactId>dubbo-metadata-report-etcd</artifactId>
            <version>3.2.0</version>
        </dependency>

Thanks. It's still the same problem after modification

[glynnwei]

Why the url in registry is grpc://10.63.9.0:8082/proto.helloworld.Greeter? Do you set dubbo.protocol.name=tri? application.yml

  protocol:
    name: grpc
    port: 8082
  registry:
    address: etcd3://127.0.0.1:2379

[chickenlj]

More likely to be an issue of skywalking integrating with etcd.

[glynnwei]

I solved the problem by removing conflicting plugins:

skywalking-agent/plugins/apm-grpc-1.x-plugin-x.x.x.jar