Dubbo-go Observability to follow the Dubbo standards.

[ev1lQuark]

• Improve dubbo-go's metrics instrumentation and strictly follow the naming convention of metrics. （完善 dubbo-go 的指标埋点并严格遵循指标的命名规范）
• Realize the metrics reporting strategy of active push and passive pull. （实现主动推送和被动拉取两种方式的指标上报策略）

[chickenlj]

Here's the Metrics specification defined in Dubbo for every specific language implementation to follow.

https://cn.dubbo.apache.org/zh-cn/overview/core-features/observability/

[ev1lQuark]

dubbo-go metrics Existing code combing

/metrics encapsulates basic indicator type definitions and common methods

• Count
  • IncCount
  • IncCountWithLabel
• Gauge /ɡeɪdʒ/
  • SetGauge
  • SetGaugeWithLabel
• Summary
  • IncSummary
  • IncSummaryWithLabel
• Histogram

/filter/metrics The provider’s rt metrics are exported, but there are problems with the indicator name and label format

The implementation idea is to inject the report method of metrics into the filter chain of dubbo-go through the filter method, and realize the buried point of metrics in the dubbo-go RPC call process

• community standard format

dubbo_{type}_{action}_{unit}_{aggregate}

• dubbo-go metrics format

# HELP dubbo_provider_service_rt
# TYPE dubbo_provider_service_rt summary
dubbo_provider_service_rt{group="dubbo-go",method="SayHello",service="org.apache.dubbogo.samples.api.Greeter",timeout="",version="3.0.1",quantile="0.5"} 2.008e+06
dubbo_provider_service_rt{group="dubbo-go",method="SayHello",service="org.apache.dubbogo.samples.api.Greeter",timeout="",version="3.0.1",quantile="0.75"} 3.7044e+06
dubbo_provider_service_rt{group="dubbo-go",method="SayHello",service="org.apache.dubbogo.samples.api.Greeter",timeout="",version="3.0.1",quantile="0.9"} 6.2967e+06
dubbo_provider_service_rt{group="dubbo-go",method="SayHello",service="org.apache.dubbogo.samples.api.Greeter",timeout="",version="3.0.1",quantile="0.98"} 3.71908e+07
dubbo_provider_service_rt{group="dubbo-go",method="SayHello",service="org.apache.dubbogo.samples.api.Greeter",timeout="",version="3.0.1",quantile="0.99"} 3.71908e+07
dubbo_provider_service_rt{group="dubbo-go",method="SayHello",service="org.apache.dubbogo.samples.api.Greeter",timeout="",version="3.0.1",quantile="0.999"} 3.71908e+07
dubbo_provider_service_rt_sum{group="dubbo-go",method="SayHello",service="org.apache.dubbogo.samples.api.Greeter",timeout="",version="3.0.1"} 1.175357e+08
dubbo_provider_service_rt_count{group="dubbo-go",method="SayHello",service="org.apache.dubbogo.samples.api.Greeter",timeout="",version="3.0.1"} 35

• dubbo-java metrics format

# HELP dubbo_provider_rt_milliseconds_sum Sum Response Time
# TYPE dubbo_provider_rt_milliseconds_sum gauge
dubbo_provider_rt_milliseconds_sum{application_name="metrics-prometheus-provider",group="",hostname="HUAWEI-MATEBOOK14",interface="org.apache.dubbo.samples.metrics.prometheus.api.DemoService",ip="192.168.0.108",method="sayHello",version="0.0.0",} 528.0

# HELP dubbo_provider_rt_milliseconds_p99 Response Time P99
# TYPE dubbo_provider_rt_milliseconds_p99 gauge
dubbo_provider_rt_milliseconds_p99{application_name="metrics-prometheus-provider",group="",hostname="HUAWEI-MATEBOOK14",interface="org.apache.dubbo.samples.metrics.prometheus.api.DemoService",ip="192.168.0.108",method="sayHello",version="0.0.0",} 1.0

# HELP dubbo_provider_rt_milliseconds_p95 Response Time P99
# TYPE dubbo_provider_rt_milliseconds_p95 gauge
dubbo_provider_rt_milliseconds_p95{application_name="metrics-prometheus-provider",group="",hostname="HUAWEI-MATEBOOK14",interface="org.apache.dubbo.samples.metrics.prometheus.api.DemoService",ip="192.168.0.108",method="sayHello",version="0.0.0",} 1.0

Preparation

• Build a Prometheus + Grafana test environment locally for subsequent development and debugging
• Based on the metrics-sample in dubbo-go-samples, it is rewritten as a consumer polling request provider demo, simulating the basic dubbo-go observation metrics scene, and accessing the local test environment

Todo List

• According to the indicators sorted out by dubbo-metrics documents, complete the supplementary work of indicator embedding, which can be mainly divided into the following parts:
  • provider (partially implemented)
  • consumer
  • config center
  • registry center
  • metadata center
  • RpcException
• For the RPC call between Provider and Consumer, the existing logic is all written in the report method of the Report interface called by the filter, which may need to be refactored into multiple files/packages later
• In terms of technology selection, the existing design of dubbo-go metrics is to use prometheus client-go. Due to the interface-oriented programming, the coupling degree is very low, and the support for other frameworks such as Otel can be easily added in the future (optional)
• In terms of specific implementation, the design of the dubbo-java metrics module is relatively complicated, so the design goal of dubbo-go metrics is to ensure that the indicators are aligned. As for the specific implementation, it follows the code style of the go language

[ev1lQuark]

Dubbo-go Tracing Todo List

• In the Dubbo-go kernel, the configuration loading and component initialization of tracing should be supplemented and improved, and the application can realize the comprehensive control of Dubbo-go tracing ability by simply modifying the configuration file.
• For context propagation, currently Dubbo-go only introduces the propagator that supports W3C standards, and additional support for the B3 standard is required. The implementation of the B3 protocol can use the implementation in opentelemetry-go-contrib
• The configuration of Tracing needs to be further aligned with Dubbo-java, such as adding support for configuration items such as sampling probability to achieve a more refined sampling strategy.
• Tracing's observability backend supports multiple types, such as OTel, Jaeger, Zipkin, etc. Currently, only Jaeger is supported.