linkerd2 icon indicating copy to clipboard operation
linkerd2 copied to clipboard

The counter metrics about the number of sent/received events in streams

Open loyd opened this issue 1 year ago • 2 comments

What problem are you trying to solve?

Hi, I'm choosing between linkerd (first of all because the proxy is written in Rust, which is lightweight on my tests) and envoy-based solutions for a cluster of GRPC services. My primary requirement is excellent telemetry.

We heavily rely on streaming between all our services. However, linkerd doesn't provide any event counter for long-lived streams:

route_request_total{direction="outbound",dst="lve-server.default.svc.cluster.local:4242",rt_route="Subscribe"} 1

(no response metrics until the subscription is ended)

2024-03-21_19-41-33

In the meantime, envoy provides such statistics, including in the response counter:

envoy_cluster_grpc_Subscribe_response_message_count{envoy_grpc_bridge_method="CounterService",envoy_grpc_bridge_service="counter",envoy_cluster_name="lve_server"} 327336

How should the problem be solved?

route_request_total (or another dedicated metric) should calculate the number of events in long-lived streaming

Any alternatives you've considered?

How would users interact with this feature?

No response

Would you like to work on this feature?

Maybe

loyd avatar Mar 21 '24 19:03 loyd

It sounds like this is more generally asking for gRPC-specific metrics. As far as I understand, it wouldn't be possible for Linkerd to measure this sort of thing arbitrarily at the HTTP-level. We'd have to parse the response bodies to count the number of gRPC messages included in the response, for instance.

olix0r avatar Apr 03 '24 17:04 olix0r

It seems that envoy count frames for this metric. I think it's a reasonable trade-off. Although, I'm not sure it's easy to implement with outdated hyper (0.14).

Anyway, will a patch with this metric be considered?

loyd avatar Apr 04 '24 14:04 loyd

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in 14 days if no further activity occurs. Thank you for your contributions.

stale[bot] avatar Jul 03 '24 18:07 stale[bot]

Linked PR is activity

loyd avatar Jul 08 '24 07:07 loyd

The bot's logic is ill-conceived =(. It should be reopened

loyd avatar Aug 04 '24 20:08 loyd