promscale icon indicating copy to clipboard operation
promscale copied to clipboard

Add telemetry event for Jaeger gRPC remote storage writer

Open arajkumar opened this issue 2 years ago • 1 comments

Relates to #1535

arajkumar avatar Aug 04 '22 13:08 arajkumar

@VineethReddy02 We already have telemetry event around otel ingestion and #1535 is implemented on top of otel. Hence all the events which we send for otel ingestion is also applicable for Jaeger ingestion. Do we need any additional signal from the product perspective?

arajkumar avatar Aug 11 '22 06:08 arajkumar

I agree with your point Arun, But we need to know if the user is using Otel or Jaeger ingestion endpoints or both? And rest all we can see from Otel-specific telemetry that has already been captured. Maybe it's a good idea to change telemetry keys from otel to traces to make it more generic. So that in telemetry the data points with traces represents traces in general i.e both Jaeger and Otel.

VineethReddy02 avatar Aug 17 '22 11:08 VineethReddy02

I've been exploring following options to enable telemetry,

  1. Add metrics in WriteSpan and expose it to telemetry ( similar to e.g. https://github.com/timescale/promscale/blob/master/pkg/jaeger/store/store.go#L64)
  2. Expose intrinsic metrics exposed by gRPC framework(grpc_server_handling_seconds_count{grpc_method="WriteSpan"})
  3. Use internal span attribute populated by jaeger framework(internal.span.format=jaeger)

Option 1 and Option 2 requires code changes in promscale and extension, but 3 requires change only in extension.

arajkumar avatar Aug 23 '22 06:08 arajkumar