opentelemetry-grpc-gateway-boilerplate icon indicating copy to clipboard operation
opentelemetry-grpc-gateway-boilerplate copied to clipboard

Passing traceparent header doesnt appear to work

Open johnmayock opened this issue 1 year ago • 2 comments

Hello. I have followed this guide (and others) but I cannot successfully make a http json request for a grpc endpoint (which has traceparent present in a header of the GET request) to propagate through this OTEL "instrumented" grpc-gateway" following this boilerplate guide.

Use case: An ingress controller is OTEL enabled and generates a trace/span (sent to OTEL collector) upon http request to a service running in k8s. The ingress routes that request to the grpc-gateway (compiled for the service) on port 80.

GET /v1/service-name/api/ HTTP/1.1 traceparent: 00-000000000000000010c04a5dc44899bb-83fda3a85333b7a8-01 User-Agent: Go-http-client/1.1

This works when the request is sent to an auto-instrumented flask application (python) so I know the format of the OTEL w3c context is correct but I am trying to get the grpc-gateway to propagate this so that the span it creates while handling the http request (before it passes toe the grpc service endpoint) is linked as a child of the parent (provided from the inbound http get traceparent).

This guide suggests that setting the otel test map propagator will do that but it doesnt work for me.

I've code inspected the UnaryClientInterceptor here: https://github.com/open-telemetry/opentelemetry-go-contrib/blob/5cc3715df180510fa37f11bd6921c7e546dfcf78/instrumentation/google.golang.org/grpc/otelgrpc/interceptor.go#LL63C61-L63C61

And it seems to not extract the OTEL context but create a new root span and inject that forwards.

This would not propagate the context which is what I am seeing.

Whereas the UnaryStreamInterceptor: https://github.com/open-telemetry/opentelemetry-go-contrib/blob/5cc3715df180510fa37f11bd6921c7e546dfcf78/instrumentation/google.golang.org/grpc/otelgrpc/interceptor.go#L302

does correctly extract the context

ctx = extract(ctx, cfg.Propagators)

	name, attr := spanInfo(info.FullMethod, peerFromCtx(ctx))
	ctx, span := tracer.Start(
		trace.ContextWithRemoteSpanContext(ctx, trace.SpanContextFromContext(ctx)),

Does this boilerplate guide suggest the it propagates a request (with a traceparent header in the http request) through?

The trace example (which uses a stdout trace exporter) shows no propagation, it displays the trace created by the UnaryClientInterceptor but not in the use case where the http request already has a W3C context present.

johnmayock avatar Jun 09 '23 12:06 johnmayock