Support propagation of trace context headers in http gateway
Checklist
- [X] My issue is specific & actionable.
- [X] I am not suggesting a protocol enhancement.
- [X] I have searched on the issue tracker for my issue.
Description
The HTTP gateway should support propagation of trace context headers using the W3C Trace Context recommendation.
This consists of supporting two optional HTTP headers traceparent and tracestate on incoming requests, adopting the trace id from the traceparent header for internal traces and emitting updated or new headers on responses.
This will allow:
- people using the gateway as a component in their infrastructure to correlate traces that pass through it
- gateway providers to trace incoming requests and correlate with logs
- performance optimizers to correlate traces with experiments
Of particular use will be the "sampled" flag on the emitted traceparent header that indicates whether the request was sampled for tracing by Kubo.
Implementation Note
This should be as simple as using the opentelemetry TraceContext, roughly:
func (h *Handler) ServeHTTP(w http.ResponseWriter, r *http.Request) {
p := propagation.TraceContext{}
ctx := p.Extract(r.Context(), propagation.HeaderCarrier(r.Header))
// do stuff
p.Inject(ctx, propagation.HeaderCarrier(w.Header()))
}
The HTTP handler is already wrapped with the OTel handler which applies any propagators (see https://github.com/ipfs/kubo/blob/master/core/corehttp/gateway.go#L106 and https://github.com/open-telemetry/opentelemetry-go-contrib/blob/main/instrumentation/net/http/otelhttp/handler.go#L130).
There's an env var in the spec, OTEL_PROPAGATORS, that allows one to install propagators, and it is specified with a default of "tracecontext,baggage" (see https://opentelemetry.io/docs/reference/specification/sdk-environment-variables/#general-sdk-configuration). I've wired that up in this PR: https://github.com/ipfs/kubo/pull/9169 (just needs a sharness test to ensure the propagation works and doesn't break).
Neat. Do you need help on the tests for #9169 @guseggert ?
That would be great if you have the time!
That would be great if you have the time!
@guseggert see https://github.com/ipfs/kubo/pull/9180