kubo icon indicating copy to clipboard operation
kubo copied to clipboard

Support propagation of trace context headers in http gateway

Open iand opened this issue 3 years ago • 4 comments

Checklist

  • [X] My issue is specific & actionable.
  • [X] I am not suggesting a protocol enhancement.
  • [X] I have searched on the issue tracker for my issue.

Description

The HTTP gateway should support propagation of trace context headers using the W3C Trace Context recommendation.

This consists of supporting two optional HTTP headers traceparent and tracestate on incoming requests, adopting the trace id from the traceparent header for internal traces and emitting updated or new headers on responses.

This will allow:

  1. people using the gateway as a component in their infrastructure to correlate traces that pass through it
  2. gateway providers to trace incoming requests and correlate with logs
  3. performance optimizers to correlate traces with experiments

Of particular use will be the "sampled" flag on the emitted traceparent header that indicates whether the request was sampled for tracing by Kubo.

Implementation Note

This should be as simple as using the opentelemetry TraceContext, roughly:

func (h *Handler) ServeHTTP(w http.ResponseWriter, r *http.Request) {
    p := propagation.TraceContext{}
    ctx := p.Extract(r.Context(), propagation.HeaderCarrier(r.Header))
    
    // do stuff

   p.Inject(ctx, propagation.HeaderCarrier(w.Header()))
}

iand avatar Aug 05 '22 11:08 iand

The HTTP handler is already wrapped with the OTel handler which applies any propagators (see https://github.com/ipfs/kubo/blob/master/core/corehttp/gateway.go#L106 and https://github.com/open-telemetry/opentelemetry-go-contrib/blob/main/instrumentation/net/http/otelhttp/handler.go#L130).

There's an env var in the spec, OTEL_PROPAGATORS, that allows one to install propagators, and it is specified with a default of "tracecontext,baggage" (see https://opentelemetry.io/docs/reference/specification/sdk-environment-variables/#general-sdk-configuration). I've wired that up in this PR: https://github.com/ipfs/kubo/pull/9169 (just needs a sharness test to ensure the propagation works and doesn't break).

guseggert avatar Aug 05 '22 12:08 guseggert

Neat. Do you need help on the tests for #9169 @guseggert ?

iand avatar Aug 05 '22 12:08 iand

That would be great if you have the time!

guseggert avatar Aug 05 '22 13:08 guseggert

That would be great if you have the time!

@guseggert see https://github.com/ipfs/kubo/pull/9180

iand avatar Aug 11 '22 16:08 iand