Explore using API server tracing for additional context

[embano1]

Now that https://github.com/kubernetes/kubernetes/pull/94942 has landed, we might want to think about ways to use such tracing functionality to assist users in replicating/debugging time-travel issues, e.g. by injecting custom trace data, visualizing flows, etc.

Just thinking out loud here, haven‘t spend much time thinking deeper about the applicability of tracing in the various areas of sieve (beyond detection algo).

[lalithsuresh]

@embano1 thanks for sharing this. This might significantly simplify some of the mechanical work we've had to do thus far in finding causally related sequences of events.

Is this tracing context propagated to clients/controllers (and are they expected to further propagate it out)?

[embano1]

Sorry for my delayed response.

As per the implementation KEP (https://github.com/kubernetes/enhancements/pull/1458/files), my understanding is that for this KEP K8s controller instrumentation is out of scope, BUT incoming requests (to API server), as well as outgoing with trace information will be forwarded.

IMHO that means that if you instrument your controller accordingly (Otel) these traces should be propagated.