Add TraceId based search

[Pryz]

Observability super powers come from been able to correlate multiple telemetry. Jumping from one telemetry to another require them to share the same context. Distributed tracing solutions propagate context using a Trace ID. Using the same Trace ID and attaching them to profiles could give a very powerful way of correlating traces and profiles.

[yeya24]

I am interested in this as well. But this sounds more like an instrumentation question to me? I think it is possible to use traceID in pprof labels, but seems this is not a common practice and I don't see anyone else doing it. Would like to hear more people doing such things.

[kakkoyun]

+1 to @yeya24 it's mostly an instrumentation problem. The biggest issue I see here is the profiling data is sampling-based and not request scoped. So in a given interval, we might have processed several requests. That being said we would be happy to make life easier for users to achieve this goal by providing an instrumentation library or extending existing libraries.

We can also bounce off some ideas to extend the parca-agent to support this scenario. In theory, we can obtain tracing ids from the network stack using eBPF and we can use these to argument profiles with labels. It's easier said than done of course :D But at least we can start discussing the possibilities here.

[brancz]

Agreed that pprof labels are the place for this. Maybe a long shot but we're already walking the stack anyway in eBPF, so I imagine with some languages (like Go) we will probably get pretty far with heuristics like looking for and extracting the trace id from things on the stack (context.Context in the Go case).

[yeya24]

Pyroscope instead implement this correlation by searching profiles using profile ID from traces. Trace will have profile ID as a tag. https://github.com/pyroscope-io/pyroscope/blob/main/examples/tracing/jaeger/README.md

[brancz]

Yeah, I think it should be possible to do it both ways.