Proxy spans from subprocess

[cpuguy83]

Problem Statement

I work on containerd. I am looking at getting tracing throughout the containerd stack.

Unfortunately one of the things containerd relies on is (multiple) subprocesses, 1 per container(-ish). Because of the density of these shims, we need to keep the memory footprint extremely light and do not want to import a bunch of extra packages into those binaries (or the complexities of configuring those).

Instead I'm working on passing traces through the event system, which the shims already use for other things. From the main containerd process I'm planning to have a worker that sucks up those spans and passes them to the configured span processors OnEnd function. Unfortunately in order to call this function, the spans need to be ReadOnlySpans which can only be implemented inside the otel lib, and there doesn't seem to be a way to create one from existing data (I am passing the official protos over the wire to the event system).

Is there something else I should be trying to here?

[MrAlias]

Embedding a ReadOnlySpan interface in your implementation of the type will also embed the private method. E.g.

type span struct {
	trace.ReadOnlySpan
	// ...
}

Any unimplemented method that is called of your type will panic. This is something that should be captured in a test to ensure on upgrade all methods are implemented. The OTel Go project explicitly does not provide compatibility guarantees for this type.

[dmathieu]

@cpuguy83 this issue has been waiting for your answer for a few months. Can it be closed?

[cpuguy83]

I would say this is less a question and more "I'm trying to accomplish X and it doesn't seem possible"

[dmathieu]

Do you need to pass spans between processes? If each process is able to export the spans it creates, then you could setup some custom propagator which would allow creating spans that are childs of the parent process one.

There is this tool, otel-cli which uses environment variables to pass trace context for example. this proposal isn't really active. But it's heading towards the same thing.

[cpuguy83]

What we really want to do is export the span to another process without importing a bunch of new things or passing down new configurations. For the case of containerd it already has an event system that would be suitable for routing things. And basically the main containerd daemon would forward those spans along to the configured OTLP endpoint.

[dmathieu]

I feel like that may be an edge case, so you may be better off setting up something of your own rather than having something supported by otel. In any case, this kind of change is also not something that can be taken on by otel-go directly. It should be a specification first.

[cpuguy83]

The issue here was, IIRC, with the go interfaces containing unexported fields so nothing outside of the otel-go lib can implement them.