Support opentracing span references of FOLLOWS_FROM or opentelemtry span links

[shaicantor]

In the startSpan method, there's an optional spanOptions object which its type is inherited from opentracing. You can set the references array attribute which can have various causally related spans.

https://github.com/DataDog/dd-trace-js/blob/master/index.d.ts#L18 https://github.com/opentracing/opentracing-javascript/blob/fb897fa9d5fd18eb0bbcff9286d4daa476b6e4f7/src/tracer.ts#L7 https://github.com/opentracing/opentracing-javascript/blob/master/src/reference.ts#L33

https://github.com/opentracing/specification/blob/master/specification.md#references-between-spans

https://github.com/open-telemetry/opentelemetry-specification/blob/874a451e7f6ac7fc54423ee3f03e5394197be35b/specification/compatibility/opentracing.md#L120

https://github.com/open-telemetry/opentelemetry-specification/blob/034e706d2b93570dd04282ea34cd46fa4bb86d6c/specification/overview.md#links-between-spans

Is links or references supported in the datadog platform? If so, does the datadog exporter in the opentelemtry contrib repo support sending it? https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/9516285e706b9a4a8918372050cdad923bc72c25/exporter/datadogexporter

Thanks

[rochdev]

This is not currently supported by our APM so unfortunately this is not something that can work with either dd-trace nor dd-opentelemetry-exporter at the moment. We are however exploring our options to support this in the future, but it's unlikely to happen in the short term.

What is your use case? In a lot of cases this limitation can be worked around with childOf.

[shaicantor]

My use case is the following: Use opentelemetry node js agent and export to datadog.

OpenTelemetry messaging systems specification has the scenario of "Batch Receiving"

Given is a process P, that sends two messages to a queue Q on messaging system MS, and a process C, which receives both of them in one batch (Span Recv1) and processes each message separately (Spans Proc1 and Proc2).

Since a span can only have one parent and the propagated trace and span IDs are not known when the receiving span is started, the receiving span will have no parent and the processing spans are correlated with the producing spans using links.

I want to datadog to have the option to show links between spans cause that's how it is modeled in OTLP. I do see a reference to the links in the datadog agent - https://github.com/DataDog/datadog-agent/blob/0a44566e4b2a7d1bf3da30fca4bbce4849d4e73c/pkg/trace/pb/otlppb/trace.pb.go#L338

Process P: | Span Prod1 | Span Prod2 |
--
Process C:                      | Span Recv1 |
                                        | Span Proc1 |
                                               | Span Proc2 |

Field or Attribute	Span Prod1	Span Prod2	Span Recv1	Span Proc1	Span Proc2
Span name	"Q send"	"Q send"	"Q receive"	"Q process"	"Q process"
Parent				Span Recv1	Span Recv1
Links				Span Prod1	Span Prod2
SpanKind	PRODUCER	PRODUCER	CONSUMER	CONSUMER	CONSUMER
Status	Ok	Ok	Ok	Ok	Ok
net.peer.name	"ms"	"ms"	"ms"	"ms"	"ms"
net.peer.port	1234	1234	1234	1234	1234
messaging.system	"rabbitmq"	"rabbitmq"	"rabbitmq"	"rabbitmq"	"rabbitmq"
messaging.destination	"Q"	"Q"	"Q"	"Q"	"Q"
messaging.destination_kind	"queue"	"queue"	"queue"	"queue"	"queue"
messaging.operation			"receive"	"process"	"process"
messaging.message_id	"a1"	"a2"		"a1"	"a2"

[rochdev]

Thanks for the clarification. This is definitely not supported right now, and the way we usually work around this is to have one consumer span per message instead of for the batch when that's possible and if it makes sense to do so, so when messages are handled individually. These spans would then ignore the incoming request and would lose that information instead of losing the information from the producers. Basically, right now it's a trade-off and you have to pick the information most relevant to you.

If you use dd-trace, you could use a similar workaround if it can be applied to your use case. If you use the OpenTelemetry Datadog exporter, then I'd recommend opening an issue on that repo instead since it would be unrelated to dd-trace. In any case, I'd still recommend opening a feature request through support to make sure you're added to the list of users asking for this.

[g-farrow]

We have a similar requirement. Specifically trying to link spans together across AWS services, such as DynamoDB or EventBridge. For example:

• Lambda A "producer" writes an event to EventBridge
• Lambda B "consumer" receives event and processes it

"producer"  -->  EventBridge -->  "consumer"

What we want to do is to have a span in the "producer" and a span in the "consumer" linked together as siblings, so that we can see the flow of events through a system. Including:

• Flame graph for "producer" shows both spans
• Flame graph for "consumer" shows both spans
• Trace maps show the connection between the 2 Lambda Functions

Note: we are using dd-trace for creating new spans and had been attempting to use the reference functionality from opentracing to achieve this requirement. We couldn't get this working and that's what lead us to find this ticket.

[melvinkcx]

I'm running into a similar issue with using span linking in APM. I'm using the OpenTelemetry collector with both Datadog and Jaeger exporter.

On Jaeger, I can trace the FOLLOWS_FROM span, but not on Datadog:

[ithompson-gp]

Interested to hear and and all progress on this. Seems like a major gap with regard tracing in EDA