Datadog and otel_ngx_module

[dgoffredo]

Datadog wants to integrate more closely with otel_ngx_module, if for no other reason than ingress-nginx plans to drop support for nginx-opentracing in favor of otel_ngx_module.

This has been discussed previously in an ingress-nginx issue and in an opentelemetry-cpp issue.

The Datadog Agent (Datadog's equivalent to an OpenTelemetry collector) speaks OTLP, and so today Datadog customers can use otel_ngx_module as is.

However, Datadog tracing libraries support some features that are not currently expressible in the OpenTelemetry API. The most prominent example in the case of C++ is how sampling is configured.

Instead of restricting Datadog to the collector, we are considering (with the blessing of opentelemetry-cpp and ingress-nginx) writing a library that implements the opentelemetry-cpp API by defining a TracerProvider and related classes. Then, OpenTelemetry projects that wish to provide full-featured Datadog tracing can depend on this library and install its TracerProvider at runtime. Any code written in terms of the opentelemetry-cpp API would remain unchanged.

Let's call this hypothetical library "dd-opentelemetry-cpp".

I've opened up this issue to discuss with the maintainers of otel_ngx_module whether it would be acceptable to propose changes that would integrate dd-opentelemetry-cpp into otel_ngx_module.

At a glance, the configuration and runtime setup of the nginx module would have to be revised to allow for a "provider" degree of freedom, and whatever modifications to existing configuration that would require. The build would also be modified to include dd-opentelemetry-cpp.

dd-opentelemetry-cpp would be a C++17 library that depends only on opentelemetry-cpp, dd-trace-cpp, and optionally libcurl.

Finally, as background information, here is a survey of relevant Datadog libraries:

• dd-opentracing-cpp is an OpenTracing-based plugin that we seek to deprecate. It is what is currently used by ingress-nginx in conjunction with nginx-opentracing.
• dd-trace-cpp is a new dedicated Datadog tracing library that I wrote to replace dd-opentracing-cpp. It's a C++17 library with no dependencies aside from, optionally, libcurl.
• nginx-datadog is a Datadog-specific nginx module, based largely on nginx-opentracing, that uses dd-trace-cpp under the hood and allows for minimal-config tracing of nginx using Datadog.

What do you think about the idea of integrating the hypothetical dd-opentelemetry-cpp into otel_ngx_module?

[dgoffredo]

@seemk @johanneswuerbach @tobiasstadler @esigo

Pardon the @'ing. I'm wondering who/where I should ping to get feedback on the idea described above. I'm new to the governance of opentelemetry-cpp-contrib.

[esigo]

Hi @dgoffredo thanks for the reminder. Per discussion in opentelemetry-cpp repo, this should be the way. As I'm maintaining the integration in ingress-nginx repo, I could also support you here. Please also note that my capacity for opensource activities (as hobby) is quite limited these days.

[dgoffredo]

Thanks, @esigo. Ideally this initiative will not take any more of your time than what is required for a code review. I wanted to get this project's blessing before I proceeded to begin work on the proposed dd-opentelemetry-cpp library. Wouldn't want to write it if the nginx module project were unwilling to use it.

I have this work planned for this quarter (July through September), unless priorities change. So, you'll be hearing from me again somewhere in that time period.

[dgoffredo]

Happy 2024. I am revisiting dd-opentelemetry-cpp this quarter.