DataDog
Unknown service created when using Hangfire instrumentation
Describe the bug We use the Hangfire OTEL instrumentation (located here) to add root spans to the running jobs.
OTEL is configured like so to add the .NET Activity instrumentation but the export is delegated to DD Trace's automated OTEL support:
using var tracer = Sdk
.CreateTracerProviderBuilder()
.AddHangfireInstrumentation()
.Build();
However this results in the hangfire root spans showing up under their own service "unknown_service:dotnet" in the DD console. All other spans from the service are correctly aggregated.
The span metadata does have the correct service name:
This application is running in App Services with the tracer baked into the image:
COPY --from=datadog/serverless-init:1 /datadog-init /app/datadog-init
COPY --from=datadog/dd-lib-dotnet-init /datadog-init/monitoring-home/ /dd_tracer/dotnet/
ENV DD_SERVICE=(hidden)-integration-api
ENV DD_LOGS_INJECTION=true
ENV DD_LOGS_DIRECT_SUBMISSION_INTEGRATIONS='Serilog'
ENV DD_TRACE_OTEL_ENABLED=true
ENV DD_DATA_STREAMS_ENABLED=true
ENV DD_TRACE_SPAN_ATTRIBUTE_SCHEMA='v1'
Runtime environment:
- Instrumentation mode: Automatic
- Tracer version: 2.50.0
- OS: Azure App Services (Linux)
- CLR: .NET 8
Thanks @Hawxy, I was able to replicate this and will work on getting a better fix implemented for the Tracer.
There does seem to be a potential workaround though by setting the environment variable OTEL_SERVICE_NAME to the same value as DD_SERVICE.
Note that we are currently working on supporting remapping OpenTelemetry environment variables to their Datadog equivalent ones: https://github.com/DataDog/dd-trace-dotnet/pull/5661 I don't think this would have an impact here though, but just wanted to raise it.
But I'll work on getting a fix to address it without needing to do that
Hi @Hawxy I've merged a fix for this in https://github.com/DataDog/dd-trace-dotnet/pull/5671 It isn't released yet, but we'll let you know when the next release is out with the fix.