No custom traces when upgrading to v3 (Az Container App)

[JoeGaggler]

Describe the bug We upgraded Datadog.Trace.Bundle from 2.59.0 to 3.3.1 and no longer have custom traces in Datadog. Automatic traces still are working, i.e. ASP.NET Core Requests.

To Reproduce / Runtime environment Steps to reproduce the behavior:

• Azure Container App
• Linux Docker image: FROM mcr.microsoft.com/dotnet/aspnet:8.0
• Datadog serverless-init: 1.2.8
• NuGet: Datadog.Trace.Bundle 3.3.1
• In code: invoke Datadog.Trace.Tracer.Instance.StartActive to create custom traces.

Expected behavior Expecting to still have custom traces in addition to automatic traces. Downgrading to 2.59.0 immediately resolved the issue.

Additional context Nothing in the v2 to v3 migration guide seemed to point to the issue. Wondering if the serverless-init is somehow not compatible with the latest NuGet package.

[andrewlock]

Hi @JoeGaggler, sorry to hear you're having issues.

The symptom you're describing is typically related to using v2 of the automatic instrumentation with v3 of the manual instrumentation. Would it be possible to show your dockerfile? Is it possible you're installing the 2.x tracer there?

[JoeGaggler]

Sure, here is our dockerfile:

FROM mcr.microsoft.com/dotnet/sdk:8.0 AS build
ARG PAT
WORKDIR /source

COPY containerapp containerapp
COPY NuGet.config .
RUN dotnet nuget update source "Internal Feed" -u pat -p "${PAT}" --store-password-in-clear-text --configfile NuGet.config ; \
    dotnet restore "containerapp/Project.csproj" \
    --locked-mode \
    -v n \
    --configfile NuGet.config \
    --ignore-failed-sources \
    --runtime linux-x64 \
    /p:PublishReadyToRun=true

RUN dotnet publish "containerapp/Project.csproj" \
    --no-restore \
    --configuration Release \
    --output /publish \
    --no-self-contained \
    --runtime linux-x64 \
    /p:PublishReadyToRun=true

FROM mcr.microsoft.com/dotnet/aspnet:8.0 as base 

COPY --from=datadog/serverless-init:1.2.8 /datadog-init /app/datadog-init

WORKDIR /app
COPY --from=build /publish .

RUN /app/datadog/createLogPath.sh

USER $APP_UID

ENTRYPOINT ["/app/datadog-init"]
CMD ["dotnet", "Project.dll"]

[andrewlock]

Thanks @JoeGaggler, that's really useful, I'm a little confused about one aspect - in order to use the Datadog.Trace.Bundle Nuget, you need to set a bunch of environment variables (CORECLR_PROFILER_PATH, DD_DOTNET_TRACER_HOME etc), but AFAICT, your dockerfile isn't setting any of those 🤔

Is there something I'm missing, with that dockerfile as far as I can tell, it shouldn't work at all i.e. you shouldn't get auto-instrumentation or manual traces 😅

[JoeGaggler]

Ah, yes, sorry about that. Those environment variables are being set in the Azure Container App via an ARM template, below with minor redactions:

resourceGroup: 'REDACTED_SECRET_VALUE'
location: 'centralus'
name: 'REDACTED_SECRET_VALUE'
type: Microsoft.App/containerApps
properties:
  template:
    containers:
      - name: 'REDACTED_SECRET_VALUE'
        image: REDACTED_SECRET_VALUE
        env:
          - name: CORECLR_ENABLE_PROFILING
            value: '1'
          - name: CORECLR_PROFILER
            value: '{846F5F1C-F9AE-4B07-969E-05C26BC060D8}'
          - name: CORECLR_PROFILER_PATH
            value: '/app/datadog/linux-x64/Datadog.Trace.ClrProfiler.Native.so'
          - name: LD_PRELOAD
            value: '/app/datadog/linux-x64/Datadog.Linux.ApiWrapper.x64.so'
          - name: DD_API_KEY
            secretRef: datadog-api-key
          - name: DD_AZURE_RESOURCE_GROUP
            value: 'REDACTED_SECRET_VALUE'
          - name: DD_AZURE_SUBSCRIPTION_ID
            value: 'REDACTED_SECRET_VALUE'
          - name: DD_DBM_PROPAGATION_MODE
            value: service
          - name: DD_DOTNET_TRACER_HOME
            value: '/app/datadog'
          - name: DD_ENV
            value: 'esone-dev'
          - name: DD_LOG_LEVEL
            value: 'info'
          - name: DD_LOGS_ENABLED
            value: 'true'
          # Container Apps already use the docker integration for log submission
          # - name: DD_LOGS_DIRECT_SUBMISSION_INTEGRATIONS
          #   value: ''
          - name: DD_LOGS_INJECTION
            value: 'true'
          - name: DD_RUNTIME_METRICS_ENABLED
            value: 'true'
          - name: DD_SERVICE
            value: 'REDACTED_SECRET_VALUE'
          - name: DD_SITE
            value: 'datadoghq.com'
          - name: DD_TRACE_ENABLED
            value: 'true'
          - name: DD_TRACE_DEBUG
            value: 'false'
          - name: DD_VERSION
            value: '24.1009.7.3_merge'
          - name: DD_TRACE_SAMPLE_RATE
            value: '1.0'
          - name: DD_PROFILING_ENABLED # NOTE: Datadog requires cpu >= 1.0 for profiling
            value: '1'
          - name: DD_PROFILING_ALLOCATION_ENABLED
            value: '1'
          - name: DD_PROFILING_EXCEPTION_ENABLED
            value: '1'
          - name: DD_PROFILING_HEAP_ENABLED
            value: '1'
          - name: DD_PROFILING_LOCK_ENABLED
            value: '1'
          - name: DD_PROFILING_WALLTIME_ENABLED
            value: '0'
        resources: # for consumption plan, memory must be 2x cpu (in Gi)
          cpu: '1.0'
          memory: '2.0Gi'

[andrewlock]

Thanks for that @JoeGaggler! Unfortunately I can't see anything obvious at all, that all looks ok to me 🤔 Would it be possible to raise a ticket with support so that we can look at collecting more details like the tracer log files etc? Thanks!

[JoeGaggler]

Will do, thanks @andrewlock!

[andrewlock]

Hey @JoeGaggler, just wanted to follow up to say that those logs you sent in the ticket were really useful, thank you! We'll should keep general communication there, but just wanted to say that we have managed to find a scenario that reproduces the issue you have. Specifically, we can reproduce the issue if you have a static constructor in your Program class, e.g. something like this:

public static class Program
{
    private static readonly Datadog.Trace.Tracer _tracer;
    static Program()
    {
        _tracer = Datadog.Trace.Tracer.Instance;;
    }

    public static void Main(string[] args)
    {
        var builder = WebApplication.CreateBuilder(args);
        var app = builder.Build();

        app.MapGet("/", () => {
            using var scope = _tracer.StartActive("custom-operation-name");
            return "Hello World!";
        });

        app.Run();
    }
}

Do you have code similar to that in your application? If not, would you mind sharing an example of the code (on the Zendesk ticket is fine!). Thanks!

[JoeGaggler]

Hi @andrewlock, We're not using an explicit static constructor in Program (Minimal API), but I will update the Zendesk ticket with the actual source code.

[andrewlock]

Hey @JoeGaggler, just FYI, this should be fixed by #6184, which will be part of the next release 🙂

[JoeGaggler]

That's awesome, thanks!

[JoeGaggler]

3.5.0 is working great for us, thanks!