[Question] TraceId propagation in Spring Boot 3.x app using RestTemplate and Micrometer

[mszumlanski]

Hey, I see some questions related to tracing on my app. I have 2 services - Service1 is using Spring Boot 3.1.6 with SL4J/Logback.

I use DataDog as a Java agent (latest version) and added following dependencies (versions brought from Spring Boot BOM):

io.micrometer:micrometer-tracing
io.micrometer:micrometer-tracing-bridge-otel
io.micrometer:micrometer-registry-datadog

and following logback configuration:

  <appender class="ch.qos.logback.core.ConsoleAppender" name="STDOUT">
    <encoder>
      <pattern>%d{yyyy-MM-dd HH:mm:ss.SSS} [%X{traceId}] [%X{dd.trace_id}] %-5level %logger{5} - %msg%n</pattern>
    </encoder>
  </appender>

  <appender class="ch.qos.logback.core.ConsoleAppender" name="STDOUT_JSON">
    <encoder class="net.logstash.logback.encoder.LogstashEncoder">
      <!-- https://github.com/logfellow/logstash-logback-encoder/issues/387#issuecomment-581603444 -->
      <findAndRegisterJacksonModules>false</findAndRegisterJacksonModules>
    </encoder>
  </appender>

I also have following configuration in .yml file:

  tracing:
    sampling:
      probability: 1.0
  datadog:
    metrics:
      export:
        uri: https://app.datadoghq.com
        enabled: true
        api-key: [...]
        step: 10s
        application-key: [...]
  metrics:
    tags:
      application: ${spring.application.name}

I did not add any extra code related to tracing or metrics.

First off I'm calling one of Service1 endpoints with no trace context headers whatsoever. Initially on Service1 I'm seeing different values in stdout for traceId and dd.trace_id:

traceId: "77826f8c5e0d47f1044e39e32f07cebd"
dd.trace_id: "7000308470029666414"

Afer I make a call to Service2 using RestTemplate I can see that W3C'straceparent header in request, specifically it's traceId part is made up of _dd.p.tid tag and dd.trace_id hex representation values (made up of two 64 bit parts). These are trace related headers I can observe in request:

x-datadog-trace-id:"7000308470029666414"
x-datadog-tags:"_dd.p.dm=-1,_dd.p.tid=6569ed5700000000"
traceparent:"00-6569ed570000000061261776da46c86e-28f00f7e507843cb-01"

It is then deserialised and now in Service2 I see following trace values in stdout:

traceId: "6569ed570000000061261776da46c86e" # grabbed from w3c `traceparent` traceId part
dd.trace_id: "7000308470029666414" # I'm assuming grabbed directly from `x-datadog-trace-id` or extracted from `traceparent`

• Are u guys by default injecting any sort of instrumentation/interceptors/whatever so that RestTemplate builds a traceparent header out of DataDog values? It was my initial assumption that framework will grab traceId value and not dd.* stuff to build that part of request metadata.
• If this is intended and you're full on overriding default framework behaviour then I would expecttraceId in Service1 to also have your 6569ed570000000061261776da46c86e value and not 77826f8c5e0d47f1044e39e32f07cebd. It is now impossible to track logs from same flow using traceId. Any clue why traceId in Service1 is set to this seemingly out-of-place value? This seems like a bug.

EDIT: Spring Boot 3.2.0 - same issue

[zromano]

We experienced the same thing...

We saw this happen on Nov 17, unclear which version caused it, but we know this happened sometime after version 1.23.0. If we pin to version 1.23.0 we see the expected dd.trace_id, but on the versions after Nov 17 we get the hex ones shown in the issue above.

I believe it's because you're generating a trace ID in this PR: https://github.com/DataDog/dd-trace-java/pull/6189/files

---

Also as a fix for OP, you're probably pull the latest dd-trace-java in your Dockerfile like so:

# Datadog Java tracing
# https://docs.datadoghq.com/tracing/setup/java/
# https://mvnrepository.com/artifact/com.datadoghq/dd-java-agent
RUN wget -nv -O /opt/dd-java-agent.jar 'https://dtdg.co/latest-java-tracer'

You can update it to pin to a specific version like so:

################################################################################
#### DOWNLOAD DATADOG TRACING AGENT ####
# FROM: https://docs.datadoghq.com/tracing/setup/java/
# https://mvnrepository.com/artifact/com.datadoghq/dd-java-agent
################################################################################
RUN wget -O /opt/dd-java-agent.jar 'https://github.com/DataDog/dd-trace-java/releases/download/v1.23.0/dd-java-agent.jar'
################################################################################

[roeniss]

Hi. Did you solve this issue? My team is struggling with the same tissue.