with istio, trace context is not propagated

[keyolk]

I'm trying to set up the distributed tracing with Istio. And confirmed that the envoy issues related HTTP headers like

• x-request-id
• x-datadog-trace-id
• x-datadog-parent-id
• x-datadog-sampling-priority and passed it to the application.

Then, what I expected was the dd-trace-java in my application extracts the trace context and propagates it to the next span by default. But the application issues another trace context that has a different trace id. And also I can see it from the DD trace view. Which has been separated.

I checked this behavior from the envoy's HTTP debug log.

the below is request header from istio-ingress-gateway to sidecar proxy. removed some sensitive lines.

 2022-05-11T13:20:33.962118Z   debug   envoy http      [C41789] new stream
 2022-05-11T13:20:33.962201Z   debug   envoy http      [C41789][S13892805576615822259] request headers complete (end_stream=true):
 ':method', 'GET'
 ':path', '/v3/applications'
 ':scheme', 'https'
 'user-agent', 'curl/7.79.1'
 'accept', '*/*'
 'x-request-id', 'ec13bace-8646-40e4-8506-17ff5ad1116d'
 'x-forwarded-proto', 'https'
 'x-envoy-attempt-count', '1'
 'x-datadog-trace-id', '1907456185000481633'
 'x-datadog-parent-id', '9102782866208357317'
 'x-datadog-sampling-priority', '1'

here, initial trace-id is "1907456185000481633"

and next is from application to sidecar proxy

 2022-05-11T13:20:33.962207Z   debug   envoy http      [C41789][S13892805576615822259] request end stream
 2022-05-11T13:20:33.974151Z   debug   envoy http      [C37345] new stream
 2022-05-11T13:20:33.974208Z   debug   envoy http      [C37345][S9159178598143297846] request headers complete (end_stream=false):
 ':method', 'POST'
 ':scheme', 'http'
 'content-type', 'application/grpc'
 'te', 'trailers'
 'user-agent', 'grpc-java-netty/1.44.0'
 'x-datadog-trace-id', '2875767659309492513'
 'x-datadog-parent-id', '8474561997615422615'
 'x-datadog-sampling-priority', '1'
 'grpc-accept-encoding', 'gzip'
 2022-05-11T13:20:33.974497Z   debug   envoy http      [C37345][S9159178598143297846] request end stream
 2022-05-11T13:20:33.982197Z   debug   envoy http      [C37345][S9159178598143297846] encoding headers via codec (end_stream=false):
 ':status', '200'
 'content-type', 'application/grpc'
 'grpc-encoding', 'identity'
 'grpc-accept-encoding', 'gzip'
 'x-envoy-upstream-service-time', '7'
 'date', 'Wed, 11 May 2022 13:20:33 GMT'
 'server', 'envoy'
 2022-05-11T13:20:33.982416Z   debug   envoy http      [C37345][S9159178598143297846] encoding trailers via codec:
 'grpc-status', '0'

here new trace contexet is issued, trace id is "2875767659309492513" now the trace id is propagated to the next request the below

 2022-05-11T13:20:33.985589Z   debug   envoy http      [C37345] new stream
 2022-05-11T13:20:33.985635Z   debug   envoy http      [C37345][S18146508500385699012] request headers complete (end_stream=false):
 ':method', 'POST'
 ':scheme', 'http'
 'content-type', 'application/grpc'
 'te', 'trailers'
 'user-agent', 'grpc-java-netty/1.44.0'
 'x-datadog-trace-id', '2875767659309492513'
 'x-datadog-parent-id', '2326705856414738264'
 'x-datadog-sampling-priority', '1'
 'grpc-accept-encoding', 'gzip'
 2022-05-11T13:20:33.985791Z   debug   envoy http      [C37345][S18146508500385699012] request end stream
 2022-05-11T13:20:33.989584Z   debug   envoy http      [C37345][S18146508500385699012] encoding headers via codec (end_stream=false):
 ':status', '200'
 'content-type', 'application/grpc'
 'grpc-encoding', 'identity'
 'grpc-accept-encoding', 'gzip'
 'x-envoy-upstream-service-time', '3'
 'date', 'Wed, 11 May 2022 13:20:33 GMT'
 'server', 'envoy'
 2022-05-11T13:20:33.989793Z   debug   envoy http      [C37345][S18146508500385699012] encoding trailers via codec:
 'grpc-status', '0'
 2022-05-11T13:20:33.994296Z   debug   envoy http      [C41789][S13892805576615822259] encoding headers via codec (end_stream=false):
 ':status', '400'
 'content-type', 'application/json'

Is there something I have to do more?

I'm using the latest version of dd-java-agent.

$ curl -L 'https://dtdg.co/latest-java-tracer' -o /app/dd-java-agent.jar

I use the below stack

kotlin.code.style=official
awsSdkVersion = 2.17.123
kotlinVersion = 1.6.10
springBootVersion = 2.6.6
sqlDelightVersion = 1.5.3
grpcVersion = 1.44.0
grpcKotlinVersion = 1.2.1
protobufVersion = 3.19.2
lettuceVersion = 6.1.6.RELEASE
sentrySpringBootVersion = 5.7.0
mariadbJdbcVersion = 3.0.3
hikariVersion = 5.0.1

and some envs

DD_VERSION=dev
DD_ENV=dev
DD_SERVICE=my