Spans not available in Java Kafka consumer

[juliusz-cwiakalski]

Hi all, need a hint - I'm working on a setup where two spring boot services are talking through Kafka. service A emits an event1 that is consumed by service B. Then service B emits another event2 that is later consumed by service A. Both events has set x-datadog-trace-id and I can find all logs from both services in DataDog filtering by trace_id. However, I cannot see parts of service B processing in below graph (also no spans from this service, etc) - it seems I'm missing something in my setup.

Both services are running inside Kubernetes. Service B is built using new pipelines we're building from scratch and setup and has a sidecar taking care of communication with DataDog. Java arguments to run it are -javaagent:/app/dd-java-agent.jar -Ddd.agent.port=8126 -XX:FlightRecorderOptions=stackdepth=256 -Ddd.profiling.enabled=true -Ddd.logs.injection=true -Ddd.service=service-b -Ddd.env=prod -Dcom.sun.management.jmxremote=0.0.0.0 -Dcom.sun.management.jmxremote.rmi.port=9010 -Dcom.sun.management.jmxremote.port=9010 -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false'

Service A is built using old pipelines but with the same Java params as above. What I observe is that event 1 emitted from service A has the following Kafka headers:

{
	"dd-pathway-ctx-base64": "oRPXEoWN613etJ39h2OKu539h2M=",
	"x-datadog-parent-id": "796467821006552146",
	"x-datadog-sampling-priority": "1",
	"tracestate": "dd=s:1;t.dm:-1;t.tid:6570987e00000000",
	"x-datadog-tags": "_dd.p.dm=-1,_dd.p.tid=6570987e00000000",
	"traceparent": "00-6570987e000000005aeebb41438851ed-0b0d9f436a1ffc52-01",
	"x-datadog-trace-id": "6552380396850926061",
	"__TypeId__": "event1"
}

And event 2 emitted from service B has the following Kafka headers:

{
	"x-datadog-parent-id": "4188098030954103618",
	"x-datadog-sampling-priority": "1",
	"x-datadog-tags": "_dd.p.dm=-1,_dd.p.tid=6570987e00000000",
	"x-datadog-trace-id": "6552380396850926061",
	"__TypeId__": "event2"
}

clearly event2 is missing several headers that, AFAIK are added automatically by the DataDog agent in service A. Also, see that x-datadog-parent-id has a different value - is that fine, or should be the same? What should I change in the service B setup to make it all work correctly together? I'm quite new to DataDog - tried to find answers in docs and tried different settings but failing so far. Guys who set up old service A are no longer available, and we cannot find a difference in the setup now.

[ygree]

Can you please open a support ticket at https://www.datadoghq.com/support/?