Kafka consumer client does not inherit service name

[ffalexandererben]

The Kafka consumer client instrumentation uses a hardcoded service name kafka and does not inherit the service name. I think it should inherit the service name given via environment or vm property - maybe this behaviour should even be configurable. It's possible to work around this issue by using dd.service.mapping, but that is only a workaround in my opinion.

https://github.com/DataDog/dd-trace-java/blob/5bc3ec82c2adf2e7a7254d046fc9b725a4370621/dd-java-agent/instrumentation/kafka-clients-0.11/src/main/java/datadog/trace/instrumentation/kafka_clients/KafkaConsumerInstrumentation.java#L107

[tylerbenson]

In my opinion, it should probably be kafka for the producer but use the configured service name for the consumer. What do you think?

[ffalexandererben]

I agree, that makes sense.

Just one thought on the consequences if the service name for Kafka consume traces is changed to the configured service name: The operation name kafka.consume might cause a problem for services that also have a HTTP interface, where the dominant trace operation name will be servlet.request or express.request or whatever framework is used. As fas as I'm aware, selecting operation names in the APM UI is still a beta feature, so the consume traces will not be shown in the UI for the service (or the other way around if the Kafka traces are more frequent).

But this is a problem for the current state of the Kafka instrumentation as well: Produce traces and consume traces cannot show up at the same time in the UI without the operation selector available - the most active operation wins as it is the primary operation in terms of the statistics.

[matt-canty-dragon]

But this is a problem for the current state of the Kafka instrumentation as well: Produce traces and consume traces cannot show up at the same time in the UI without the operation selector available - the most active operation wins as it is the primary operation in terms of the statistics.

@ffalexandererben I am curious as to what you would recommend at this time? I am in a position right now where I should choose a path:

1. Do nothing, live with the issue until a resolution is released.
2. Map service kafka to service-name
3. Map service kafka to service-name-kafka (or service-name-kafka-client) current preference

Rationale for number 3 is:

1. We avoid the service-type contention between HTTP and Kafka interfaces.
2. We get the additional information on the UI as to which service is consuming.

The implications of implementing 2 or 3 will mean that once this is resolved there will inevitably be some cleanup too.

Note that RabbitMQ consumer uses service name already.

[ffalexandererben]

We ended up using the second option as it best fits our needs. To do this, you do not have to implement anything and just use dd.service.mapping as stated above. Same goes for option three by the way. You can just try either option without much work involved.

[matt-canty-dragon]

Do your services also have HTTP endpoints? I think this is the clincher for 3 over 2 specifically.

[ffalexandererben]

They do, but we just use the span type selector in the UI. We got used to it, but it's far from ideal. But we have more than 120 services in production and almost every one of them uses Kafka. Having separate services for the Kafka integration would double the number of services to 240, which would be even more unwieldy.

[bobthearsonist]

just stumbled upon this still-open request. the dotnet tracers appear to already prpend with the service name ie servicename-kafka, is the java tracer ever going to be updated to do this by default?

[mcculls]

If you set the following system property on both the producing and consuming sides:

-Ddd.kafka.legacy.tracing.enabled=false

then the consume span will now have the local servicename and a new span will be added between the producer and consumer to represent the "time-in-queue" for the message. This new "time-in-queue" span will have a service name of "kafka", and sits between the producing service name and consuming service name in the service map.

[samsaggace]

Thanks so much for this awesome feature, it's super powerful !

Is there a documentation that lists of all the other "non-legacy" features that can be activated to improve the experience ? I'm so eager to test them all !!! 😄

Thanks again

[bm1549]

@samsaggace we don't have a comprehensive document, but they are searchable in the repository itself. Please be careful about using these as they may introduce breaking changes

https://github.com/search?q=repo%3ADataDog%2Fdd-trace-java+isLegacyTracingEnabled%28true&type=code