Application Insights not tracing Kafka Dependencies in a Java JDK 11 local App

[Justrebl]

Expected behavior

Having a Java Kafka Consumer App instrumented with the application-insights agent running locally, i am expecting to see the tracked dependencies (application insights logs and application map not showing the consumer dependency) while the Java consumer retrieves events from a Kafka enabled Event Hub.

Actual behavior

Currently, all i can see being traced with the java app insight agent are the requests logs from my java client app to the Kafka (event-hub) source, but not the dependancy logs (thus application map is not showing the link between my java client/consumer app

To Reproduce

Steps to reproduce the behavior:

• Create a kafka event producer to aliment a Event Hub Kafka enabled namespace (i created a simple Azure Function with an EventHub output binding to create new event in the same topic on a per second basis, that can be seen in the application map representation).
• Clone the following Microsoft Kafka consumer example here : https://github.com/Azure/azure-event-hubs-for-kafka/tree/master/quickstart/java and configure the resources/consumer.config as detailed in the consumer/readme.md to point at the kafka endpoint and topic of a Premium Event Hub.
• Instrument the Java consumer app with the application-insight java agent 3.4.1
• Run the Java application to consume the Event Hub/Kafka endpoint
• Check the application insights logs (table : requests & dependencies)
• Run the following queries in Log Analytics :
  • requests | where cloud_RoleName contains "consumer" => 1 requests Line per event consumed showing in the KQL query results
  • dependencies | where cloud_RoleName contains "consumer" => 0 line result

Sample Application

This Microsoft provided Java Kafka Consumer reproduces the behavior : https://github.com/Azure/azure-event-hubs-for-kafka/tree/master/quickstart/java

.mvn/jvm.config :

-javaagent:./applicationinsights-agent-3.4.1.jar

applicationinsights.json :

{ "connectionString": "InstrumentationKey=0000;IngestionEndpoint=https://00000.in.applicationinsights.azure.com/;LiveEndpoint=https://0000.livediagnostics.monitor.azure.com/", "role":{ "name":"Java Kafka Consumer", "instance":"OnPremise" }, "instrumentation": { "logging": { "level": "TRACE" } }, "sampling":{ "percentage":100 } }

System information

Please provide the following information:

• SDK Version:
  • jdk-11.0.16
  • App insight java agent 3.4.1
• OS type and version: Windows 11 (same behavior on Linux WSL2 - Ubuntu 20.04 LTS)
• Application Server type and version (if applicable):
• Using spring-boot? No

Screenshots

Please let me know if i can extend on above explanations to help you reproduce the behavior.

[trask]

hi @Justrebl!

i created a simple Azure Function with an EventHub output binding to create new event in the same topic on a per second basis, that can be seen in the application map representation

can you try with a local kafka app that pushes to the queue to see if that works? I suspect that this is related to Azure Functions output bindings or not propagating W3C traceparent header on the messages, but I'm not positive. thx!

[Justrebl]

Hello @trask, Thanks for reaching out and sorry for the late response here !

New test today (code here https://github.com/Justrebl/azure-event-hubs-for-kafka/tree/master/quickstart/java) with a local kafka java app pushing to the same event hub kafka enabled consumed by the local app.

It actually brings something interesting in the intelligent view (Application map preview) :

When i leave the preview i can see direct communication between the producer and the consumer (without seeing the event hub in the middle, and 0 call on the producer : is this intended?)

Checking the logs, here is what i find for requests (only traces for the consumer are appearing) :

And Dependencies (only the producer events are traced in the dependencies):

Checking the logs i have for the Azure Function discussed earlier, i can see entries in the Dependencies AND the requests (Azure function being a producer working directly with the Az Function Event Hub output binding) : Requests :

Dependencies:

So having a kafka producer AND a kafka consumer seem to be tracked as expected in the application map (even if we're losing the actual kafka/event hub queue in the picture : is this expected?), but how about tracking other producers' dependecies?

Thanks again for your help on this topic!

[trask]

hey @Justrebl, I was able to run your demo,

In "preview mode" I got this picture:

Give it another try, sometimes it takes several minutes for the backend indexer to draw the right app map. Also, it's possible that the app map team has rolled out a fix since you tried.

The "0 calls" is correct b/c it's the number of requests to that node, and it has no "requests", since it's just generating messages from the command line.