ApplicationInsights-Java
ApplicationInsights-Java copied to clipboard
microsoft
Custom attributes not propagated & no span for reactor rabbit
Expected behavior
Custom dimensions are propagated to application insights (and children spans) and/or span is created for reactor rabbit messages
Actual behavior
Span is not created for reactor rabbit subscription and custom dimensions are not propagated
Code example
receiver
.consumeManualAck("${queueConfiguration.name}.${RabbitConstants.QUEUE}", ConsumeOptions().qos(10))
.delaySubscription(Duration.ofMillis(subscriptionDelay))
.subscribe { delivery ->
mono(context = Dispatchers.IO + CustomRequestContext(/*some custom context*/) {
val b3 = delivery.properties.headers["b3"]?.toString()?.split('-') ?: emptyList()
val currentSpan = createSpan(b3).startSpan().apply { makeCurrent() }
//Span wasn't created so I had to do it myself
Span.current().setAttribute("MyCustomDimension", "myCustomDimensionValue")
try {
logger.info { "Received rabbitMQ message" }
doSomeWork(delivery.body)
delivery.ack()
} catch (exception: Exception) {
delivery.nack(false)
} finally {
currentSpan.end()
}
}.subscribe()
}
private fun createSpan(b3: List<String>): SpanBuilder {
val tracer = OpenTelemetry.noop().getTracer("")
return if (b3.size == 3) {
val traceId = if (b3[0].length < 32) {
"0000000000000000${b3[0]}"
} else b3[0]
val spanId = b3[1]
val sampled = if ("1" == b3[2]) {
TraceFlags.getSampled()
} else {
TraceFlags.getDefault()
}
val parentSpanContext = SpanContext.create(traceId, spanId, sampled, TraceState.getDefault())
val parentSpan = Span.wrap(parentSpanContext)
val parentContext = Context.root().with(parentSpan)
tracer.spanBuilder("my span with parent").apply { setParent(parentContext) }
} else {
val parentSpanContext = SpanContext.create(Random.nextBytes(ByteArray(16)).toHex(), Random.nextBytes(ByteArray(8)).toHex(), TraceFlags.getSampled(), TraceState.getDefault())
val parentSpan = Span.wrap(parentSpanContext)
val parentContext = Context.root().with(parentSpan)
tracer.spanBuilder("my span with generated parent").apply { setParent(parentContext) }
}
}
private fun ByteArray.toHex(): String = joinToString(separator = "") { eachByte -> "%02x".format(eachByte) }
private fun doSomeWork(body: byte[]){
val tracer = OpenTelemetry.noop().getTracer("")
val parentSpan = Span.current()
//I was loosing spans in middle of processing of request, so I create new one here
val newSpan = tracer.spanBuilder("my span with parent").apply { setParent(Context.current().with(parentSpan)) }.startSpan().apply { makeCurrent() }
val response = try {
someAsynchronousClientCalls(status, notification)
} finally {
newSpan.end()
}
//some processing here (no client calls, just orchestration)
//another Span as I lost if sometimes here
val messageSpan = tracer.spanBuilder("my span with parent").apply { setParent(Context.current().with(parentSpan)) }.startSpan().apply { makeCurrent() }
try {
sendMessageWithAzureServiceBus()
} finally {
messageSpan.end()
}
}
System information
Please provide the following information:
- SDK Version:
- applicationinsights-agent-3.3.0-BETA
- opentelemetry-api 1.12
- applicationinsights-core 2.6.4
- OS type and version: it behaves the same on windows/linux under Java 15 (compiled by 15 or 17)
- Application Server type and version (if applicable):
- Using spring-boot
- spring-boot-starter-parent 2.7.0
- spring cloud 2021.0.3
- Kotlin
- kotlin 1.6.21
- kotlin-coroutines 1.6.2
- Additional relevant libraries :
- io.projectreactor.rabbitmq reactor-rabbitmq 1.5.4
Screenshots
Telementry in application insights looks like that:
So it isn't great too
Please use RabbitTest (it needs docker to run) from this example together with agent from resources (just adjust connection string in applicationinsights.json) https://github.com/xsmrcek/minimal-example This issue may be caused just by me implementing it totally wrong. But cna you please have look anyway? Thank you!
@xsmrcek Thank you for the repo reproducing the problem! Could you please replace val tracer = OpenTelemetry.noop().getTracer("") by val tracer = GlobalOpenTelemetry.getTracerProvider().tracerBuilder("my-tracer").build()? With that modification I can see your custom spans and attributes. OpenTelemetry.noop() supplies an OpenTelemetry object that does not not propagate spans. Is everything now working the way you want it to?
This issue has been automatically marked as stale because it has been marked as requiring author feedback but has not had any activity for 7 days. It will be closed if no further activity occurs within 7 days of this comment.
Hello @jeanbisutti , sorry for late response I was on 2 weeks vacation :) Which opentelemetry-api version I should be using? I am using 1.13. Should I use latest and it will be OK with latest agent application insights agent? Thank you!
Hello @xsmrcek, I see a test in our code checking that Application Insight is working from opentelemetry-api 1.0.0. However I would recommend to use the last opentelemetry-api version, that is to say 1.16.0. In case you did not see my previous comment, your problem came from the tracer.
This issue has been automatically marked as stale because it has been marked as requiring author feedback but has not had any activity for 7 days. It will be closed if no further activity occurs within 7 days of this comment.
Hello @jeanbisutti ,
thank you for your help. Not is working almost as expected :D
I have updated minimal example project from above with new code.
I had to turn off rabbbitMQ instrumentation as it is not doing what I want.
Now I send request telemetry to application insights manually. But I am missing Custom properties in request telemetry. Can you please help me with this too?
Hi @xsmrcek,
To see the custom key, please select a thing different from "MY ROLE NAME".
For example, see this screenshot:
This issue has been automatically marked as stale because it has been marked as requiring author feedback but has not had any activity for 7 days. It will be closed if no further activity occurs within 7 days of this comment.
Hello @jeanbisutti thank you for reply! Unfortunately I need custom preperties visioble on request telementry :( I am setting alerts based on application insights logs and request telementry without custom properties is useless :(
hi @xsmrcek, you can use applicationinsights-web 2.x artifact and use
RequestTelemetry requestTelemetry =
ThreadContext.getRequestTelemetryContext().getHttpRequestTelemetry();
requestTelemetry.getProperties().put("myattr1", "myvalue1");
this will add the custom properties onto the request
[btw, we are releasing a new version of applicationinsights-web shortly at version 3.4.0-BETA so that you will no longer need to pull in the (whole) 2.x SDK for this use case]
Hello @trask ! Thank you, it worked! But when I try to use new dependency I don;t see request at all :( I adjusted my minimal example again :)
hey @xsmrcek, I ran RabbitTest, using 3.4.0-BETA javaagent and was able to see:
can you confirm exact repro steps from your end?
This issue has been automatically marked as stale because it has been marked as requiring author feedback but has not had any activity for 7 days. It will be closed if no further activity occurs within 7 days of this comment.
hey @xsmrcek, I ran
RabbitTest, using 3.4.0-BETA javaagent and was able to see:
can you confirm exact repro steps from your end?
Thank you! My bad. I used agent 3.3.0-BETA instead. Thank you guys for help!