camel-quarkus-opentelemetry2 doesn't connect spans from Vert.x HTTP with Camel spans

[llowinge]

Bug description

Having route as follows:

      from("platform-http:/foo")
                .setBody().constant("Foo");

and invoking it with curl http://localhost:8080/foo

gives different user experience as for camel-quarkus-opentelemetry2 it results in two separated/disconnected traces (2 and 1 spans)

but with camel-quarkus-opentelemetry, those traces are glued together in one trace (2 spans):

I think there is a bug in camel-quarkus-opentelemetry2, because the one span is disconnected (probably not set correct parentId).

[jamesnetherton]

@squakez unlike https://github.com/apache/camel-quarkus/issues/7801, I think this case is more of an issue.

With the original opentelemetry component you get the expected span hierarchy. With opentelemetry2, you get a dangling trace from the Vert.x HTTP server.

The problem is maybe here:

https://github.com/apache/camel/blob/0c97f9eb145042ccd07626ea3c6c3e2db22d7057/components/camel-opentelemetry2/src/main/java/org/apache/camel/opentelemetry2/OpenTelemetryTracer.java#L112-L115

The extractor.get("traceparent") part is trying to find a Camel header named traceparent. But it doesn't exist, so we end up passing a null context into contextPropagators.getTextMapPropagator().extract(), which is why we end up with disconnected spans.

If it were to pass Context.current() it would probably work. But I'm no otel expert...

[jamesnetherton]

Another similar scenario that previously worked with camel-opentelemetry is JAX-RS services:

This:

@Path("/trace")
@GET
@Produces(MediaType.TEXT_PLAIN)
public String traceRoute() {
    return template.requestBodyAndHeaders("direct:start", null, headers, String.class);
}

Would create a span hierarchy like:

|--- JAX-RS service
|------- direct:start

With camel-opentelemetry2, the spans are disconnected. To make it work you have to hack the traceparent header:

@Path("/trace")
@GET
@Produces(MediaType.TEXT_PLAIN)
public String traceRoute() {
    Map<String, Object> headers = new HashMap<>();
    headers.put("traceparent", Span.current().getSpanContext().getSpanId());
    return template.requestBodyAndHeaders("direct:start", null, headers, String.class);
}

[squakez]

I think we have some test covering those use case. I am giving a look at it. From the snapshot I can see that the new telemetry is producing 2 separate traces. One containing 2 spans (likely the one produced by Camel which we're interested to) and the other 1 single span which I am not sure where it's coming from. I'll keep you posted.

[squakez]

I made some test and I suspect this problem is the fact vertx is not honoring the traceparent (see https://github.com/quarkusio/quarkus/issues/50466). Ideally we need this to get fixed by the platform and see later how it performs on Camel side.

[jamesnetherton]

Until there's a fix for this, I was thinking of switching camel-quarkus-observability-services back to using camel-quarkus-opentelemetry. At least in the 3.27.x LTS stream. Since we don't know when a fix may land, it could be desirable to do it on the main branch also.

[squakez]

The problem would be that the other runtimes have the otel2 component. We'd have some asymmetry between runtimes that I personally don't like.

[jamesnetherton]

I think we've inadvertently polluted https://github.com/apache/camel-quarkus/issues/7801 with comments that relate specifically to this issue. #7801 should perhaps be closed. The real issue to solve is the disconnected spans.

Given the feedback here, I hacked up a POC of having Vert.x Web propagate traces in a way that Camel can work with. It seems to work, I see the Vert.x span connected as the parent of the Platform HTTP span.

If it's an acceptable approach, we could maybe add it here initially and then try to get it pushed to Quarkus.

[squakez]

Hey James, it's exactly the same change I'm doing on the agent on Vertx side as well. And it should be the proposed solution for Vertx as a library, ie, to propagate downstream traceparent in the way expected by the specification.

[squakez]

FYI, I've just submitted an issue and a fix PR to the agent. This is not affecting directly the Quarkus runtime, but we can evaluate the acceptance to understand if the approach proposed is good or not.