Error propagating cutom trace-id on pub/sub

[rochabr]

Expected Behavior

Dapr runtime: 1.15.5 Dapr Cli: 1.15.0 Java SDK: 1.13.3

I'm expecting to see the same trace-id for my publisher and subscriber on zipkin when I set a custom trace-id as a metadata when I call PublishEvent. Ie:

Sample code:

// Create order data
Map<String, Object> order = new HashMap<>();
String orderId = UUID.randomUUID().toString();
order.put("orderId", orderId);
order.put("customer", "test-customer");
order.put("amount", 100.00);
order.put("timestamp", System.currentTimeMillis());

// Create metadata to override CloudEvent properties
Map<String, String> metadata = new HashMap<>();
metadata.put("cloudevent.traceid", "your-custom-trace-id");
metadata.put("cloudevent.source", "order-service");
metadata.put("cloudevent.type", "com.example.order");
metadata.put("cloudevent.id", UUID.randomUUID().toString());

// Publish with metadata
return daprClient.publishEvent(
        PUBSUB_NAME,
        TOPIC_NAME,
        order,
        metadata
).then(Mono.fromCallable(() -> {
    String response = "Published order with ID: " + orderId;
    logger.info(response);
    return response;
}));

Actual Behavior

Following the example above, Zipkin shows 2 different entries for the publisher and subscriber.each with its own trace-id:

Steps to Reproduce the Problem

Run the sample here: https://github.com/rochabr/dapr-trace-example

[salaboy]

@rochabr I am not sure if this is a bug.. I think adding a cloudevent trace.id doesn't automatically propagate the trace to the right place.

Look at this example on how the context is propagated: https://github.com/salaboy/pizza/blob/observability/pizza-store/src/main/java/io/diagrid/dapr/PizzaStore.java#L220

The trick is in:

.contextWrite(getReactorContext(context))

By adding this code, we are grabbing the context from the Reactor framework in Spring and propagating the incoming trace to Dapr.

[rochabr]

@salaboy I think I'm missing a few steps here. to pass this context from the publisher to the subscriber it needs to be fetched somehow. I couldn't find in your project how to do it.

I see an import statement to import io.diagrid.dapr.otel.OpenTelemetryConfig.getReactorContext; which cannot be resolved with the dapr-spring-boot-starter dependency. Can you help me understand or point me to the docs that explain how to fetch context, modify the trace and propagate to the subscriber?

[rochabr]

Updated my example with the otel sdk. The only issue I have now is the publisher trace which is duplicated but one of them (the one with the custom id) propagates properly to the subscriber.

[salaboy]

Can you share more details? like the zipkin graph showing the duplicated IDs? can you share the trace graph too?