smallrye-reactive-messaging icon indicating copy to clipboard operation
smallrye-reactive-messaging copied to clipboard

Attach exceptions to spans in error handlers for better Observability

Open michalcukierman opened this issue 5 months ago • 0 comments

It appears that attaching exceptions to spans in error handlers could be implemented relatively easily, improving the observability of failures.

The approach taken in the Quarkus handler (AttachExceptionHandler) uses the following line to record exceptions:

LocalRootSpan.current().recordException(throwable);

We could adopt a similar approach in PulsarIncomingChannel to enhance traceability. For example, modifying the method here: smallrye-reactive-messaging-pulsar/PulsarIncomingChannel.java#L199:

public synchronized void reportFailure(Throwable failure, boolean fatal) {
    // Don't keep all the failures, only keep them for reporting.
    if (failures.size() == 10) {
        failures.remove(0);
    }
    failures.add(failure);
    
    // Attach the exception to the current span for observability
    LocalRootSpan.current().recordException(failure);

    if (fatal) {
        close();
    }
}

Adding this simple modification would provide a more comprehensive view of exceptions for observability tools like OpenTelemetry, Jaeger, or similar, enabling better tracking and debugging of issues across the system.

I can create the PR, if you think that the investigation is correct.

michalcukierman avatar Oct 03 '24 13:10 michalcukierman