Mismatched slf4j MDC context in Reactor threads

[jakubmalek]

Hi, in our Kafka Connect cluster we deploy several instances of Event Hub connectors. Recently, we've noticed that connector name printed in the error logs is mixed up The problem was observed when one of the connector threw exception in Reactor thread.

Here is an example of log entry (with obfuscated details) :

ERROR [reactor-executor-6265] [WRONG-CONNECTOR-NAME|task-0] [Operators] Operator called default onErrorDropped reactor.core.Exceptions$ErrorCallbackNotImplemented: com.azure.core.amqp.exception.AmqpException: errorContext[NAMESPACE: XXXXXXXXXX.servicebus.windows.net. ERROR CONTEXT: N/A] Caused by: com.azure.core.amqp.exception.AmqpException: errorContext[NAMESPACE: XXXXXXXXXX.servicebus.windows.net. ERROR CONTEXT: N/A] at com.azure.core.amqp.implementation.ExceptionUtil.toException(ExceptionUtil.java:85) at com.azure.core.amqp.implementation.handler.ConnectionHandler.notifyErrorContext(ConnectionHandler.java:351) at com.azure.core.amqp.implementation.handler.ConnectionHandler.onTransportError(ConnectionHandler.java:253) at org.apache.qpid.proton.engine.BaseHandler.handle(BaseHandler.java:191) at org.apache.qpid.proton.engine.impl.EventImpl.dispatch(EventImpl.java:108) at org.apache.qpid.proton.reactor.impl.ReactorImpl.dispatch(ReactorImpl.java:324) at org.apache.qpid.proton.reactor.impl.ReactorImpl.process(ReactorImpl.java:291) at com.azure.core.amqp.implementation.ReactorExecutor.run(ReactorExecutor.java:91) at reactor.core.scheduler.SchedulerTask.call(SchedulerTask.java:68) at reactor.core.scheduler.SchedulerTask.call(SchedulerTask.java:28) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829)

The MDC context for the logger was set for connector that didn't use Event Hub namespace printed in stack trace. This was later confirmed, when error was logged in connector task thread (this time with right connector name).

As far as I know, the reactor can be explicitly configured with a aontext: https://simonbasle.github.io/2018/02/contextual-logging-with-reactor-context-and-mdc/

But I don't know if it's possible to configure it for Apache Camel components.

The problem was observed in camel-kafka-connect v3.18.2.