Azure
Service Bus FailoverProvider is having issues to recover
From time to time, Service Bus Java client cannot recover keeping reporting that connection has been restored while it is not. And it keeps doing it forever. In result, both JMS producers and consumers are effectively dead.
Java library used: com.azure.spring:spring-cloud-azure-starter-servicebus-jms:4.17.0.
Below are symptom log messages, repeating every minute (real namespace is replaced with [namespace]): 2024-11-02T00:06:02.745-04:00 jobexecutor-6b58cff8b8-9ppqn INFO FailoverProvider: async work thread org.apache.qpid.jms.JmsConnection Connection ID:bec7119d-6f30-4e63-a1ac-a6f3838dec2a:1 restored to server: amqps://[namespace].servicebus.windows.net 2024-11-02T00:07:02.945-04:00 jobexecutor-6b58cff8b8-9ppqn INFO FailoverProvider: async work thread org.apache.qpid.jms.JmsConnection Connection ID:bec7119d-6f30-4e63-a1ac-a6f3838dec2a:1 restored to server: amqps://[namespace].servicebus.windows.net 2024-11-02T00:08:03.177-04:00 jobexecutor-6b58cff8b8-9ppqn INFO FailoverProvider: async work thread org.apache.qpid.jms.JmsConnection Connection ID:bec7119d-6f30-4e63-a1ac-a6f3838dec2a:1 restored to server: amqps://[namespace].servicebus.windows.net 2024-11-02T00:09:03.425-04:00 jobexecutor-6b58cff8b8-9ppqn INFO FailoverProvider: async work thread org.apache.qpid.jms.JmsConnection Connection ID:bec7119d-6f30-4e63-a1ac-a6f3838dec2a:1 restored to server: amqps://[namespace].servicebus.windows.net 2024-11-02T00:10:03.662-04:00 jobexecutor-6b58cff8b8-9ppqn INFO FailoverProvider: async work thread org.apache.qpid.jms.JmsConnection Connection ID:bec7119d-6f30-4e63-a1ac-a6f3838dec2a:1 restored to server: amqps://[namespace].servicebus.windows.net 2024-11-02T00:11:03.903-04:00 jobexecutor-6b58cff8b8-9ppqn INFO FailoverProvider: async work thread org.apache.qpid.jms.JmsConnection Connection ID:bec7119d-6f30-4e63-a1ac-a6f3838dec2a:1 restored to server: amqps://[namespace].servicebus.windows.net 2024-11-02T00:12:04.083-04:00 jobexecutor-6b58cff8b8-9ppqn INFO FailoverProvider: async work thread org.apache.qpid.jms.JmsConnection Connection ID:bec7119d-6f30-4e63-a1ac-a6f3838dec2a:1 restored to server: amqps://[namespace].servicebus.windows.net 2024-11-02T00:13:04.342-04:00 jobexecutor-6b58cff8b8-9ppqn INFO FailoverProvider: async work thread org.apache.qpid.jms.JmsConnection Connection ID:bec7119d-6f30-4e63-a1ac-a6f3838dec2a:1 restored to server: amqps://[namespace].servicebus.windows.net 2024-11-02T00:14:04.522-04:00 jobexecutor-6b58cff8b8-9ppqn INFO FailoverProvider: async work thread org.apache.qpid.jms.JmsConnection Connection ID:bec7119d-6f30-4e63-a1ac-a6f3838dec2a:1 restored to server: amqps://[namespace].servicebus.windows.net 2024-11-02T00:15:04.672-04:00 jobexecutor-6b58cff8b8-9ppqn INFO FailoverProvider: async work thread org.apache.qpid.jms.JmsConnection Connection ID:bec7119d-6f30-4e63-a1ac-a6f3838dec2a:1 restored to server: amqps://[namespace].servicebus.windows.net 2024-11-02T00:16:04.858-04:00 jobexecutor-6b58cff8b8-9ppqn INFO FailoverProvider: async work thread org.apache.qpid.jms.JmsConnection Connection ID:bec7119d-6f30-4e63-a1ac-a6f3838dec2a:1 restored to server: amqps://[namespace].servicebus.windows.net 2024-11-02T00:17:05.064-04:00 jobexecutor-6b58cff8b8-9ppqn INFO FailoverProvider: async work thread org.apache.qpid.jms.JmsConnection Connection ID:bec7119d-6f30-4e63-a1ac-a6f3838dec2a:1 restored to server: amqps://[namespace].servicebus.windows.net 2024-11-02T00:18:05.268-04:00 jobexecutor-6b58cff8b8-9ppqn INFO FailoverProvider: async work thread org.apache.qpid.jms.JmsConnection Connection ID:bec7119d-6f30-4e63-a1ac-a6f3838dec2a:1 restored to server: amqps://[namespace].servicebus.windows.net 2024-11-02T00:19:05.39-04:00 jobexecutor-6b58cff8b8-9ppqn INFO FailoverProvider: async work thread org.apache.qpid.jms.JmsConnection Connection ID:bec7119d-6f30-4e63-a1ac-a6f3838dec2a:1 restored to server: amqps://[namespace].servicebus.windows.net ...
We have added an investigation task for this, we will update this thread once we have more details.
