router's failure state :: rabbitmq: connection_closed_abruptly

[yongshengma]

Hello,

A storage router's state becomes FAILURE in red color after it was rebooted, but everything still looks working well.

(1) ovs.storagerouter.ping

In sys log, the task ovs.storagerouter.ping never shows up on node1 since last reboot, but it is not absent on the other two nodes (master) every minute.

(2) RabbitMQ

RabbitMQ reports error once a minute in [email protected]

closing AMQP connection <0.1095.106> (192.168.0.33:53762 -> 192.168.0.33:5672):
connection_closed_abruptly

=WARNING REPORT==== 22-Jul-2019::16:25:05 ===
closing AMQP connection <0.1170.106> (192.168.0.35:42262 -> 192.168.0.33:5672):
connection_closed_abruptly

=WARNING REPORT==== 22-Jul-2019::16:25:07 ===
closing AMQP connection <0.1260.106> (192.168.0.34:53716 -> 192.168.0.33:5672):
connection_closed_abruptly

All 3 nodes including itself can't make connection to 192.168.0.33.

(3) Celery

Checked celery:

celery inspect ping -b amqp://ovs:[email protected]// --timeout=5
-> celery@node1: OK
        pong
-> celery@node3: OK
        pong
-> celery@node2: OK
        pong

Any idea about how to fix it? The failure state has happened before and there's no way to cure it all the time. I've read the issue #558 and #565, but not sure if they have the same cause.

Best regards, Yongsheng