Periodic "msg_dropped" between cluster nodes on Kubernetes

[jhhurwitz]

Installation version: 1.9.2-1-alpine Metadata plugin: plumtree

Periodically (every ~1-2 days) one of the nodes in the vernemq cluster will lose connection to the other resulting in message dropping and an elevated rate of the metric cluster_bytes_dropped. After a few seconds the system will return to normal.

Logs:

21:44:56.771 [warning] can't send 366 bytes to '[email protected]' due to timeout, reconnect!
....
21:44:59.039 [warning] can't publish to remote node '[email protected]' due to 'msg_dropped'
21:44:59.039 [warning] can't publish to remote node '[email protected]' due to 'msg_dropped'
21:44:59.039 [warning] can't publish to remote node '[email protected]' due to 'msg_dropped'
21:44:59.039 [warning] can't publish to remote node '[email protected]' due to 'msg_dropped'
21:44:59.039 [warning] can't publish to remote node '[email protected]' due to 'msg_dropped'
21:44:59.039 [warning] can't publish to remote node '[email protected]' due to 'msg_dropped'
21:44:59.040 [warning] can't publish to remote node '[email protected]' due to 'msg_dropped'
21:44:59.040 [warning] can't publish to remote node '[email protected]' due to 'msg_dropped'
21:44:59.042 [warning] can't publish to remote node '[email protected]' due to 'msg_dropped'
21:44:59.042 [warning] can't publish to remote node '[email protected]' due to 'msg_dropped'
21:44:59.042 [warning] can't publish to remote node '[email protected]' due to 'msg_dropped'
21:44:59.042 [warning] can't publish to remote node '[email protected]' due to 'msg_dropped'
21:44:59.043 [warning] can't publish to remote node '[email protected]' due to 'msg_dropped'
21:44:59.043 [warning] can't publish to remote node '[email protected]' due to 'msg_dropped'
21:44:59.044 [warning] can't publish to remote node '[email protected]' due to 'msg_dropped'
21:44:59.044 [warning] can't publish to remote node '[email protected]' due to 'msg_dropped'
21:44:59.044 [info] successfully connected to cluster node '[email protected]'

Is this a known issue with VerneMQ on Kubernetes?

Thanks