Sometimes the replicator stops subscribing new events

[hrbustor]

What is the bug? When the hub encounters an exception or RPC cannot be connected, the replicator may experience connection failure leading to downtime. But the docker container is still running, this causes the synchronization data to stop.

The error log is as follows: [11:48:56.300] INFO (7): Hub event stream processing halted unexpectedly: 1 CANCELLED: Call cancelled [11:48:56.300] INFO (7): HubSubscriber all-events restarting hub event stream in 5 seconds... [11:48:56.300] INFO (7): HubSubscriber all-events stream closed [11:49:01.302] INFO (7): Starting HubSubscriber all-events [11:49:01.803] ERROR (7): HubSubscriber all-events failed to connect to hub: Error: Failed to connect before the deadline [11:49:01.803] ERROR (7): Unhandled promise rejection: Error: Failed to connect before the deadline

Suspected to be caused by the following code：

How can it be reproduced? (optional) Stop the hubble service.

Additional context (optional) There may be two things that can help solve the problem: One is whether it is possible to stop the Docker program if the program is abnormal, so that the Docker restart strategy can be relied on to ensure that the replicator runs again. Additionally, adding a retry mechanism when connecting to a hub exception.