Software Design: Add reconnection possibility

[haerdib]

Currently, only the tungstenite client offers something in that direction. Jsonrpsee sadly does offer any easy reconnection yet. See https://github.com/paritytech/jsonrpsee/issues/678 or https://github.com/paritytech/jsonrpsee/issues/949

Tricky: How to avoid endless loops? A node might be very well down for good. In that case, it would be better to shut down, rather than staying alive, waiting for a dead node. And as parity wrote in https://github.com/paritytech/jsonrpsee/issues/236, it should offer all the options (e.g. how long until a dead node should really be considered dead, how many retries, ..)

Also check out : https://github.com/paritytech/subxt/issues/551

[haerdib]

I've looked into that some - and I have to agree with #551 about the problem with the subscription.

Scenario:

Let's say the client has subscribed for runtime upgrades (https://github.com/scs/substrate-api-client/issues/348), obviously in its own loop which the client created. Now the node dies. As a result, the loop will break as well, as the subscription will return None. Or at least that's how I'd imagine it, because otherwise the client will have no knowledge of something going wrong.

If we simply reconnect the api, the loop will not be restarted, as it has been created on the client side. Resulting in a new api, that does not upgrade. No good.

Workarounds/ thoughts:

• Do not terminate the loop for some time due to a disconnected subscription. Instead, try to resubscribe for a given amount of time or tries. Should probably be choosable from the client side. Drawback: This only works in case of the subscription is managed in the crate. In case the client as subscribed to something manually, this does not help.
• When a subscription returns None do not directly return this to the client subscription, but instead try to resubscribe for a given amount of tries / time. (Do NOT try to reconnect in case the None is returned to an explicit unsubscribe(..)) Not sure how exactly this should be done. Maybe an additional subscription wrapper is needed and/or a lock caching the active subscriptions.

[xlc]

I am working on this that implements retry for ws clients https://github.com/AcalaNetwork/subway/blob/master/src/client/mod.rs