Improve error handling in RPC client

[skunert]

This is a reminder issue to take another look at error handling once the RPC collator functionality is more mature. Todo:

• If channels run full unexpectedly, kill the worker task
• If a relay chain RPC node dies, connect to a potential backup node

Umbrella Issue: #989 PR where this came up: #1462

[ehsanhajian]

hello here. During node sync, I got this error.

Apr 07 08:58:00 .srv.q-01842.com astar-collator[105967]: 2023-04-07 08:58:00 [Parachain] Event distribution channel has reached its limit. This can lead to missed notifications. error=TrySendError { kind: Full }

it's a new node and been synced from 0.

Node syntax:

ExecStart=/usr/local/bin/astar-collator \
  --chain astar \
  --rpc-cors all \
  --name {nodename} \
  --base-path /var/lib/astar \
  --state-pruning archive \
  --blocks-pruning archive \
  --prometheus-external \
  --rpc-external \
  --ws-external \
  --rpc-methods Safe \
  --rpc-max-request-size 1 \
  --rpc-max-response-size 1 \
  --ws-max-connections 5000 \
  --max-runtime-instances 64 \
  --telemetry-url 'wss://telemetry.polkadot.io/submit/ 0' \
  --enable-evm-rpc  --relay-chain-rpc-url "ws://{ip}"

[skunert]

Thanks for the report, I will investigate this!