Eventstream API fallback support

[nflaig]

Problem description

The current implementation of eventstream does not support fallback beacon nodes.

Only the URL of the primary node is used https://github.com/ChainSafe/lodestar/blob/959a8af1b757c46a0e57450788fb72773a372608/packages/api/src/beacon/client/index.ts#L31

This means if primary node is down, the validator client will stop receiving head events which are currently used to detect reorgs and to prevent querying beacon node for block root when producing sync committe messages https://github.com/ChainSafe/lodestar/blob/959a8af1b757c46a0e57450788fb72773a372608/packages/validator/src/services/attestationDuties.ts#L62 https://github.com/ChainSafe/lodestar/blob/959a8af1b757c46a0e57450788fb72773a372608/packages/validator/src/services/syncCommittee.ts#L126 Not detecting dependent root changes might lead to missed attestation duties.

Solution description

Eventstream API should support fallback beacon nodes

• if primary node does not have any issue only open one eventstream
• if primary node has an error that is recoverable e.g. not status 400 or 500 then open eventstream with secondary node
• if second has an error then open eventstream with third node, and so on
• the eventstream to primary node should never be closed, if at some point it does not error then swap back to only using primary node and close all fallback eventstream connections

Additional context

No response

[enyinnaya1234]

Hi, @nflaig , i'd love to take this up, may i?

[nflaig]

I assigned you to the issue @enyinnaya1234