Investigate "No peers available on topic messages"

[lucassaldanha]

Investigate if we have an underlying issue or if things are working as expected.

TODO: add a bit more context

[rolfyone]

I've setup a simple minimal node going through all the different forks, and several examples like this, but basically

i see on one node:

Failed to publish sync committee message(s) for slot 53; No peers for message topics [/eth2/dc4c476b/sync_committee_0/ssz_snappy] found
GossipFailureLogger | Failed to publish attestation(s) for slot 53; No peers for message topics [/eth2/dc4c476b/beacon_attestation_5/ssz_snappy] found

and on its pair:

Received message for topic Topic(topic=/eth2/dc4c476b/sync_committee_0/ssz_snappy)
Received message for topic Topic(topic=/eth2/dc4c476b/sync_committee_0/ssz_snappy)
Received message for topic Topic(topic=/eth2/dc4c476b/sync_committee_0/ssz_snappy)
Received message for topic Topic(topic=/eth2/dc4c476b/beacon_attestation_5/ssz_snappy)

It does look like if i get NoPeersForOutboundMessageException on outbound in jvm-libp2p, that should mean nothing went to the list of items to wait for. But the message is clearly sent, I may have to talk to @StefanBratanov

[rolfyone]

I hacked some more notes in, and got primary:

AttestationGossipManager | Failed to publish attestation 0x2f7714e10dfad636daddef194998d0df4c5946f47987556eaef11866c270e170 for validator 
GossipFailureLogger | Failed to publish attestation(s) for slot 12; No peers for message topics [/eth2/e9195d94/beacon_attestation_4/ssz_snappy] found

SingleAttestationTopicHandler | validate attestation 0x2f7714e10dfad636daddef194998d0df4c5946f47987556eaef11866c270e170

[rolfyone]

I did some analysis between 2 nodes, and around half the attestations we report back as 'no subscribers on subnet' has corresponding logs on the other side for reading in that attestation over gossip, so it's quite odd. There's been some big libp2p changes so i can't really change back to using an old version...