lodestar icon indicating copy to clipboard operation
lodestar copied to clipboard

Apply peer action when there are gossip validation errors

Open twoeths opened this issue 3 years ago • 6 comments

Describe the bug

Hetzner-s2 shows a lot of rejected ContributionAndProof gossip messages but other nodes show nothing Screen Shot 2022-02-18 at 14 28 24

The lodestar_gossip_validation_accept_total metric is the same for all nodes.

ls 2022-02-16 | grep "contribution" |  wc -l
174
 ls 2022-02-15 | grep "contribution" |  wc -l
125

Expected behavior

  • It should be consistent across nodes

twoeths avatar Feb 18 '22 07:02 twoeths

all errors are the same: SYNC_COMMITTEE_ERROR_VALIDATOR_NOT_IN_SYNC_COMMITTEE

Feb-16 00:14:20.583 [NETWORK]         ^[[34mdebug^[[39m: The invalid gossip contribution and proof was written to .../contributionAndProof_0x6bb15ca4bbe8160f97304dfe42521a0f3df6cd518ccefe091f3a6f9f929d784a_1644966860583.ssz

twoeths avatar Feb 18 '22 07:02 twoeths

all of the errors are correct as the validator is really not in the sync committee at the specified slot.

This works as designed since some nodes could catch invalid gossip messages and not spread them to the network, other nodes may not see them.

This brings another concern: if a node keeps receiving invalid messages from another node, right now it only decreases gossip score of peer, should we also decrease score at lodestar side, and in the end disconnect the peer?

twoeths avatar Feb 18 '22 07:02 twoeths

if a node keeps receiving invalid messages from another node, right now it only decreases gossip score of peer, should we also decrease score at lodestar side, and in the end disconnect the peer?

Maybe the logic to decide what peers to disconnect should consider gossip score too

dapplion avatar Feb 18 '22 08:02 dapplion

if a node keeps receiving invalid messages from another node, right now it only decreases gossip score of peer, should we also decrease score at lodestar side, and in the end disconnect the peer?

Maybe the logic to decide what peers to disconnect should consider gossip score too

What lighthouse does is to also penalize peer at node side depending on the gossip error https://github.com/sigp/lighthouse/blob/c3a793fd73a3b11b130b82032904d39c952869e4/beacon_node/network/src/beacon_processor/worker/gossip_methods.rs#L1339

for example if a peer sends invalid signature attestations for 5 times, it'll get banned at node (lodestar) side due to LowToleranceError PeerAction

twoeths avatar Feb 18 '22 08:02 twoeths

@twoeths is this still relevant?

wemeetagain avatar Oct 10 '24 19:10 wemeetagain

yes it is, will work on this

twoeths avatar Oct 11 '24 01:10 twoeths