Bad sync aggregates in small devnets
Running a local devnet with 4 lodestar nodes (64 validators each) results in bad sync aggregates
The interesting part is that is exactly 63.87% each block.
Looking at the validator clients I confirmed that we publish 64 messages each slot, so there is either an issue with our sync messages aggregation or sync aggregate packing in the block
Might be similar issue as Prysm noticed https://github.com/OffchainLabs/prysm/pull/15608
Some preliminary investigation was done but warrants a more thorough review of the code, which hasn't been touched likely since Altair.
This only affects smaller devnets and doesn't show up in larger testnets nor mainnet. It may have something to do with the validator having multiple validators in the same committee. The likelihood of this happening is very low on larger testnets and mainnet.
It will also fix itself when adding another node to the devnet, so there is potentially an issue in gossiping.