Lodestar produces blocks with empty sync aggregate

[nflaig]

This behavior has been observed on the Mekong testnet.

Eg. https://dora.mekong.ethpandaops.io/slot/274010 contains an empty sync aggregate (0% participation)

There are no obvious error logs that could explain this, the only log (from lodestar-geth-3) I found was from our light client server which was complaining about it

Dec-13 17:22:00.136[chain]           debug: sync committee below required MIN_SYNC_COMMITTEE_PARTICIPANTS syncPeriod=33, attestedPeriod=33, syncAggregateParticipation=0

This seems to be isolated to electra / mekong / devnet-4 where it can be observed quite frequently, also has been reported by Luca on discord.

[nflaig]

This also happened on holesky, see https://holesky.beaconcha.in/slot/3255000

[prelude444]

same here. also with attestation only 1.

and it's happening on mainnet. Also, ever since the upgrade to the latest v1.24.0, my luck for finding block went straight down to less than 20%.

[nflaig]

same here. also with attestation only 1.

could please share which slot this was, the empty sync aggregate issue is likely due to the same issue as on holesky which was related to building a block on a really late parent block (> 4 seconds into the slot), this will be resolved once we enable proposer boost reorg by default as we would not build on top of such late blocks. I am a bit surprised that it only includes a single attestation though, this should be unrelated.

Also, ever since the upgrade to the latest v1.24.0, my luck for finding block went straight down to less than 20%.

I am guessing by finding a block you mean be selected to propose a block? This is not something related to the client but the ethereum protocol and is out of your control

[prelude444]

slot 10798404

It's my first time ever propose a block that has only 1 attestation. Yes it seems strange. My validator is not slow, 99% efficiency avg.

The luck in proposing a block also went to below 20%. Normally its around 60%-130%ish. never went this low. Maybe I was just extremely unlucky lately. If it's not due to the outlier in luck, i can only attribute the cause to maybe 2 things, 1, I switched my router to a new one, or 2. I upgraded to v1.24.0. Everything else was the same as before.

[nflaig]

slot 10798404

I was checking on my node when I received the block from previous slot and it was around ~3 seconds into the slot when it was fully validated and imported as head, late parent block does not seem to be the issue here. We are keeping an eye on this issue but so far it's not something that happens consistently, especially on mainnet. Other than late parent block I am not sure what else could be causing this, maybe someone from team has other ideas

The luck in proposing a block also went to below 20%. Normally its around 60%-130%ish. never went this low. Maybe I was just extremely unlucky lately. If it's not due to the outlier in luck, i can only attribute the cause to maybe 2 things, 1, I switched my router to a new one, or 2. I upgraded to v1.24.0. Everything else was the same as before.

This is completely out of your control and random, you could be offline and be selected as proposer as long as your validator is active on the beacon chain, you can read up on this here https://ethereum.org/en/developers/docs/consensus-mechanisms/pos/block-proposal/#random-selection. Something to keep in mind is that the amount of validators on mainnet increases over time which reduces the probability for you to be selected as proposer.

[philknows]

Noting from Feb 18 standup that we still see this in pectra-devnet-6 and requires further investigation.

[nflaig]

I don't see this issue anymore on holesky, based on https://light-holesky.beaconcha.in/slots/filtered?f&f.missing=1&f.orphaned=1&f.pname=lodestar all our blocks look pretty good.

We should only observe empty sync aggregate for blocks that were build on a parent block that was seen late by network > 4 seconds but once we have proposer boost reorg enabled by default it should not happen anymore as we will reorg such late blocks.

My guess why we saw this on devnets so far was related to the attestation bug which caused us to having really bad subnet peers.

[philknows]

Let's look at how this does with proposer boost reorg enabled before closing.

[nflaig]

Let's look at how this does with proposer boost reorg enabled before closing.

It's expected that we have very poor or even empty sync aggregate if we don't reorg late blocks (that arrive >4 seconds) as the network does not vote for this parent block and we can only include sync committee mesages that match the parent root of the block we build on top of.

The actual issue noted here is resolved, closing.

[nflaig]

Reopening, seems like there is still an edge case where this happens even if parent block was not late as reported here

[twoeths]

some more logs for this

2025-06-05 23:58:47.530 | Jun-05 16:58:47.530[chain]            warn: SyncContributionAndProofPool.getAggregate: no contributions for root slot=11859891, root=0x6f7f9e41d60c4bd4d763259672fbcf0d2e0b38925c4342354b428afaec30a216, availableRoots=0x437f572e8952b23d875257813edbadb4bff6caed3aa8eb589ef0e52bd127da13,0x6f7f9e41d60c4bd4d763259672fbcf0d2e0b38925c4342354b428afaec30a216 |  
-- | -- | --
  |   | 2025-06-05 23:02:35.287 | Jun-05 16:02:35.287[chain]            warn: SyncContributionAndProofPool.getAggregate: no contributions for root slot=11859610, root=0xd3fe53ee82c6c7f8ebd04846c4605b385b1d786870f2464f605227563c20eb57, availableRoots=0x8d24d201e4a0c0b0ac0c01adab6251994c46763ba5b33fb31963adb23ac0c81f,0xd3fe53ee82c6c7f8ebd04846c4605b385b1d786870f2464f605227563c20eb57 |  
  |   | 2025-06-05 21:26:35.320 | Jun-05 14:26:35.320[chain]            warn: SyncContributionAndProofPool.getAggregate: no contributions for root slot=11859130, root=0x42096a1120d6462fed28fd6a623bb413da26be02c34b79a651305058b4812eaa, availableRoots=0xff4dd9cf47595f56dd6d62f81dabaeb5e2ebb0526a4d2df7f13b13631c167538,0xf3ad633fa630c52d333496b7f4abf0c271bf6a9685aab023801447e95f2781a3,0x42096a1120d6462fed28fd6a623bb413da26be02c34b79a651305058b4812eaa