sigp
Implement Electra `devnet-1` changes
Should I retarget this PR at this branch?
- https://github.com/ethDreamer/lighthouse/pull/16
Should I retarget this PR at this branch?
yes please cherry-pick your commits and target electra-devnet-1 now :)
I had a quick look at the failing test and I'm not sure why it would be failing.
The test is trying to check that the implementation of shuffling_is_compatible is correct. In particular, it is failing on FORK_NAME=electra in a case where the function returns false, which should imply that the shuffling from the state is not compatible with the shuffling for the block at that epoch. For some reason, in this case the shuffling is identical. It could be a coincidence where although the block roots are different for the target epoch, the shufflings are the same because of different balances or something, but we need to investigate more.
https://github.com/sigp/lighthouse/blob/d6ba8c397557f5c977b70f0d822a9228e98ca214/beacon_node/beacon_chain/src/beacon_chain.rs#L2349-L2354
I had a quick look at the failing test and I'm not sure why it would be failing.
The test is trying to check that the implementation of
shuffling_is_compatibleis correct. In particular, it is failing onFORK_NAME=electrain a case where the function returnsfalse, which should imply that the shuffling from the state is not compatible with the shuffling for the block at that epoch. For some reason, in this case the shuffling is identical. It could be a coincidence where although the block roots are different for the target epoch, the shufflings are the same because of different balances or something, but we need to investigate more.https://github.com/sigp/lighthouse/blob/d6ba8c397557f5c977b70f0d822a9228e98ca214/beacon_node/beacon_chain/src/beacon_chain.rs#L2349-L2354
I tried looking at it for a couple hours but I'm really not familiar with a lot of this. Things I noticed that might be related:
- note the first block in
electrawill not have attestations (not sure if relevant) - this comment
- the test fails on the first out of four calls to this function
- the original issue this test was created for
- the PR that introduces this test
- compiling with
logging/test_loggerreveals more information (though similiar logs show up withdeneb):
Aug 15 18:10:11.253 DEBG Skipping attestation with incompatible shuffling, reason: target epoch less than block epoch, target_epoch: 2, block_root: 0xd302e84c225a5dcc90399ff398016c9e586a16f4cb49bc3240b17fe162a9302d, module: beacon_chain::beacon_chain:2417
Aug 15 18:10:11.253 DEBG Skipping attestation with incompatible shuffling, reason: BeaconStateError(RelativeEpochError(EpochTooLow { base: Epoch(3), other: Epoch(1) })), target_epoch: 1, block_root: 0xd302e84c225a5dcc90399ff398016c9e586a16f4cb49bc3240b17fe162a9302d, module: beacon_chain::beacon_chain:2372
Aug 15 18:10:11.253 DEBG Skipping attestation with incompatible shuffling, reason: BeaconStateError(RelativeEpochError(EpochTooLow { base: Epoch(3), other: Epoch(1) })), target_epoch: 1, block_root: 0x315a49c54e251fc6deb99c2b1f13adfb5af130ca1994f7e912d9291853fc1248, module: beacon_chain::beacon_chain:2372
Aug 15 18:10:11.253 DEBG Skipping attestation with incompatible shuffling, reason: BeaconStateError(RelativeEpochError(EpochTooLow { base: Epoch(3), other: Epoch(1) })), target_epoch: 1, block_root: 0x20ad937216bfe975cb81aae7c606eed450e224795aaad984ca9dfb0fed8f0b9e, module: beacon_chain::beacon_chain:2372
Aug 15 18:10:11.253 DEBG Skipping attestation with incompatible shuffling, reason: BeaconStateError(RelativeEpochError(EpochTooLow { base: Epoch(3), other: Epoch(1) })), target_epoch: 1, block_root: 0xc92a1cd1d26c1c994e9e359229d43023570e957f5221e75cf71204dd7ded14b0, module: beacon_chain::beacon_chain:2372
Aug 15 18:10:11.253 DEBG Skipping attestation with incompatible shuffling, reason: BeaconStateError(RelativeEpochError(EpochTooLow { base: Epoch(3), other: Epoch(1) })), target_epoch: 1, block_root: 0x859fe968a700b007839dd749087337283517e42ded47a8179d01e8033061134d, module: beacon_chain::beacon_chain:2372
Aug 15 18:10:11.253 DEBG Skipping attestation with incompatible shuffling, reason: BeaconStateError(RelativeEpochError(EpochTooLow { base: Epoch(3), other: Epoch(1) })), target_epoch: 1, block_root: 0xb0698492eb32fe086c65c68e02447dd9479724afc925a33211e3cddb60a15477, module: beacon_chain::beacon_chain:2372
Aug 15 18:10:11.253 DEBG Skipping attestation with incompatible shuffling, reason: BeaconStateError(RelativeEpochError(EpochTooLow { base: Epoch(3), other: Epoch(1) })), target_epoch: 1, block_root: 0xdaa46d507f717a011cac6be00eda1c82a4e135b7949fc0ce310a86fa7f520883, module: beacon_chain::beacon_chain:2372
Aug 15 18:10:11.253 DEBG Skipping attestation with incompatible shuffling, reason: BeaconStateError(RelativeEpochError(EpochTooLow { base: Epoch(3), other: Epoch(1) })), target_epoch: 1, block_root: 0xbeb529ebede7b568e7b46d7b54e935cecc384caff06fa1f9f882078240041b35, module: beacon_chain::beacon_chain:2372
Aug 15 18:10:11.253 DEBG Skipping attestation with incompatible shuffling, reason: BeaconStateError(RelativeEpochError(EpochTooLow { base: Epoch(3), other: Epoch(1) })), target_epoch: 1, block_root: 0xababe5a8c5a459d1f43f71a1fa17cc7cc34c81d04c38d223d5e6ced7813c4c43, module: beacon_chain::beacon_chain:2372
Aug 15 18:10:11.253 DEBG Skipping attestation with incompatible shuffling, reason: BeaconStateError(RelativeEpochError(EpochTooLow { base: Epoch(3), other: Epoch(1) })), target_epoch: 1, block_root: 0xa3a9eee5714700763423d3eefee7b7d98518210250116841c8a16067638d98f2, module: beacon_chain::beacon_chain:2372
Aug 15 18:10:11.253 DEBG Skipping attestation with incompatible shuffling, block_shuffling_id: AttestationShufflingId { shuffling_epoch: Epoch(3), shuffling_decision_block: 0x60a5831944c1fc55577ce221bc2e25d89840ecbb5e022c91efe40c4b0a041973 }, head_shuffling_id: AttestationShufflingId { shuffling_epoch: Epoch(3), shuffling_decision_block: 0xa3a9eee5714700763423d3eefee7b7d98518210250116841c8a16067638d98f2 }, target_epoch: 3, block_root: 0x60a5831944c1fc55577ce221bc2e25d89840ecbb5e022c91efe40c4b0a041973, module: beacon_chain::beacon_chain:2430
Aug 15 18:10:11.253 DEBG Skipping attestation with incompatible shuffling, reason: BeaconStateError(RelativeEpochError(EpochTooLow { base: Epoch(3), other: Epoch(1) })), target_epoch: 1, block_root: 0x60a5831944c1fc55577ce221bc2e25d89840ecbb5e022c91efe40c4b0a041973, module: beacon_chain::beacon_chain:2372
Aug 15 18:10:11.253 DEBG Skipping attestation with incompatible shuffling, block_shuffling_id: AttestationShufflingId { shuffling_epoch: Epoch(3), shuffling_decision_block: 0x5a0aefa73d89b7729a394644518a2ab94fc712512030910536299537e9731cd5 }, head_shuffling_id: AttestationShufflingId { shuffling_epoch: Epoch(3), shuffling_decision_block: 0xa3a9eee5714700763423d3eefee7b7d98518210250116841c8a16067638d98f2 }, target_epoch: 3, block_root: 0x5a0aefa73d89b7729a394644518a2ab94fc712512030910536299537e9731cd5, module: beacon_chain::beacon_chain:2430
Aug 15 18:10:11.253 DEBG Committee cache miss, head_block_root: 0x5a0a…1cd5, shuffling_id: Epoch(3), module: beacon_chain::beacon_chain:6389
Aug 15 18:10:11.253 DEBG Skipping attestation with incompatible shuffling, reason: BeaconStateError(RelativeEpochError(EpochTooLow { base: Epoch(3), other: Epoch(1) })), target_epoch: 1, block_root: 0x5a0aefa73d89b7729a394644518a2ab94fc712512030910536299537e9731cd5, module: beacon_chain::beacon_chain:2372
Aug 15 18:10:11.253 DEBG Skipping attestation with incompatible shuffling, block_shuffling_id: AttestationShufflingId { shuffling_epoch: Epoch(3), shuffling_decision_block: 0x4c2af55b7cff18749c253f92588e138b19dd27a1d06834ec98d87d8400b72ab8 }, head_shuffling_id: AttestationShufflingId { shuffling_epoch: Epoch(3), shuffling_decision_block: 0xa3a9eee5714700763423d3eefee7b7d98518210250116841c8a16067638d98f2 }, target_epoch: 3, block_root: 0x4c2af55b7cff18749c253f92588e138b19dd27a1d06834ec98d87d8400b72ab8, module: beacon_chain::beacon_chain:2430
Aug 15 18:10:11.253 DEBG Committee cache miss, head_block_root: 0x4c2a…2ab8, shuffling_id: Epoch(3), module: beacon_chain::beacon_chain:6389
Aug 15 18:10:11.253 DEBG Skipping attestation with incompatible shuffling, reason: BeaconStateError(RelativeEpochError(EpochTooLow { base: Epoch(3), other: Epoch(1) })), target_epoch: 1, block_root: 0x4c2af55b7cff18749c253f92588e138b19dd27a1d06834ec98d87d8400b72ab8, module: beacon_chain::beacon_chain:2372
Aug 15 18:10:11.253 DEBG Skipping attestation with incompatible shuffling, block_shuffling_id: AttestationShufflingId { shuffling_epoch: Epoch(3), shuffling_decision_block: 0x56d1e2f8b9f59923f6d49165db56f812b902c0122ef4855bd14af70b74fdbb96 }, head_shuffling_id: AttestationShufflingId { shuffling_epoch: Epoch(3), shuffling_decision_block: 0xa3a9eee5714700763423d3eefee7b7d98518210250116841c8a16067638d98f2 }, target_epoch: 3, block_root: 0x56d1e2f8b9f59923f6d49165db56f812b902c0122ef4855bd14af70b74fdbb96, module: beacon_chain::beacon_chain:2430
Aug 15 18:10:11.253 DEBG Committee cache miss, head_block_root: 0x56d1…bb96, shuffling_id: Epoch(3), module: beacon_chain::beacon_chain:6389
Aug 15 18:10:11.253 DEBG Skipping attestation with incompatible shuffling, reason: BeaconStateError(RelativeEpochError(EpochTooLow { base: Epoch(3), other: Epoch(1) })), target_epoch: 1, block_root: 0x56d1e2f8b9f59923f6d49165db56f812b902c0122ef4855bd14af70b74fdbb96, module: beacon_chain::beacon_chain:2372
Aug 15 18:10:11.253 DEBG Skipping attestation with incompatible shuffling, block_shuffling_id: AttestationShufflingId { shuffling_epoch: Epoch(3), shuffling_decision_block: 0x65cad4da00c123912ea1f1be360331dc9251da5c4b9b1b95b2e274b47bebb5c9 }, head_shuffling_id: AttestationShufflingId { shuffling_epoch: Epoch(3), shuffling_decision_block: 0xa3a9eee5714700763423d3eefee7b7d98518210250116841c8a16067638d98f2 }, target_epoch: 3, block_root: 0x65cad4da00c123912ea1f1be360331dc9251da5c4b9b1b95b2e274b47bebb5c9, module: beacon_chain::beacon_chain:2430
Aug 15 18:10:11.253 DEBG Committee cache miss, head_block_root: 0x65ca…b5c9, shuffling_id: Epoch(3), module: beacon_chain::beacon_chain:6389
Aug 15 18:10:11.253 DEBG Skipping attestation with incompatible shuffling, reason: BeaconStateError(RelativeEpochError(EpochTooLow { base: Epoch(3), other: Epoch(1) })), target_epoch: 1, block_root: 0x65cad4da00c123912ea1f1be360331dc9251da5c4b9b1b95b2e274b47bebb5c9, module: beacon_chain::beacon_chain:2372
Aug 15 18:10:11.253 DEBG Skipping attestation with incompatible shuffling, block_shuffling_id: AttestationShufflingId { shuffling_epoch: Epoch(3), shuffling_decision_block: 0x75a0ef7401de9500ccb064619fce6d43ea111be782af3499fc9c51126aa89837 }, head_shuffling_id: AttestationShufflingId { shuffling_epoch: Epoch(3), shuffling_decision_block: 0xa3a9eee5714700763423d3eefee7b7d98518210250116841c8a16067638d98f2 }, target_epoch: 3, block_root: 0x75a0ef7401de9500ccb064619fce6d43ea111be782af3499fc9c51126aa89837, module: beacon_chain::beacon_chain:2430
Aug 15 18:10:11.253 DEBG Committee cache miss, head_block_root: 0x75a0…9837, shuffling_id: Epoch(3), module: beacon_chain::beacon_chain:6389
Aug 15 18:10:11.253 DEBG Skipping attestation with incompatible shuffling, reason: BeaconStateError(RelativeEpochError(EpochTooLow { base: Epoch(3), other: Epoch(1) })), target_epoch: 1, block_root: 0x75a0ef7401de9500ccb064619fce6d43ea111be782af3499fc9c51126aa89837, module: beacon_chain::beacon_chain:2372
Aug 15 18:10:11.253 DEBG Skipping attestation with incompatible shuffling, block_shuffling_id: AttestationShufflingId { shuffling_epoch: Epoch(3), shuffling_decision_block: 0x7020fb41face5c6e1f03f6d1c6a6134bc69351cfb2e426296e646471d306e84e }, head_shuffling_id: AttestationShufflingId { shuffling_epoch: Epoch(3), shuffling_decision_block: 0xa3a9eee5714700763423d3eefee7b7d98518210250116841c8a16067638d98f2 }, target_epoch: 3, block_root: 0x7020fb41face5c6e1f03f6d1c6a6134bc69351cfb2e426296e646471d306e84e, module: beacon_chain::beacon_chain:2430
Aug 15 18:10:11.253 DEBG Committee cache miss, head_block_root: 0x7020…e84e, shuffling_id: Epoch(3), module: beacon_chain::beacon_chain:6389
Aug 15 18:10:11.253 DEBG Skipping attestation with incompatible shuffling, reason: BeaconStateError(RelativeEpochError(EpochTooLow { base: Epoch(3), other: Epoch(1) })), target_epoch: 1, block_root: 0x7020fb41face5c6e1f03f6d1c6a6134bc69351cfb2e426296e646471d306e84e, module: beacon_chain::beacon_chain:2372
Aug 15 18:10:11.253 DEBG Skipping attestation with incompatible shuffling, block_shuffling_id: AttestationShufflingId { shuffling_epoch: Epoch(3), shuffling_decision_block: 0x97f52636fb8ff871cff07a262ac655f518280c0a90bd873b828020d31d6cd0d1 }, head_shuffling_id: AttestationShufflingId { shuffling_epoch: Epoch(3), shuffling_decision_block: 0xa3a9eee5714700763423d3eefee7b7d98518210250116841c8a16067638d98f2 }, target_epoch: 3, block_root: 0x97f52636fb8ff871cff07a262ac655f518280c0a90bd873b828020d31d6cd0d1, module: beacon_chain::beacon_chain:2430
Aug 15 18:10:11.253 DEBG Committee cache miss, head_block_root: 0x97f5…d0d1, shuffling_id: Epoch(3), module: beacon_chain::beacon_chain:6389
Aug 15 18:10:11.253 DEBG Skipping attestation with incompatible shuffling, reason: BeaconStateError(RelativeEpochError(EpochTooLow { base: Epoch(3), other: Epoch(1) })), target_epoch: 1, block_root: 0x97f52636fb8ff871cff07a262ac655f518280c0a90bd873b828020d31d6cd0d1, module: beacon_chain::beacon_chain:2372
Aug 15 18:10:11.253 DEBG Skipping attestation with incompatible shuffling, block_shuffling_id: AttestationShufflingId { shuffling_epoch: Epoch(3), shuffling_decision_block: 0xd8605e323e06d5d0f47faec5bde80542ca05cf0b931cca3891e13edaf3383357 }, head_shuffling_id: AttestationShufflingId { shuffling_epoch: Epoch(3), shuffling_decision_block: 0xa3a9eee5714700763423d3eefee7b7d98518210250116841c8a16067638d98f2 }, target_epochthread ':store_tests::shuffling_compatible_short_fork ' panicked at 3, block_root: 0xd8605e323e06d5d0f47faec5bde80542ca05cf0b931cca3891e13edaf3383357, beacon_node/beacon_chain/tests/store_tests.rsmodule::1488 :beacon_chain::beacon_chain:243025:
Ok I've worked out what's going on and it's pretty insane.
The shufflings for epoch 2 derived from the block at slot 3 and the block at slot 7 are identical, because the RANDAO is updated with 2 copies of the same 2 randao reveals.
Aug 16 03:04:30.576 DEBG Randao mix for current epoch after block, randao_mix: 0xb74a3e6b38202ebc5ff324be848d735e1572b4543129a6293ce8586da2f88b95ede63ceca287a609789a09e07f95e86e033e513666d1d440f19412458d330eab3f5513e465c6d20510a131907cbc9144c896960cc0324bd973b6c29d481bcc60, randao: 0x97544f4064ee58d3c09daab459da18492fd76b2285b623862d46bb29566e68c8, slot: 3, block_root: 0x54eeef777fc9269c43b64fd2722540b61da9e5a8460a389842efc777b0a5fc85, module: beacon_chain::beacon_chain:3732
This shows that the randao (in the state) is 0x97544f4064ee58d3c09daab459da18492fd76b2285b623862d46bb29566e68c8 after the block at slot 3.
We also see that the randao has the same value after slot 7:
Aug 16 03:04:30.716 DEBG Randao mix for current epoch after block, randao_mix: 0xb6885349b6e59ccafb77bedaf543f1f1b0ffbaa84395cee195f50630a4a05659e05f75eab4b78dc9a2a9a46431bd972e0acaff928277080f99224e2283e7d00d1171d3e1e862f995b565a463950d9a41bc39a2bbacddd40ddd59a51360966b3f, randao: 0x97544f4064ee58d3c09daab459da18492fd76b2285b623862d46bb29566e68c8, slot: 7, block_root: 0x97f52636fb8ff871cff07a262ac655f518280c0a90bd873b828020d31d6cd0d1, module: beacon_chain::beacon_chain:3732
Again, randao in the state is 0x97544f4064ee58d3c09daab459da18492fd76b2285b623862d46bb29566e68c8. At first I thought this would be an impossibly unlikely coincidence, but RANDAO works by XOR, and it happens that the 4 values XOR in during slots 4, 5, 6, 7 exactly cancel each other out:
Aug 16 03:04:30.593 DEBG Randao mix for current epoch after block, randao_mix: 0xa3ad9bdfa11bc5985be0789a81ce5a12a16b855a5afd3d3fec75337503e28a0cbd0c80e5865ff52db412530a7b49f21617ad95d7016db2b3ffc7681ca70414e52c3f8300bb8b7c0f28d51b88c9bac7229079a8504ceacdaa7b90311f148c2fda, randao: 0xfbfde93428e5d9522b04eff529f84649e43f117714e5add7e3e47d59ed222c2e, slot: 4, block_root: 0x0d60b83bb8428017b63b592eafd982e901fb04724f6781f7a36374f64f13bf25, module: beacon_chain::beacon_chain:3732 Aug 16 03:04:30.640 DEBG Randao mix for current epoch after block, randao_mix: 0xb6885349b6e59ccafb77bedaf543f1f1b0ffbaa84395cee195f50630a4a05659e05f75eab4b78dc9a2a9a46431bd972e0acaff928277080f99224e2283e7d00d1171d3e1e862f995b565a463950d9a41bc39a2bbacddd40ddd59a51360966b3f, randao: 0x03ea6ce6c94d27274b62e92e24815e89fc0925cd02c4befa7735182339010d1b, slot: 5, block_root: 0x0940ba3a8f79741a36d26b073b311001b354e40126988c01febeaadb5da93067, module: beacon_chain::beacon_chain:3732 Aug 16 03:04:30.676 DEBG Randao mix for current epoch after block, randao_mix: 0xa3ad9bdfa11bc5985be0789a81ce5a12a16b855a5afd3d3fec75337503e28a0cbd0c80e5865ff52db412530a7b49f21617ad95d7016db2b3ffc7681ca70414e52c3f8300bb8b7c0f28d51b88c9bac7229079a8504ceacdaa7b90311f148c2fda, randao: 0x6f43ca928546a6a6a0fbac6f54a3008937e15f98939730abb997de53824d49fd, slot: 6, block_root: 0xd8605e323e06d5d0f47faec5bde80542ca05cf0b931cca3891e13edaf3383357, module: beacon_chain::beacon_chain:3732 Aug 16 03:04:30.716 DEBG Randao mix for current epoch after block, randao_mix: 0xb6885349b6e59ccafb77bedaf543f1f1b0ffbaa84395cee195f50630a4a05659e05f75eab4b78dc9a2a9a46431bd972e0acaff928277080f99224e2283e7d00d1171d3e1e862f995b565a463950d9a41bc39a2bbacddd40ddd59a51360966b3f, randao: 0x97544f4064ee58d3c09daab459da18492fd76b2285b623862d46bb29566e68c8, slot: 7, block_root: 0x97f52636fb8ff871cff07a262ac655f518280c0a90bd873b828020d31d6cd0d1, module: beacon_chain::beacon_chain:3732
The randao mixes being XORed in are:
-
0xa3ad9bdfa11bc5985be0789a81ce5a12a16b855a5afd3d3fec75337503e28a0cbd0c80e5865ff52db412530a7b49f(slot 4) -
0xb6885349b6e59ccafb77bedaf543f1f1b0ffbaa84395cee195f50630a4a05659e05f75eab4b78dc9a2a9a46431bd9(slot 5) -
0xa3ad9bdfa11bc5985be0789a81ce5a12a16b855a5afd3d3fec75337503e28a0cbd0c80e5865ff52db412530a7b49f(slot 6, same as slot 4) -
0xb6885349b6e59ccafb77bedaf543f1f1b0ffbaa84395cee195f50630a4a05659e05f75eab4b78dc9a2a9a46431bd9(slot 7, same as slot 5)
So we are essentially doing A ^ B ^ C ^ B ^ C which is equal to A (because ^ is commutative and X ^ X = 0).
The reason the randao reveals are the same is that the proposers for slots 4&6 and 5&7 are the same. This is not that unlikely on a small network (24 validators).
So this is a bug completed unrelated to Electra which has been lying dormant since shuffling_is_compatible was implemented.
I need to think more about the best way to fix it in the short-term and the long-term
There's one more failing test in the validator monitor. It seems like it's just poorly written and requires manual tweaking. I haven't had a chance to fix it yet
queue
✅ The pull request has been merged automatically
The pull request has been merged automatically at 9a295d09b4d79100ead84c25a2e6e7ee23e942a1
![mergify[bot] avatar](https://avatars.githubusercontent.com/in/10562?v=4 "Published by a pub.dev verified publisher"){kind=small}