lodestar
lodestar copied to clipboard
ChainSafe
GC non pruned states
Motivation
- PR https://github.com/ChainSafe/lodestar/pull/4508 fixes an issue where Lodestar stores more states than necessary.
For Prater/Goerli nodes running for a few months that extra space accounted for 50GB / 120GB of db data pre-pruning.
I would be nice to automatically prune the existing states since it's very cheap and reliable to detect the non-pruned states.
Description
Once on start-up, read the archive_states bucket keys to check if there are states that should be pruned. If there are:
- Remove them all in batches
- Trigger compactRange on archive_states bucket. If this is not done the total db size does not decrease. See for details on some experiments https://github.com/ChainSafe/lodestar/issues/4515
marking as draft as test of this branch has caused issues in nodes, still unknown
Sep-07 14:02:22.367[eth1] error: Error updating eth1 chain cache Database is not open
Error: Database is not open
at ClassicLevel.values (/usr/src/lodestar/node_modules/abstract-level/abstract-level.js:726:13)
at LevelDbController.values (file:///usr/src/lodestar/packages/db/src/controller/level.ts:134:36)
at DepositEventRepository.values (file:///usr/src/lodestar/packages/db/src/abstractRepository.ts:161:32)
at DepositEventRepository.lastValue (file:///usr/src/lodestar/packages/db/src/abstractRepository.ts:226:31)
at Eth1DepositsCache.add (file:///usr/src/lodestar/packages/beacon-node/src/eth1/eth1DepositsCache.ts:44:48)
at Eth1DepositDataTracker.updateDepositCache (file:///usr/src/lodestar/packages/beacon-node/src/eth1/eth1DepositDataTracker.ts:211:30)
at runMicrotasks (<anonymous>)
at processTicksAndRejections (node:internal/process/task_queues:96:5)
at Eth1DepositDataTracker.update (file:///usr/src/lodestar/packages/beacon-node/src/eth1/eth1DepositDataTracker.ts:191:33)
at Eth1DepositDataTracker.runAutoUpdate (file:///usr/src/lodestar/packages/beacon-node/src/eth1/eth1DepositDataTracker.ts:154:29)
From https://github.com/ChainSafe/lodestar/pull/4508#issuecomment-1239311911
Performance Report
✔️ no performance regression detected
Full benchmark results
| Benchmark suite | Current: 73034a17cbba323a3053b6d2a3f61318e5aefd38 | Previous: ac755d9e317651199953dd36acbd035ce9c3c224 | Ratio |
|---|---|---|---|
| getPubkeys - index2pubkey - req 1000 vs - 250000 vc | 2.0825 ms/op | 1.8254 ms/op | 1.14 |
| getPubkeys - validatorsArr - req 1000 vs - 250000 vc | 67.028 us/op | 64.583 us/op | 1.04 |
| BLS verify - blst-native | 2.1647 ms/op | 2.1636 ms/op | 1.00 |
| BLS verifyMultipleSignatures 3 - blst-native | 4.4853 ms/op | 4.4772 ms/op | 1.00 |
| BLS verifyMultipleSignatures 8 - blst-native | 9.7204 ms/op | 9.6675 ms/op | 1.01 |
| BLS verifyMultipleSignatures 32 - blst-native | 35.218 ms/op | 35.144 ms/op | 1.00 |
| BLS aggregatePubkeys 32 - blst-native | 46.462 us/op | 46.462 us/op | 1.00 |
| BLS aggregatePubkeys 128 - blst-native | 182.14 us/op | 182.29 us/op | 1.00 |
| getAttestationsForBlock | 158.47 ms/op | 151.08 ms/op | 1.05 |
| isKnown best case - 1 super set check | 469.00 ns/op | 480.00 ns/op | 0.98 |
| isKnown normal case - 2 super set checks | 456.00 ns/op | 470.00 ns/op | 0.97 |
| isKnown worse case - 16 super set checks | 454.00 ns/op | 467.00 ns/op | 0.97 |
| CheckpointStateCache - add get delete | 9.0720 us/op | 8.9150 us/op | 1.02 |
| validate gossip signedAggregateAndProof - struct | 5.0279 ms/op | 5.0262 ms/op | 1.00 |
| validate gossip attestation - struct | 2.3648 ms/op | 2.3852 ms/op | 0.99 |
| pickEth1Vote - no votes | 2.1272 ms/op | 2.1516 ms/op | 0.99 |
| pickEth1Vote - max votes | 20.443 ms/op | 18.824 ms/op | 1.09 |
| pickEth1Vote - Eth1Data hashTreeRoot value x2048 | 13.073 ms/op | 13.265 ms/op | 0.99 |
| pickEth1Vote - Eth1Data hashTreeRoot tree x2048 | 21.819 ms/op | 21.634 ms/op | 1.01 |
| pickEth1Vote - Eth1Data fastSerialize value x2048 | 1.5818 ms/op | 1.5836 ms/op | 1.00 |
| pickEth1Vote - Eth1Data fastSerialize tree x2048 | 13.966 ms/op | 13.093 ms/op | 1.07 |
| bytes32 toHexString | 1.0710 us/op | 976.00 ns/op | 1.10 |
| bytes32 Buffer.toString(hex) | 831.00 ns/op | 790.00 ns/op | 1.05 |
| bytes32 Buffer.toString(hex) from Uint8Array | 1.0730 us/op | 1.0460 us/op | 1.03 |
| bytes32 Buffer.toString(hex) + 0x | 837.00 ns/op | 808.00 ns/op | 1.04 |
| Object access 1 prop | 0.38900 ns/op | 0.39200 ns/op | 0.99 |
| Map access 1 prop | 0.30600 ns/op | 0.31500 ns/op | 0.97 |
| Object get x1000 | 12.695 ns/op | 10.900 ns/op | 1.16 |
| Map get x1000 | 1.0080 ns/op | 1.0990 ns/op | 0.92 |
| Object set x1000 | 82.954 ns/op | 72.958 ns/op | 1.14 |
| Map set x1000 | 52.260 ns/op | 48.857 ns/op | 1.07 |
| Return object 10000 times | 0.44140 ns/op | 0.44070 ns/op | 1.00 |
| Throw Error 10000 times | 5.9418 us/op | 5.9723 us/op | 0.99 |
| enrSubnets - fastDeserialize 64 bits | 2.8320 us/op | 2.4940 us/op | 1.14 |
| enrSubnets - ssz BitVector 64 bits | 785.00 ns/op | 716.00 ns/op | 1.10 |
| enrSubnets - fastDeserialize 4 bits | 409.00 ns/op | 360.00 ns/op | 1.14 |
| enrSubnets - ssz BitVector 4 bits | 792.00 ns/op | 738.00 ns/op | 1.07 |
| prioritizePeers score -10:0 att 32-0.1 sync 2-0 | 90.576 us/op | 82.147 us/op | 1.10 |
| prioritizePeers score 0:0 att 32-0.25 sync 2-0.25 | 115.35 us/op | 115.11 us/op | 1.00 |
| prioritizePeers score 0:0 att 32-0.5 sync 2-0.5 | 212.22 us/op | 195.56 us/op | 1.09 |
| prioritizePeers score 0:0 att 64-0.75 sync 4-0.75 | 407.75 us/op | 382.31 us/op | 1.07 |
| prioritizePeers score 0:0 att 64-1 sync 4-1 | 405.56 us/op | 407.10 us/op | 1.00 |
| RateTracker 1000000 limit, 1 obj count per request | 189.44 ns/op | 182.78 ns/op | 1.04 |
| RateTracker 1000000 limit, 2 obj count per request | 140.78 ns/op | 133.01 ns/op | 1.06 |
| RateTracker 1000000 limit, 4 obj count per request | 113.76 ns/op | 108.43 ns/op | 1.05 |
| RateTracker 1000000 limit, 8 obj count per request | 105.09 ns/op | 98.759 ns/op | 1.06 |
| RateTracker with prune | 4.2700 us/op | 4.1090 us/op | 1.04 |
| array of 16000 items push then shift | 51.593 us/op | 51.613 us/op | 1.00 |
| LinkedList of 16000 items push then shift | 12.689 ns/op | 12.569 ns/op | 1.01 |
| array of 16000 items push then pop | 208.34 ns/op | 213.45 ns/op | 0.98 |
| LinkedList of 16000 items push then pop | 12.313 ns/op | 12.123 ns/op | 1.02 |
| array of 24000 items push then shift | 77.359 us/op | 77.349 us/op | 1.00 |
| LinkedList of 24000 items push then shift | 13.385 ns/op | 12.956 ns/op | 1.03 |
| array of 24000 items push then pop | 199.34 ns/op | 196.41 ns/op | 1.01 |
| LinkedList of 24000 items push then pop | 12.413 ns/op | 12.247 ns/op | 1.01 |
| intersect bitArray bitLen 8 | 10.711 ns/op | 10.846 ns/op | 0.99 |
| intersect array and set length 8 | 141.74 ns/op | 133.58 ns/op | 1.06 |
| intersect bitArray bitLen 128 | 55.526 ns/op | 55.666 ns/op | 1.00 |
| intersect array and set length 128 | 1.9064 us/op | 1.7903 us/op | 1.06 |
| Buffer.concat 32 items | 2.0220 ns/op | 2.0250 ns/op | 1.00 |
| pass gossip attestations to forkchoice per slot | 3.0672 ms/op | 3.0652 ms/op | 1.00 |
| computeDeltas | 3.2675 ms/op | 3.6153 ms/op | 0.90 |
| computeProposerBoostScoreFromBalances | 808.85 us/op | 803.84 us/op | 1.01 |
| altair processAttestation - 250000 vs - 7PWei normalcase | 3.7964 ms/op | 3.3991 ms/op | 1.12 |
| altair processAttestation - 250000 vs - 7PWei worstcase | 5.6622 ms/op | 5.1740 ms/op | 1.09 |
| altair processAttestation - setStatus - 1/6 committees join | 184.22 us/op | 180.58 us/op | 1.02 |
| altair processAttestation - setStatus - 1/3 committees join | 360.49 us/op | 354.19 us/op | 1.02 |
| altair processAttestation - setStatus - 1/2 committees join | 513.14 us/op | 500.06 us/op | 1.03 |
| altair processAttestation - setStatus - 2/3 committees join | 668.29 us/op | 657.56 us/op | 1.02 |
| altair processAttestation - setStatus - 4/5 committees join | 931.97 us/op | 921.26 us/op | 1.01 |
| altair processAttestation - setStatus - 100% committees join | 1.1342 ms/op | 1.1140 ms/op | 1.02 |
| altair processBlock - 250000 vs - 7PWei normalcase | 26.815 ms/op | 24.124 ms/op | 1.11 |
| altair processBlock - 250000 vs - 7PWei normalcase hashState | 34.419 ms/op | 38.249 ms/op | 0.90 |
| altair processBlock - 250000 vs - 7PWei worstcase | 89.303 ms/op | 82.245 ms/op | 1.09 |
| altair processBlock - 250000 vs - 7PWei worstcase hashState | 101.88 ms/op | 111.64 ms/op | 0.91 |
| phase0 processBlock - 250000 vs - 7PWei normalcase | 3.5791 ms/op | 3.2540 ms/op | 1.10 |
| phase0 processBlock - 250000 vs - 7PWei worstcase | 52.086 ms/op | 50.665 ms/op | 1.03 |
| altair processEth1Data - 250000 vs - 7PWei normalcase | 763.28 us/op | 690.64 us/op | 1.11 |
| Tree 40 250000 create | 713.52 ms/op | 707.85 ms/op | 1.01 |
| Tree 40 250000 get(125000) | 232.75 ns/op | 264.59 ns/op | 0.88 |
| Tree 40 250000 set(125000) | 2.1660 us/op | 2.2053 us/op | 0.98 |
| Tree 40 250000 toArray() | 27.954 ms/op | 27.183 ms/op | 1.03 |
| Tree 40 250000 iterate all - toArray() + loop | 28.420 ms/op | 27.685 ms/op | 1.03 |
| Tree 40 250000 iterate all - get(i) | 118.38 ms/op | 110.62 ms/op | 1.07 |
| MutableVector 250000 create | 14.888 ms/op | 12.510 ms/op | 1.19 |
| MutableVector 250000 get(125000) | 10.621 ns/op | 10.888 ns/op | 0.98 |
| MutableVector 250000 set(125000) | 548.47 ns/op | 532.57 ns/op | 1.03 |
| MutableVector 250000 toArray() | 6.6104 ms/op | 6.0997 ms/op | 1.08 |
| MutableVector 250000 iterate all - toArray() + loop | 6.7712 ms/op | 6.0809 ms/op | 1.11 |
| MutableVector 250000 iterate all - get(i) | 2.7256 ms/op | 2.7632 ms/op | 0.99 |
| Array 250000 create | 6.4647 ms/op | 5.8296 ms/op | 1.11 |
| Array 250000 clone - spread | 3.2837 ms/op | 3.1366 ms/op | 1.05 |
| Array 250000 get(125000) | 1.5290 ns/op | 1.4370 ns/op | 1.06 |
| Array 250000 set(125000) | 1.5090 ns/op | 1.4720 ns/op | 1.03 |
| Array 250000 iterate all - loop | 150.95 us/op | 151.00 us/op | 1.00 |
| effectiveBalanceIncrements clone Uint8Array 300000 | 50.261 us/op | 137.46 us/op | 0.37 |
| effectiveBalanceIncrements clone MutableVector 300000 | 1.0800 us/op | 640.00 ns/op | 1.69 |
| effectiveBalanceIncrements rw all Uint8Array 300000 | 247.17 us/op | 247.07 us/op | 1.00 |
| effectiveBalanceIncrements rw all MutableVector 300000 | 174.05 ms/op | 131.75 ms/op | 1.32 |
| phase0 afterProcessEpoch - 250000 vs - 7PWei | 193.77 ms/op | 189.15 ms/op | 1.02 |
| phase0 beforeProcessEpoch - 250000 vs - 7PWei | 74.800 ms/op | 57.362 ms/op | 1.30 |
| altair processEpoch - mainnet_e81889 | 552.90 ms/op | 574.62 ms/op | 0.96 |
| mainnet_e81889 - altair beforeProcessEpoch | 133.24 ms/op | 81.213 ms/op | 1.64 |
| mainnet_e81889 - altair processJustificationAndFinalization | 18.163 us/op | 21.123 us/op | 0.86 |
| mainnet_e81889 - altair processInactivityUpdates | 8.9575 ms/op | 9.4378 ms/op | 0.95 |
| mainnet_e81889 - altair processRewardsAndPenalties | 81.303 ms/op | 126.80 ms/op | 0.64 |
| mainnet_e81889 - altair processRegistryUpdates | 2.7580 us/op | 4.2820 us/op | 0.64 |
| mainnet_e81889 - altair processSlashings | 714.00 ns/op | 1.2270 us/op | 0.58 |
| mainnet_e81889 - altair processEth1DataReset | 681.00 ns/op | 1.1550 us/op | 0.59 |
| mainnet_e81889 - altair processEffectiveBalanceUpdates | 2.1931 ms/op | 2.4479 ms/op | 0.90 |
| mainnet_e81889 - altair processSlashingsReset | 4.4500 us/op | 6.6480 us/op | 0.67 |
| mainnet_e81889 - altair processRandaoMixesReset | 4.2830 us/op | 6.3480 us/op | 0.67 |
| mainnet_e81889 - altair processHistoricalRootsUpdate | 721.00 ns/op | 994.00 ns/op | 0.73 |
| mainnet_e81889 - altair processParticipationFlagUpdates | 2.1450 us/op | 2.6100 us/op | 0.82 |
| mainnet_e81889 - altair processSyncCommitteeUpdates | 609.00 ns/op | 833.00 ns/op | 0.73 |
| mainnet_e81889 - altair afterProcessEpoch | 202.30 ms/op | 200.94 ms/op | 1.01 |
| phase0 processEpoch - mainnet_e58758 | 492.82 ms/op | 494.43 ms/op | 1.00 |
| mainnet_e58758 - phase0 beforeProcessEpoch | 187.57 ms/op | 188.67 ms/op | 0.99 |
| mainnet_e58758 - phase0 processJustificationAndFinalization | 17.546 us/op | 21.146 us/op | 0.83 |
| mainnet_e58758 - phase0 processRewardsAndPenalties | 120.74 ms/op | 71.354 ms/op | 1.69 |
| mainnet_e58758 - phase0 processRegistryUpdates | 7.9190 us/op | 10.478 us/op | 0.76 |
| mainnet_e58758 - phase0 processSlashings | 618.00 ns/op | 1.3330 us/op | 0.46 |
| mainnet_e58758 - phase0 processEth1DataReset | 687.00 ns/op | 800.00 ns/op | 0.86 |
| mainnet_e58758 - phase0 processEffectiveBalanceUpdates | 1.6737 ms/op | 1.9315 ms/op | 0.87 |
| mainnet_e58758 - phase0 processSlashingsReset | 3.7710 us/op | 4.8330 us/op | 0.78 |
| mainnet_e58758 - phase0 processRandaoMixesReset | 4.2080 us/op | 5.9760 us/op | 0.70 |
| mainnet_e58758 - phase0 processHistoricalRootsUpdate | 735.00 ns/op | 1.0230 us/op | 0.72 |
| mainnet_e58758 - phase0 processParticipationRecordUpdates | 3.6070 us/op | 5.3320 us/op | 0.68 |
| mainnet_e58758 - phase0 afterProcessEpoch | 166.31 ms/op | 165.05 ms/op | 1.01 |
| phase0 processEffectiveBalanceUpdates - 250000 normalcase | 1.9396 ms/op | 2.0652 ms/op | 0.94 |
| phase0 processEffectiveBalanceUpdates - 250000 worstcase 0.5 | 2.2176 ms/op | 2.5194 ms/op | 0.88 |
| altair processInactivityUpdates - 250000 normalcase | 40.119 ms/op | 33.386 ms/op | 1.20 |
| altair processInactivityUpdates - 250000 worstcase | 33.517 ms/op | 40.137 ms/op | 0.84 |
| phase0 processRegistryUpdates - 250000 normalcase | 6.5580 us/op | 9.2880 us/op | 0.71 |
| phase0 processRegistryUpdates - 250000 badcase_full_deposits | 378.81 us/op | 375.10 us/op | 1.01 |
| phase0 processRegistryUpdates - 250000 worstcase 0.5 | 194.67 ms/op | 182.03 ms/op | 1.07 |
| altair processRewardsAndPenalties - 250000 normalcase | 71.669 ms/op | 104.28 ms/op | 0.69 |
| altair processRewardsAndPenalties - 250000 worstcase | 98.597 ms/op | 108.90 ms/op | 0.91 |
| phase0 getAttestationDeltas - 250000 normalcase | 12.160 ms/op | 11.958 ms/op | 1.02 |
| phase0 getAttestationDeltas - 250000 worstcase | 13.105 ms/op | 12.124 ms/op | 1.08 |
| phase0 processSlashings - 250000 worstcase | 5.0442 ms/op | 5.2024 ms/op | 0.97 |
| altair processSyncCommitteeUpdates - 250000 | 301.20 ms/op | 301.94 ms/op | 1.00 |
| BeaconState.hashTreeRoot - No change | 545.00 ns/op | 563.00 ns/op | 0.97 |
| BeaconState.hashTreeRoot - 1 full validator | 74.475 us/op | 71.903 us/op | 1.04 |
| BeaconState.hashTreeRoot - 32 full validator | 752.66 us/op | 650.60 us/op | 1.16 |
| BeaconState.hashTreeRoot - 512 full validator | 7.0660 ms/op | 7.4920 ms/op | 0.94 |
| BeaconState.hashTreeRoot - 1 validator.effectiveBalance | 91.210 us/op | 90.327 us/op | 1.01 |
| BeaconState.hashTreeRoot - 32 validator.effectiveBalance | 1.3482 ms/op | 1.3303 ms/op | 1.01 |
| BeaconState.hashTreeRoot - 512 validator.effectiveBalance | 17.489 ms/op | 17.578 ms/op | 0.99 |
| BeaconState.hashTreeRoot - 1 balances | 72.891 us/op | 69.135 us/op | 1.05 |
| BeaconState.hashTreeRoot - 32 balances | 744.92 us/op | 660.03 us/op | 1.13 |
| BeaconState.hashTreeRoot - 512 balances | 6.3135 ms/op | 6.4038 ms/op | 0.99 |
| BeaconState.hashTreeRoot - 250000 balances | 99.764 ms/op | 100.25 ms/op | 1.00 |
| aggregationBits - 2048 els - zipIndexesInBitList | 31.086 us/op | 25.137 us/op | 1.24 |
| regular array get 100000 times | 60.560 us/op | 64.141 us/op | 0.94 |
| wrappedArray get 100000 times | 60.581 us/op | 60.862 us/op | 1.00 |
| arrayWithProxy get 100000 times | 28.317 ms/op | 29.011 ms/op | 0.98 |
| ssz.Root.equals | 535.00 ns/op | 485.00 ns/op | 1.10 |
| byteArrayEquals | 523.00 ns/op | 477.00 ns/op | 1.10 |
| shuffle list - 16384 els | 11.823 ms/op | 11.336 ms/op | 1.04 |
| shuffle list - 250000 els | 171.85 ms/op | 169.62 ms/op | 1.01 |
| processSlot - 1 slots | 15.301 us/op | 13.421 us/op | 1.14 |
| processSlot - 32 slots | 1.9723 ms/op | 2.1377 ms/op | 0.92 |
| getEffectiveBalanceIncrementsZeroInactive - 250000 vs - 7PWei | 429.99 us/op | 484.68 us/op | 0.89 |
| getCommitteeAssignments - req 1 vs - 250000 vc | 5.3973 ms/op | 5.4002 ms/op | 1.00 |
| getCommitteeAssignments - req 100 vs - 250000 vc | 7.9044 ms/op | 7.9028 ms/op | 1.00 |
| getCommitteeAssignments - req 1000 vs - 250000 vc | 8.5044 ms/op | 8.4658 ms/op | 1.00 |
| RootCache.getBlockRootAtSlot - 250000 vs - 7PWei | 7.8300 ns/op | 8.5900 ns/op | 0.91 |
| state getBlockRootAtSlot - 250000 vs - 7PWei | 1.2149 us/op | 1.1159 us/op | 1.09 |
| computeProposers - vc 250000 | 18.136 ms/op | 18.261 ms/op | 0.99 |
| computeEpochShuffling - vc 250000 | 175.25 ms/op | 173.05 ms/op | 1.01 |
| getNextSyncCommittee - vc 250000 | 299.16 ms/op | 297.85 ms/op | 1.00 |
by benchmarkbot/action
![github-actions[bot] avatar](https://avatars.githubusercontent.com/in/15368?v=4 "Published by a pub.dev verified publisher"){kind=small}
Closing as it's not worth the complexity. The original issue was fixed long ago so most users do not need this retroactive cleanup