feat: add historical state regen

[wemeetagain]

Motivation

• see #5846

Description

• Add a historical state regen module which can regenerate old states
• Optional to avoid breaking unrelated tests / mocks
• Uses a dedicated worker thread with a JobFnQueue inside
• nodejs, state transition, job queue metrics wired up
• Add historical state dashboard, which includes VM, job queue, and stfn panels
• Update VM dashboard to include historical state worker panels and update cross-worker panels
• PubkeyIndexMap persisted between runs for faster repeated use
• Deployed to feat3 on Jan 22

TODO

• [x] pull out any necessary configurability to CLI level - no CLI configurability is needed (?)
• [ ] tests
• [x] dashboard

[github-actions[bot]]

Performance Report

✔️ no performance regression detected

Full benchmark results

Benchmark suite	Current: bb8fa4911d5c0888e2fd7405ffd9da6115111518	Previous: 59f72d016758124d1dd34bba8229e465a021d400	Ratio
getPubkeys - index2pubkey - req 1000 vs - 250000 vc	2.0047 ms/op	2.8144 ms/op	0.71
getPubkeys - validatorsArr - req 1000 vs - 250000 vc	44.379 us/op	58.733 us/op	0.76
BLS verify - blst	1.4394 ms/op	1.0293 ms/op	1.40
BLS verifyMultipleSignatures 3 - blst	2.2384 ms/op	1.5085 ms/op	1.48
BLS verifyMultipleSignatures 8 - blst	2.4675 ms/op	2.2218 ms/op	1.11
BLS verifyMultipleSignatures 32 - blst	7.4301 ms/op	7.0741 ms/op	1.05
BLS verifyMultipleSignatures 64 - blst	10.596 ms/op	12.137 ms/op	0.87
BLS verifyMultipleSignatures 128 - blst	16.714 ms/op	20.810 ms/op	0.80
BLS deserializing 10000 signatures	682.46 ms/op	739.80 ms/op	0.92
BLS deserializing 100000 signatures	6.8180 s/op	7.6259 s/op	0.89
BLS verifyMultipleSignatures - same message - 3 - blst	1.2572 ms/op	1.0784 ms/op	1.17
BLS verifyMultipleSignatures - same message - 8 - blst	1.1432 ms/op	1.2278 ms/op	0.93
BLS verifyMultipleSignatures - same message - 32 - blst	1.6899 ms/op	1.7701 ms/op	0.95
BLS verifyMultipleSignatures - same message - 64 - blst	2.6680 ms/op	2.6861 ms/op	0.99
BLS verifyMultipleSignatures - same message - 128 - blst	4.4026 ms/op	4.3882 ms/op	1.00
BLS aggregatePubkeys 32 - blst	19.475 us/op	21.434 us/op	0.91
BLS aggregatePubkeys 128 - blst	68.147 us/op	76.979 us/op	0.89
notSeenSlots=1 numMissedVotes=1 numBadVotes=10	75.714 ms/op	101.51 ms/op	0.75
notSeenSlots=1 numMissedVotes=0 numBadVotes=4	68.768 ms/op	67.655 ms/op	1.02
notSeenSlots=2 numMissedVotes=1 numBadVotes=10	37.164 ms/op	40.916 ms/op	0.91
getSlashingsAndExits - default max	95.665 us/op	166.98 us/op	0.57
getSlashingsAndExits - 2k	397.85 us/op	496.12 us/op	0.80
proposeBlockBody type=full, size=empty	5.8558 ms/op	6.7414 ms/op	0.87
isKnown best case - 1 super set check	304.00 ns/op	500.00 ns/op	0.61
isKnown normal case - 2 super set checks	288.00 ns/op	535.00 ns/op	0.54
isKnown worse case - 16 super set checks	304.00 ns/op	442.00 ns/op	0.69
InMemoryCheckpointStateCache - add get delete	5.4830 us/op	6.7200 us/op	0.82
validate api signedAggregateAndProof - struct	1.8052 ms/op	1.7594 ms/op	1.03
validate gossip signedAggregateAndProof - struct	1.9093 ms/op	1.6286 ms/op	1.17
validate gossip attestation - vc 640000	1.0628 ms/op	1.1567 ms/op	0.92
batch validate gossip attestation - vc 640000 - chunk 32	129.02 us/op	141.22 us/op	0.91
batch validate gossip attestation - vc 640000 - chunk 64	111.53 us/op	125.17 us/op	0.89
batch validate gossip attestation - vc 640000 - chunk 128	109.03 us/op	114.95 us/op	0.95
batch validate gossip attestation - vc 640000 - chunk 256	108.69 us/op	117.75 us/op	0.92
pickEth1Vote - no votes	1.0222 ms/op	1.1676 ms/op	0.88
pickEth1Vote - max votes	7.6123 ms/op	12.274 ms/op	0.62
pickEth1Vote - Eth1Data hashTreeRoot value x2048	11.643 ms/op	17.013 ms/op	0.68
pickEth1Vote - Eth1Data hashTreeRoot tree x2048	19.004 ms/op	29.311 ms/op	0.65
pickEth1Vote - Eth1Data fastSerialize value x2048	467.84 us/op	558.93 us/op	0.84
pickEth1Vote - Eth1Data fastSerialize tree x2048	3.6400 ms/op	4.5172 ms/op	0.81
bytes32 toHexString	430.00 ns/op	483.00 ns/op	0.89
bytes32 Buffer.toString(hex)	240.00 ns/op	256.00 ns/op	0.94
bytes32 Buffer.toString(hex) from Uint8Array	360.00 ns/op	384.00 ns/op	0.94
bytes32 Buffer.toString(hex) + 0x	251.00 ns/op	278.00 ns/op	0.90
Object access 1 prop	0.13700 ns/op	0.14400 ns/op	0.95
Map access 1 prop	0.13100 ns/op	0.13400 ns/op	0.98
Object get x1000	5.9030 ns/op	6.0310 ns/op	0.98
Map get x1000	6.3930 ns/op	6.7980 ns/op	0.94
Object set x1000	33.167 ns/op	77.701 ns/op	0.43
Map set x1000	24.307 ns/op	51.335 ns/op	0.47
Return object 10000 times	0.28960 ns/op	0.32530 ns/op	0.89
Throw Error 10000 times	3.3706 us/op	4.0236 us/op	0.84
fastMsgIdFn sha256 / 200 bytes	2.1630 us/op	2.5380 us/op	0.85
fastMsgIdFn h32 xxhash / 200 bytes	236.00 ns/op	331.00 ns/op	0.71
fastMsgIdFn h64 xxhash / 200 bytes	270.00 ns/op	312.00 ns/op	0.87
fastMsgIdFn sha256 / 1000 bytes	7.1470 us/op	8.2050 us/op	0.87
fastMsgIdFn h32 xxhash / 1000 bytes	365.00 ns/op	423.00 ns/op	0.86
fastMsgIdFn h64 xxhash / 1000 bytes	341.00 ns/op	369.00 ns/op	0.92
fastMsgIdFn sha256 / 10000 bytes	63.363 us/op	68.185 us/op	0.93
fastMsgIdFn h32 xxhash / 10000 bytes	1.8070 us/op	1.9830 us/op	0.91
fastMsgIdFn h64 xxhash / 10000 bytes	1.1940 us/op	1.3210 us/op	0.90
send data - 1000 256B messages	12.196 ms/op	14.074 ms/op	0.87
send data - 1000 512B messages	16.340 ms/op	18.766 ms/op	0.87
send data - 1000 1024B messages	26.685 ms/op	27.920 ms/op	0.96
send data - 1000 1200B messages	26.413 ms/op	33.507 ms/op	0.79
send data - 1000 2048B messages	30.980 ms/op	36.791 ms/op	0.84
send data - 1000 4096B messages	31.351 ms/op	37.187 ms/op	0.84
send data - 1000 16384B messages	70.633 ms/op	73.300 ms/op	0.96
send data - 1000 65536B messages	216.99 ms/op	244.60 ms/op	0.89
enrSubnets - fastDeserialize 64 bits	1.0370 us/op	1.3780 us/op	0.75
enrSubnets - ssz BitVector 64 bits	349.00 ns/op	466.00 ns/op	0.75
enrSubnets - fastDeserialize 4 bits	152.00 ns/op	186.00 ns/op	0.82
enrSubnets - ssz BitVector 4 bits	343.00 ns/op	411.00 ns/op	0.83
prioritizePeers score -10:0 att 32-0.1 sync 2-0	155.68 us/op	225.14 us/op	0.69
prioritizePeers score 0:0 att 32-0.25 sync 2-0.25	139.51 us/op	189.71 us/op	0.74
prioritizePeers score 0:0 att 32-0.5 sync 2-0.5	290.91 us/op	398.79 us/op	0.73
prioritizePeers score 0:0 att 64-0.75 sync 4-0.75	531.92 us/op	595.66 us/op	0.89
prioritizePeers score 0:0 att 64-1 sync 4-1	659.40 us/op	1.1148 ms/op	0.59
array of 16000 items push then shift	1.6620 us/op	1.8577 us/op	0.89
LinkedList of 16000 items push then shift	7.2680 ns/op	8.7960 ns/op	0.83
array of 16000 items push then pop	114.98 ns/op	155.81 ns/op	0.74
LinkedList of 16000 items push then pop	6.9230 ns/op	8.2190 ns/op	0.84
array of 24000 items push then shift	2.3600 us/op	3.2274 us/op	0.73
LinkedList of 24000 items push then shift	6.9110 ns/op	17.365 ns/op	0.40
array of 24000 items push then pop	129.91 ns/op	268.90 ns/op	0.48
LinkedList of 24000 items push then pop	6.7250 ns/op	10.238 ns/op	0.66
intersect bitArray bitLen 8	6.2520 ns/op	7.8480 ns/op	0.80
intersect array and set length 8	43.939 ns/op	112.91 ns/op	0.39
intersect bitArray bitLen 128	29.073 ns/op	34.438 ns/op	0.84
intersect array and set length 128	650.22 ns/op	1.3290 us/op	0.49
bitArray.getTrueBitIndexes() bitLen 128	2.4560 us/op	2.7600 us/op	0.89
bitArray.getTrueBitIndexes() bitLen 248	3.4650 us/op	3.9640 us/op	0.87
bitArray.getTrueBitIndexes() bitLen 512	10.167 us/op	8.1840 us/op	1.24
Buffer.concat 32 items	907.00 ns/op	988.00 ns/op	0.92
Uint8Array.set 32 items	2.0360 us/op	1.6320 us/op	1.25
Buffer.copy	2.1340 us/op	1.7750 us/op	1.20
Uint8Array.set - with subarray	2.8080 us/op	2.7890 us/op	1.01
Uint8Array.set - without subarray	1.6290 us/op	2.0330 us/op	0.80
getUint32 - dataview	230.00 ns/op	315.00 ns/op	0.73
getUint32 - manual	145.00 ns/op	260.00 ns/op	0.56
Set add up to 64 items then delete first	2.1090 us/op	2.7753 us/op	0.76
OrderedSet add up to 64 items then delete first	3.1543 us/op	3.9113 us/op	0.81
Set add up to 64 items then delete last	2.3932 us/op	3.6145 us/op	0.66
OrderedSet add up to 64 items then delete last	3.7489 us/op	5.1222 us/op	0.73
Set add up to 64 items then delete middle	2.4483 us/op	2.7298 us/op	0.90
OrderedSet add up to 64 items then delete middle	5.0520 us/op	6.0286 us/op	0.84
Set add up to 128 items then delete first	4.9201 us/op	6.8297 us/op	0.72
OrderedSet add up to 128 items then delete first	7.0971 us/op	10.442 us/op	0.68
Set add up to 128 items then delete last	4.8271 us/op	5.5755 us/op	0.87
OrderedSet add up to 128 items then delete last	7.3519 us/op	9.9522 us/op	0.74
Set add up to 128 items then delete middle	4.7540 us/op	5.2590 us/op	0.90
OrderedSet add up to 128 items then delete middle	13.080 us/op	14.926 us/op	0.88
Set add up to 256 items then delete first	10.102 us/op	13.789 us/op	0.73
OrderedSet add up to 256 items then delete first	14.544 us/op	20.545 us/op	0.71
Set add up to 256 items then delete last	9.5392 us/op	12.554 us/op	0.76
OrderedSet add up to 256 items then delete last	14.909 us/op	15.758 us/op	0.95
Set add up to 256 items then delete middle	9.5571 us/op	10.667 us/op	0.90
OrderedSet add up to 256 items then delete middle	38.959 us/op	46.727 us/op	0.83
transfer serialized Status (84 B)	1.3950 us/op	1.5170 us/op	0.92
copy serialized Status (84 B)	1.1360 us/op	1.3080 us/op	0.87
transfer serialized SignedVoluntaryExit (112 B)	1.5340 us/op	1.6990 us/op	0.90
copy serialized SignedVoluntaryExit (112 B)	1.2230 us/op	1.3360 us/op	0.92
transfer serialized ProposerSlashing (416 B)	1.9780 us/op	2.1890 us/op	0.90
copy serialized ProposerSlashing (416 B)	2.0020 us/op	2.2500 us/op	0.89
transfer serialized Attestation (485 B)	2.0080 us/op	2.4170 us/op	0.83
copy serialized Attestation (485 B)	2.0440 us/op	2.2720 us/op	0.90
transfer serialized AttesterSlashing (33232 B)	2.8310 us/op	2.7010 us/op	1.05
copy serialized AttesterSlashing (33232 B)	6.0700 us/op	8.8250 us/op	0.69
transfer serialized Small SignedBeaconBlock (128000 B)	3.8330 us/op	2.8150 us/op	1.36
copy serialized Small SignedBeaconBlock (128000 B)	14.040 us/op	24.634 us/op	0.57
transfer serialized Avg SignedBeaconBlock (200000 B)	4.2260 us/op	3.6840 us/op	1.15
copy serialized Avg SignedBeaconBlock (200000 B)	20.134 us/op	25.755 us/op	0.78
transfer serialized BlobsSidecar (524380 B)	2.9910 us/op	2.6350 us/op	1.14
copy serialized BlobsSidecar (524380 B)	77.520 us/op	96.379 us/op	0.80
transfer serialized Big SignedBeaconBlock (1000000 B)	3.1340 us/op	3.8180 us/op	0.82
copy serialized Big SignedBeaconBlock (1000000 B)	159.48 us/op	183.77 us/op	0.87
pass gossip attestations to forkchoice per slot	3.2519 ms/op	3.5957 ms/op	0.90
forkChoice updateHead vc 100000 bc 64 eq 0	594.34 us/op	552.73 us/op	1.08
forkChoice updateHead vc 600000 bc 64 eq 0	3.0001 ms/op	3.2941 ms/op	0.91
forkChoice updateHead vc 1000000 bc 64 eq 0	5.1774 ms/op	6.1188 ms/op	0.85
forkChoice updateHead vc 600000 bc 320 eq 0	3.0030 ms/op	3.3470 ms/op	0.90
forkChoice updateHead vc 600000 bc 1200 eq 0	3.0512 ms/op	4.8470 ms/op	0.63
forkChoice updateHead vc 600000 bc 7200 eq 0	3.9113 ms/op	4.5896 ms/op	0.85
forkChoice updateHead vc 600000 bc 64 eq 1000	10.500 ms/op	11.543 ms/op	0.91
forkChoice updateHead vc 600000 bc 64 eq 10000	10.647 ms/op	11.883 ms/op	0.90
forkChoice updateHead vc 600000 bc 64 eq 300000	13.843 ms/op	23.222 ms/op	0.60
computeDeltas 500000 validators 300 proto nodes	3.4649 ms/op	4.0673 ms/op	0.85
computeDeltas 500000 validators 1200 proto nodes	3.4483 ms/op	4.2413 ms/op	0.81
computeDeltas 500000 validators 7200 proto nodes	3.4200 ms/op	4.6100 ms/op	0.74
computeDeltas 750000 validators 300 proto nodes	5.1447 ms/op	6.7896 ms/op	0.76
computeDeltas 750000 validators 1200 proto nodes	5.2273 ms/op	6.7521 ms/op	0.77
computeDeltas 750000 validators 7200 proto nodes	5.1799 ms/op	6.3722 ms/op	0.81
computeDeltas 1400000 validators 300 proto nodes	9.6368 ms/op	12.043 ms/op	0.80
computeDeltas 1400000 validators 1200 proto nodes	9.5534 ms/op	11.148 ms/op	0.86
computeDeltas 1400000 validators 7200 proto nodes	9.7561 ms/op	11.517 ms/op	0.85
computeDeltas 2100000 validators 300 proto nodes	14.647 ms/op	15.815 ms/op	0.93
computeDeltas 2100000 validators 1200 proto nodes	14.788 ms/op	15.677 ms/op	0.94
computeDeltas 2100000 validators 7200 proto nodes	14.860 ms/op	15.222 ms/op	0.98
altair processAttestation - 250000 vs - 7PWei normalcase	1.5750 ms/op	1.7476 ms/op	0.90
altair processAttestation - 250000 vs - 7PWei worstcase	2.3775 ms/op	2.6841 ms/op	0.89
altair processAttestation - setStatus - 1/6 committees join	87.642 us/op	92.773 us/op	0.94
altair processAttestation - setStatus - 1/3 committees join	178.88 us/op	180.33 us/op	0.99
altair processAttestation - setStatus - 1/2 committees join	247.18 us/op	265.68 us/op	0.93
altair processAttestation - setStatus - 2/3 committees join	332.73 us/op	333.22 us/op	1.00
altair processAttestation - setStatus - 4/5 committees join	496.10 us/op	507.92 us/op	0.98
altair processAttestation - setStatus - 100% committees join	586.84 us/op	604.03 us/op	0.97
altair processBlock - 250000 vs - 7PWei normalcase	5.1146 ms/op	4.5631 ms/op	1.12
altair processBlock - 250000 vs - 7PWei normalcase hashState	24.950 ms/op	32.407 ms/op	0.77
altair processBlock - 250000 vs - 7PWei worstcase	39.152 ms/op	39.460 ms/op	0.99
altair processBlock - 250000 vs - 7PWei worstcase hashState	83.361 ms/op	85.532 ms/op	0.97
phase0 processBlock - 250000 vs - 7PWei normalcase	2.8043 ms/op	2.7890 ms/op	1.01
phase0 processBlock - 250000 vs - 7PWei worstcase	31.735 ms/op	28.085 ms/op	1.13
altair processEth1Data - 250000 vs - 7PWei normalcase	308.31 us/op	467.26 us/op	0.66
getExpectedWithdrawals 250000 eb:1,eth1:1,we:0,wn:0,smpl:15	8.0680 us/op	7.1460 us/op	1.13
getExpectedWithdrawals 250000 eb:0.95,eth1:0.1,we:0.05,wn:0,smpl:219	30.391 us/op	30.398 us/op	1.00
getExpectedWithdrawals 250000 eb:0.95,eth1:0.3,we:0.05,wn:0,smpl:42	11.093 us/op	13.232 us/op	0.84
getExpectedWithdrawals 250000 eb:0.95,eth1:0.7,we:0.05,wn:0,smpl:18	8.7520 us/op	8.4650 us/op	1.03
getExpectedWithdrawals 250000 eb:0.1,eth1:0.1,we:0,wn:0,smpl:1020	70.060 us/op	119.36 us/op	0.59
getExpectedWithdrawals 250000 eb:0.03,eth1:0.03,we:0,wn:0,smpl:11777	1.0778 ms/op	916.20 us/op	1.18
getExpectedWithdrawals 250000 eb:0.01,eth1:0.01,we:0,wn:0,smpl:16384	1.5569 ms/op	1.2645 ms/op	1.23
getExpectedWithdrawals 250000 eb:0,eth1:0,we:0,wn:0,smpl:16384	904.86 us/op	1.1508 ms/op	0.79
getExpectedWithdrawals 250000 eb:0,eth1:0,we:0,wn:0,nocache,smpl:16384	2.4844 ms/op	3.0927 ms/op	0.80
getExpectedWithdrawals 250000 eb:0,eth1:1,we:0,wn:0,smpl:16384	1.5291 ms/op	2.1243 ms/op	0.72
getExpectedWithdrawals 250000 eb:0,eth1:1,we:0,wn:0,nocache,smpl:16384	3.8221 ms/op	4.8473 ms/op	0.79
Tree 40 250000 create	203.04 ms/op	483.00 ms/op	0.42
Tree 40 250000 get(125000)	140.51 ns/op	179.48 ns/op	0.78
Tree 40 250000 set(125000)	627.13 ns/op	1.6828 us/op	0.37
Tree 40 250000 toArray()	15.328 ms/op	25.753 ms/op	0.60
Tree 40 250000 iterate all - toArray() + loop	15.475 ms/op	26.396 ms/op	0.59
Tree 40 250000 iterate all - get(i)	50.189 ms/op	74.884 ms/op	0.67
MutableVector 250000 create	10.827 ms/op	16.958 ms/op	0.64
MutableVector 250000 get(125000)	6.5120 ns/op	7.9280 ns/op	0.82
MutableVector 250000 set(125000)	218.74 ns/op	300.55 ns/op	0.73
MutableVector 250000 toArray()	3.8412 ms/op	5.0849 ms/op	0.76
MutableVector 250000 iterate all - toArray() + loop	3.7757 ms/op	5.4807 ms/op	0.69
MutableVector 250000 iterate all - get(i)	1.5639 ms/op	1.7584 ms/op	0.89
Array 250000 create	2.9853 ms/op	4.8963 ms/op	0.61
Array 250000 clone - spread	1.5255 ms/op	3.6722 ms/op	0.42
Array 250000 get(125000)	0.39200 ns/op	0.51300 ns/op	0.76
Array 250000 set(125000)	0.41800 ns/op	0.53800 ns/op	0.78
Array 250000 iterate all - loop	106.08 us/op	123.48 us/op	0.86
effectiveBalanceIncrements clone Uint8Array 300000	39.044 us/op	78.460 us/op	0.50
effectiveBalanceIncrements clone MutableVector 300000	124.00 ns/op	167.00 ns/op	0.74
effectiveBalanceIncrements rw all Uint8Array 300000	195.72 us/op	221.00 us/op	0.89
effectiveBalanceIncrements rw all MutableVector 300000	70.850 ms/op	179.53 ms/op	0.39
phase0 afterProcessEpoch - 250000 vs - 7PWei	87.138 ms/op	99.330 ms/op	0.88
Array.fill - length 1000000	3.4439 ms/op	7.9360 ms/op	0.43
Array push - length 1000000	21.190 ms/op	26.478 ms/op	0.80
Array.get	0.28615 ns/op	0.31746 ns/op	0.90
Uint8Array.get	0.46409 ns/op	0.49242 ns/op	0.94
phase0 beforeProcessEpoch - 250000 vs - 7PWei	27.160 ms/op	30.087 ms/op	0.90
altair processEpoch - mainnet_e81889	425.38 ms/op	385.46 ms/op	1.10
mainnet_e81889 - altair beforeProcessEpoch	38.605 ms/op	36.538 ms/op	1.06
mainnet_e81889 - altair processJustificationAndFinalization	13.293 us/op	18.921 us/op	0.70
mainnet_e81889 - altair processInactivityUpdates	7.1656 ms/op	6.3351 ms/op	1.13
mainnet_e81889 - altair processRewardsAndPenalties	48.096 ms/op	56.362 ms/op	0.85
mainnet_e81889 - altair processRegistryUpdates	2.3470 us/op	3.3880 us/op	0.69
mainnet_e81889 - altair processSlashings	1.0980 us/op	847.00 ns/op	1.30
mainnet_e81889 - altair processEth1DataReset	930.00 ns/op	732.00 ns/op	1.27
mainnet_e81889 - altair processEffectiveBalanceUpdates	2.1365 ms/op	2.3132 ms/op	0.92
mainnet_e81889 - altair processSlashingsReset	2.9310 us/op	8.3150 us/op	0.35
mainnet_e81889 - altair processRandaoMixesReset	3.6070 us/op	8.5160 us/op	0.42
mainnet_e81889 - altair processHistoricalRootsUpdate	411.00 ns/op	1.2700 us/op	0.32
mainnet_e81889 - altair processParticipationFlagUpdates	2.5380 us/op	3.6310 us/op	0.70
mainnet_e81889 - altair processSyncCommitteeUpdates	520.00 ns/op	1.0030 us/op	0.52
mainnet_e81889 - altair afterProcessEpoch	92.329 ms/op	106.20 ms/op	0.87
capella processEpoch - mainnet_e217614	1.1733 s/op	1.9567 s/op	0.60
mainnet_e217614 - capella beforeProcessEpoch	135.51 ms/op	174.03 ms/op	0.78
mainnet_e217614 - capella processJustificationAndFinalization	23.774 us/op	32.672 us/op	0.73
mainnet_e217614 - capella processInactivityUpdates	18.297 ms/op	23.799 ms/op	0.77
mainnet_e217614 - capella processRewardsAndPenalties	259.80 ms/op	293.13 ms/op	0.89
mainnet_e217614 - capella processRegistryUpdates	18.942 us/op	22.909 us/op	0.83
mainnet_e217614 - capella processSlashings	387.00 ns/op	821.00 ns/op	0.47
mainnet_e217614 - capella processEth1DataReset	320.00 ns/op	1.2620 us/op	0.25
mainnet_e217614 - capella processEffectiveBalanceUpdates	12.707 ms/op	25.334 ms/op	0.50
mainnet_e217614 - capella processSlashingsReset	5.3460 us/op	8.8370 us/op	0.60
mainnet_e217614 - capella processRandaoMixesReset	6.8520 us/op	14.103 us/op	0.49
mainnet_e217614 - capella processHistoricalRootsUpdate	586.00 ns/op	1.2750 us/op	0.46
mainnet_e217614 - capella processParticipationFlagUpdates	2.3660 us/op	8.0520 us/op	0.29
mainnet_e217614 - capella afterProcessEpoch	274.58 ms/op	412.89 ms/op	0.67
phase0 processEpoch - mainnet_e58758	435.24 ms/op	800.70 ms/op	0.54
mainnet_e58758 - phase0 beforeProcessEpoch	110.77 ms/op	218.61 ms/op	0.51
mainnet_e58758 - phase0 processJustificationAndFinalization	21.505 us/op	41.852 us/op	0.51
mainnet_e58758 - phase0 processRewardsAndPenalties	39.973 ms/op	58.574 ms/op	0.68
mainnet_e58758 - phase0 processRegistryUpdates	10.091 us/op	20.839 us/op	0.48
mainnet_e58758 - phase0 processSlashings	562.00 ns/op	1.3770 us/op	0.41
mainnet_e58758 - phase0 processEth1DataReset	319.00 ns/op	1.3080 us/op	0.24
mainnet_e58758 - phase0 processEffectiveBalanceUpdates	1.0982 ms/op	2.0452 ms/op	0.54
mainnet_e58758 - phase0 processSlashingsReset	4.0760 us/op	7.3930 us/op	0.55
mainnet_e58758 - phase0 processRandaoMixesReset	6.3760 us/op	13.948 us/op	0.46
mainnet_e58758 - phase0 processHistoricalRootsUpdate	413.00 ns/op	1.1550 us/op	0.36
mainnet_e58758 - phase0 processParticipationRecordUpdates	3.5730 us/op	20.240 us/op	0.18
mainnet_e58758 - phase0 afterProcessEpoch	78.206 ms/op	137.32 ms/op	0.57
phase0 processEffectiveBalanceUpdates - 250000 normalcase	1.2082 ms/op	2.0470 ms/op	0.59
phase0 processEffectiveBalanceUpdates - 250000 worstcase 0.5	2.6702 ms/op	7.7077 ms/op	0.35
altair processInactivityUpdates - 250000 normalcase	21.794 ms/op	29.936 ms/op	0.73
altair processInactivityUpdates - 250000 worstcase	21.671 ms/op	34.112 ms/op	0.64
phase0 processRegistryUpdates - 250000 normalcase	8.0150 us/op	20.105 us/op	0.40
phase0 processRegistryUpdates - 250000 badcase_full_deposits	368.82 us/op	507.66 us/op	0.73
phase0 processRegistryUpdates - 250000 worstcase 0.5	128.68 ms/op	168.90 ms/op	0.76
altair processRewardsAndPenalties - 250000 normalcase	48.016 ms/op	46.280 ms/op	1.04
altair processRewardsAndPenalties - 250000 worstcase	42.888 ms/op	46.715 ms/op	0.92
phase0 getAttestationDeltas - 250000 normalcase	8.1329 ms/op	12.549 ms/op	0.65
phase0 getAttestationDeltas - 250000 worstcase	8.4655 ms/op	9.9824 ms/op	0.85
phase0 processSlashings - 250000 worstcase	121.55 us/op	110.95 us/op	1.10
altair processSyncCommitteeUpdates - 250000	116.36 ms/op	146.44 ms/op	0.79
BeaconState.hashTreeRoot - No change	266.00 ns/op	358.00 ns/op	0.74
BeaconState.hashTreeRoot - 1 full validator	114.02 us/op	129.24 us/op	0.88
BeaconState.hashTreeRoot - 32 full validator	1.1610 ms/op	1.7672 ms/op	0.66
BeaconState.hashTreeRoot - 512 full validator	13.625 ms/op	15.259 ms/op	0.89
BeaconState.hashTreeRoot - 1 validator.effectiveBalance	155.85 us/op	189.02 us/op	0.82
BeaconState.hashTreeRoot - 32 validator.effectiveBalance	2.2020 ms/op	2.7816 ms/op	0.79
BeaconState.hashTreeRoot - 512 validator.effectiveBalance	30.107 ms/op	34.706 ms/op	0.87
BeaconState.hashTreeRoot - 1 balances	127.06 us/op	140.65 us/op	0.90
BeaconState.hashTreeRoot - 32 balances	1.1569 ms/op	1.5437 ms/op	0.75
BeaconState.hashTreeRoot - 512 balances	10.029 ms/op	10.871 ms/op	0.92
BeaconState.hashTreeRoot - 250000 balances	173.14 ms/op	164.90 ms/op	1.05
aggregationBits - 2048 els - zipIndexesInBitList	23.682 us/op	28.413 us/op	0.83
byteArrayEquals 32	52.092 ns/op	56.290 ns/op	0.93
Buffer.compare 32	19.618 ns/op	17.995 ns/op	1.09
byteArrayEquals 1024	1.5541 us/op	1.6530 us/op	0.94
Buffer.compare 1024	27.173 ns/op	26.526 ns/op	1.02
byteArrayEquals 16384	24.675 us/op	26.347 us/op	0.94
Buffer.compare 16384	191.67 ns/op	189.44 ns/op	1.01
byteArrayEquals 123687377	184.12 ms/op	212.68 ms/op	0.87
Buffer.compare 123687377	6.5002 ms/op	12.212 ms/op	0.53
byteArrayEquals 32 - diff last byte	51.920 ns/op	59.934 ns/op	0.87
Buffer.compare 32 - diff last byte	17.071 ns/op	19.879 ns/op	0.86
byteArrayEquals 1024 - diff last byte	1.5742 us/op	1.8055 us/op	0.87
Buffer.compare 1024 - diff last byte	25.138 ns/op	29.804 ns/op	0.84
byteArrayEquals 16384 - diff last byte	25.026 us/op	28.535 us/op	0.88
Buffer.compare 16384 - diff last byte	197.98 ns/op	227.59 ns/op	0.87
byteArrayEquals 123687377 - diff last byte	188.18 ms/op	221.27 ms/op	0.85
Buffer.compare 123687377 - diff last byte	6.0954 ms/op	11.603 ms/op	0.53
byteArrayEquals 32 - random bytes	4.9150 ns/op	5.9520 ns/op	0.83
Buffer.compare 32 - random bytes	16.499 ns/op	20.094 ns/op	0.82
byteArrayEquals 1024 - random bytes	4.9420 ns/op	5.8630 ns/op	0.84
Buffer.compare 1024 - random bytes	16.450 ns/op	22.471 ns/op	0.73
byteArrayEquals 16384 - random bytes	4.9140 ns/op	6.1990 ns/op	0.79
Buffer.compare 16384 - random bytes	16.513 ns/op	19.813 ns/op	0.83
byteArrayEquals 123687377 - random bytes	6.0600 ns/op	7.8500 ns/op	0.77
Buffer.compare 123687377 - random bytes	17.640 ns/op	22.730 ns/op	0.78
regular array get 100000 times	31.545 us/op	41.934 us/op	0.75
wrappedArray get 100000 times	31.370 us/op	41.472 us/op	0.76
arrayWithProxy get 100000 times	13.210 ms/op	19.437 ms/op	0.68
ssz.Root.equals	45.581 ns/op	55.656 ns/op	0.82
byteArrayEquals	44.956 ns/op	54.337 ns/op	0.83
Buffer.compare	10.250 ns/op	12.061 ns/op	0.85
shuffle list - 16384 els	6.2957 ms/op	7.3299 ms/op	0.86
shuffle list - 250000 els	92.077 ms/op	112.99 ms/op	0.81
processSlot - 1 slots	14.254 us/op	22.444 us/op	0.64
processSlot - 32 slots	3.2231 ms/op	4.3905 ms/op	0.73
getEffectiveBalanceIncrementsZeroInactive - 250000 vs - 7PWei	42.216 ms/op	54.205 ms/op	0.78
getCommitteeAssignments - req 1 vs - 250000 vc	2.1382 ms/op	2.5613 ms/op	0.83
getCommitteeAssignments - req 100 vs - 250000 vc	4.1095 ms/op	5.3312 ms/op	0.77
getCommitteeAssignments - req 1000 vs - 250000 vc	4.3760 ms/op	5.2454 ms/op	0.83
findModifiedValidators - 10000 modified validators	245.48 ms/op	625.53 ms/op	0.39
findModifiedValidators - 1000 modified validators	160.38 ms/op	450.90 ms/op	0.36
findModifiedValidators - 100 modified validators	159.12 ms/op	345.77 ms/op	0.46
findModifiedValidators - 10 modified validators	134.45 ms/op	286.71 ms/op	0.47
findModifiedValidators - 1 modified validators	141.64 ms/op	303.21 ms/op	0.47
findModifiedValidators - no difference	146.10 ms/op	324.18 ms/op	0.45
compare ViewDUs	3.1927 s/op	4.3672 s/op	0.73
compare each validator Uint8Array	1.5798 s/op	2.2950 s/op	0.69
compare ViewDU to Uint8Array	948.44 ms/op	1.6281 s/op	0.58
migrate state 1000000 validators, 24 modified, 0 new	553.89 ms/op	683.43 ms/op	0.81
migrate state 1000000 validators, 1700 modified, 1000 new	894.03 ms/op	1.0534 s/op	0.85
migrate state 1000000 validators, 3400 modified, 2000 new	1.1205 s/op	1.2155 s/op	0.92
migrate state 1500000 validators, 24 modified, 0 new	659.84 ms/op	762.66 ms/op	0.87
migrate state 1500000 validators, 1700 modified, 1000 new	864.49 ms/op	832.10 ms/op	1.04
migrate state 1500000 validators, 3400 modified, 2000 new	1.0142 s/op	1.0789 s/op	0.94
RootCache.getBlockRootAtSlot - 250000 vs - 7PWei	4.8400 ns/op	4.7300 ns/op	1.02
state getBlockRootAtSlot - 250000 vs - 7PWei	932.36 ns/op	949.94 ns/op	0.98
computeProposers - vc 250000	9.2434 ms/op	7.6173 ms/op	1.21
computeEpochShuffling - vc 250000	98.265 ms/op	96.464 ms/op	1.02
getNextSyncCommittee - vc 250000	143.67 ms/op	123.15 ms/op	1.17
computeSigningRoot for AttestationData	21.426 us/op	24.217 us/op	0.88
hash AttestationData serialized data then Buffer.toString(base64)	1.6382 us/op	1.5446 us/op	1.06
toHexString serialized data	1.0419 us/op	927.19 ns/op	1.12
Buffer.toString(base64)	220.96 ns/op	194.50 ns/op	1.14

by benchmarkbot/action

[wemeetagain]

Running into issues opening the db in the worker thread.

It seems the leveldb way of allowing concurrency is have the consumer pass the pointer to the db handle to another thread. That's not currently possible with the leveldb package as it currently exists.

[wemeetagain]

Converting to draft until the multithreaded database access issue is resolved

[codecov[bot]]

Codecov Report

Attention: Patch coverage is 50.00000% with 5 lines in your changes missing coverage. Please review.

Project coverage is 62.49%. Comparing base (27012f9) to head (e8f4d97). Report is 6 commits behind head on unstable.

Additional details and impacted files

@@             Coverage Diff              @@
##           unstable    #6033      +/-   ##
============================================
- Coverage     62.49%   62.49%   -0.01%     
============================================
  Files           576      576              
  Lines         61180    61176       -4     
  Branches       2141     2134       -7     
============================================
- Hits          38236    38232       -4     
  Misses        22905    22905              
  Partials         39       39              

[philknows]

Any update on the status of this PR? Are we blocked on it somewhere externally?

[wemeetagain]

Any update on the status of this PR?

waiting on me to fix this in response to @matthewkeil 's review

[wemeetagain]

Ready for another round of review

[twoeths]

this PR implements the fundamental work for historical state regen on worker thread, however I'm not sure how likely it could be fully functional in production, a request could take so long given archiveStateEpochFrequency = 1024, it could be up to 1024 * 10s ~= 2.8h given ~10s to process 32 blocks of an epoch like what I saw in n-historical state testing

we could explore storing incremental state diff with 2 purposes:

• save disc space
• faster time to restart
• this historical state regen: apply some diffs until the checkpoint state, and reprocess blocks like in this PR

cc @nazarhussain @philknows @wemeetagain

[nazarhussain]

we could explore storing incremental state diff with 2 purposes:

That is already in progress and will be complete very soon.

[wemeetagain]

:tada: This PR is included in v1.21.0 :tada: