Drop lerna

[dapplion]

Motivation

lerna has been used in this monorepo for +3y (added in https://github.com/ChainSafe/lodestar/pull/324). While useful we have had to battle this tool more than treat it like a helpful item:

• https://github.com/ChainSafe/lodestar/pull/4052
• https://github.com/ChainSafe/lodestar/pull/4049
• https://github.com/ChainSafe/lodestar/pull/3990
• https://github.com/ChainSafe/lodestar/pull/3679
• https://github.com/ChainSafe/lodestar/pull/3637
• https://github.com/ChainSafe/lodestar/pull/3601
• https://github.com/ChainSafe/lodestar/pull/2656
• https://github.com/ChainSafe/lodestar/pull/2650
• https://github.com/ChainSafe/lodestar/pull/2649
• https://github.com/ChainSafe/lodestar/pull/1909
• https://github.com/ChainSafe/lodestar/pull/666

It also deprecates scripts introduced here, not used currently AFAIK

• https://github.com/ChainSafe/lodestar/pull/1142

Description

We use lerna for three actions:

• Run scripts in all packages: Now done yarn workspaces run <cmd>
• Bump version and dependencies of local packages: Now done with scripts/release/set_version.mjs
• Publish all packages: Now done scripts/release/publish.mjs

Please check the new scripts and you'll see they are really simple. I don't think they justify a tool so heavy and problematic like lerna. I don't guarantee this PR won't introduce bugs for each type of release, but fixing those should be trivial now that we control the tooling.

Please note that we have dropped 2_300 lines of dependencies from yarn.lock :tada:

[github-actions[bot]]

Performance Report

✔️ no performance regression detected

Full benchmark results

Benchmark suite	Current: 6cde8699893e02c4172be7626c45132aa0ffa16b	Previous: 57a2fae21fa44a289a7ac48ee8b651fb44bffbf9	Ratio
getPubkeys - index2pubkey - req 1000 vs - 250000 vc	2.1032 ms/op	2.2887 ms/op	0.92
getPubkeys - validatorsArr - req 1000 vs - 250000 vc	73.902 us/op	66.083 us/op	1.12
BLS verify - blst-native	1.8549 ms/op	2.1682 ms/op	0.86
BLS verifyMultipleSignatures 3 - blst-native	3.8007 ms/op	4.4704 ms/op	0.85
BLS verifyMultipleSignatures 8 - blst-native	8.1903 ms/op	9.6696 ms/op	0.85
BLS verifyMultipleSignatures 32 - blst-native	29.688 ms/op	35.198 ms/op	0.84
BLS aggregatePubkeys 32 - blst-native	39.451 us/op	46.502 us/op	0.85
BLS aggregatePubkeys 128 - blst-native	153.26 us/op	182.00 us/op	0.84
getAttestationsForBlock	43.424 ms/op	42.874 ms/op	1.01
isKnown best case - 1 super set check	433.00 ns/op	486.00 ns/op	0.89
isKnown normal case - 2 super set checks	419.00 ns/op	471.00 ns/op	0.89
isKnown worse case - 16 super set checks	431.00 ns/op	467.00 ns/op	0.92
CheckpointStateCache - add get delete	10.395 us/op	11.200 us/op	0.93
validate gossip signedAggregateAndProof - struct	4.2511 ms/op	5.0284 ms/op	0.85
validate gossip attestation - struct	2.0178 ms/op	2.3758 ms/op	0.85
altair verifyImport mainnet_s3766816:31	6.3408 s/op	7.1414 s/op	0.89
pickEth1Vote - no votes	2.0971 ms/op	2.1695 ms/op	0.97
pickEth1Vote - max votes	24.750 ms/op	26.037 ms/op	0.95
pickEth1Vote - Eth1Data hashTreeRoot value x2048	11.565 ms/op	13.240 ms/op	0.87
pickEth1Vote - Eth1Data hashTreeRoot tree x2048	20.778 ms/op	22.045 ms/op	0.94
pickEth1Vote - Eth1Data fastSerialize value x2048	1.5253 ms/op	1.4378 ms/op	1.06
pickEth1Vote - Eth1Data fastSerialize tree x2048	18.806 ms/op	14.530 ms/op	1.29
bytes32 toHexString	1.0580 us/op	982.00 ns/op	1.08
bytes32 Buffer.toString(hex)	669.00 ns/op	749.00 ns/op	0.89
bytes32 Buffer.toString(hex) from Uint8Array	867.00 ns/op	993.00 ns/op	0.87
bytes32 Buffer.toString(hex) + 0x	668.00 ns/op	753.00 ns/op	0.89
Object access 1 prop	0.35000 ns/op	0.35700 ns/op	0.98
Map access 1 prop	0.30400 ns/op	0.30700 ns/op	0.99
Object get x1000	18.346 ns/op	11.731 ns/op	1.56
Map get x1000	0.99400 ns/op	0.93800 ns/op	1.06
Object set x1000	121.20 ns/op	69.959 ns/op	1.73
Map set x1000	70.307 ns/op	49.895 ns/op	1.41
Return object 10000 times	0.37740 ns/op	0.43760 ns/op	0.86
Throw Error 10000 times	6.0419 us/op	6.0709 us/op	1.00
enrSubnets - fastDeserialize 64 bits	2.7170 us/op	2.6510 us/op	1.02
enrSubnets - ssz BitVector 64 bits	744.00 ns/op	854.00 ns/op	0.87
enrSubnets - fastDeserialize 4 bits	401.00 ns/op	387.00 ns/op	1.04
enrSubnets - ssz BitVector 4 bits	762.00 ns/op	855.00 ns/op	0.89
prioritizePeers score -10:0 att 32-0.1 sync 2-0	93.521 us/op	86.305 us/op	1.08
prioritizePeers score 0:0 att 32-0.25 sync 2-0.25	135.45 us/op	129.62 us/op	1.04
prioritizePeers score 0:0 att 32-0.5 sync 2-0.5	214.79 us/op	211.43 us/op	1.02
prioritizePeers score 0:0 att 64-0.75 sync 4-0.75	393.57 us/op	338.81 us/op	1.16
prioritizePeers score 0:0 att 64-1 sync 4-1	459.34 us/op	406.91 us/op	1.13
RateTracker 1000000 limit, 1 obj count per request	183.96 ns/op	183.85 ns/op	1.00
RateTracker 1000000 limit, 2 obj count per request	137.29 ns/op	135.45 ns/op	1.01
RateTracker 1000000 limit, 4 obj count per request	115.54 ns/op	112.82 ns/op	1.02
RateTracker 1000000 limit, 8 obj count per request	103.07 ns/op	98.425 ns/op	1.05
RateTracker with prune	4.2550 us/op	4.5560 us/op	0.93
array of 16000 items push then shift	3.1662 us/op	51.593 us/op	0.06
LinkedList of 16000 items push then shift	25.407 ns/op	18.103 ns/op	1.40
array of 16000 items push then pop	232.66 ns/op	209.52 ns/op	1.11
LinkedList of 16000 items push then pop	21.183 ns/op	14.491 ns/op	1.46
array of 24000 items push then shift	4.5551 us/op	77.369 us/op	0.06
LinkedList of 24000 items push then shift	24.889 ns/op	21.465 ns/op	1.16
array of 24000 items push then pop	184.21 ns/op	194.19 ns/op	0.95
LinkedList of 24000 items push then pop	21.379 ns/op	15.771 ns/op	1.36
intersect bitArray bitLen 8	11.742 ns/op	12.191 ns/op	0.96
intersect array and set length 8	161.43 ns/op	139.04 ns/op	1.16
intersect bitArray bitLen 128	62.089 ns/op	57.874 ns/op	1.07
intersect array and set length 128	2.2030 us/op	1.7836 us/op	1.24
pass gossip attestations to forkchoice per slot	7.2690 ms/op	4.3268 ms/op	1.68
computeDeltas	3.3631 ms/op	3.4747 ms/op	0.97
computeProposerBoostScoreFromBalances	907.47 us/op	804.04 us/op	1.13
altair processAttestation - 250000 vs - 7PWei normalcase	3.9294 ms/op	3.7565 ms/op	1.05
altair processAttestation - 250000 vs - 7PWei worstcase	5.9365 ms/op	5.6336 ms/op	1.05
altair processAttestation - setStatus - 1/6 committees join	204.87 us/op	178.16 us/op	1.15
altair processAttestation - setStatus - 1/3 committees join	392.45 us/op	343.63 us/op	1.14
altair processAttestation - setStatus - 1/2 committees join	547.27 us/op	492.44 us/op	1.11
altair processAttestation - setStatus - 2/3 committees join	709.31 us/op	652.95 us/op	1.09
altair processAttestation - setStatus - 4/5 committees join	969.57 us/op	926.70 us/op	1.05
altair processAttestation - setStatus - 100% committees join	1.1485 ms/op	1.0926 ms/op	1.05
altair processBlock - 250000 vs - 7PWei normalcase	26.965 ms/op	29.097 ms/op	0.93
altair processBlock - 250000 vs - 7PWei normalcase hashState	38.946 ms/op	36.714 ms/op	1.06
altair processBlock - 250000 vs - 7PWei worstcase	77.340 ms/op	83.739 ms/op	0.92
altair processBlock - 250000 vs - 7PWei worstcase hashState	99.782 ms/op	113.06 ms/op	0.88
phase0 processBlock - 250000 vs - 7PWei normalcase	3.7117 ms/op	3.4870 ms/op	1.06
phase0 processBlock - 250000 vs - 7PWei worstcase	46.008 ms/op	52.470 ms/op	0.88
altair processEth1Data - 250000 vs - 7PWei normalcase	913.71 us/op	773.70 us/op	1.18
Tree 40 250000 create	843.00 ms/op	804.74 ms/op	1.05
Tree 40 250000 get(125000)	292.07 ns/op	270.70 ns/op	1.08
Tree 40 250000 set(125000)	2.9472 us/op	2.2555 us/op	1.31
Tree 40 250000 toArray()	36.598 ms/op	28.752 ms/op	1.27
Tree 40 250000 iterate all - toArray() + loop	32.878 ms/op	28.916 ms/op	1.14
Tree 40 250000 iterate all - get(i)	111.62 ms/op	115.77 ms/op	0.96
MutableVector 250000 create	12.056 ms/op	14.673 ms/op	0.82
MutableVector 250000 get(125000)	13.221 ns/op	11.242 ns/op	1.18
MutableVector 250000 set(125000)	592.56 ns/op	597.85 ns/op	0.99
MutableVector 250000 toArray()	6.7084 ms/op	6.9360 ms/op	0.97
MutableVector 250000 iterate all - toArray() + loop	7.0226 ms/op	6.9668 ms/op	1.01
MutableVector 250000 iterate all - get(i)	3.4446 ms/op	2.7163 ms/op	1.27
Array 250000 create	5.9346 ms/op	6.8134 ms/op	0.87
Array 250000 clone - spread	3.1463 ms/op	4.2276 ms/op	0.74
Array 250000 get(125000)	1.3980 ns/op	1.6980 ns/op	0.82
Array 250000 set(125000)	1.4000 ns/op	1.6180 ns/op	0.87
Array 250000 iterate all - loop	167.83 us/op	152.36 us/op	1.10
effectiveBalanceIncrements clone Uint8Array 300000	79.117 us/op	119.65 us/op	0.66
effectiveBalanceIncrements clone MutableVector 300000	1.0730 us/op	643.00 ns/op	1.67
effectiveBalanceIncrements rw all Uint8Array 300000	252.55 us/op	247.45 us/op	1.02
effectiveBalanceIncrements rw all MutableVector 300000	202.14 ms/op	142.41 ms/op	1.42
phase0 afterProcessEpoch - 250000 vs - 7PWei	185.80 ms/op	186.34 ms/op	1.00
phase0 beforeProcessEpoch - 250000 vs - 7PWei	91.510 ms/op	59.467 ms/op	1.54
altair processEpoch - mainnet_e81889	501.13 ms/op	567.69 ms/op	0.88
mainnet_e81889 - altair beforeProcessEpoch	146.09 ms/op	130.39 ms/op	1.12
mainnet_e81889 - altair processJustificationAndFinalization	19.807 us/op	15.885 us/op	1.25
mainnet_e81889 - altair processInactivityUpdates	11.292 ms/op	9.4409 ms/op	1.20
mainnet_e81889 - altair processRewardsAndPenalties	92.856 ms/op	129.85 ms/op	0.72
mainnet_e81889 - altair processRegistryUpdates	5.2430 us/op	2.5980 us/op	2.02
mainnet_e81889 - altair processSlashings	1.0020 us/op	658.00 ns/op	1.52
mainnet_e81889 - altair processEth1DataReset	770.00 ns/op	633.00 ns/op	1.22
mainnet_e81889 - altair processEffectiveBalanceUpdates	2.2543 ms/op	2.3422 ms/op	0.96
mainnet_e81889 - altair processSlashingsReset	5.1380 us/op	4.0390 us/op	1.27
mainnet_e81889 - altair processRandaoMixesReset	9.5610 us/op	3.8010 us/op	2.52
mainnet_e81889 - altair processHistoricalRootsUpdate	1.3890 us/op	564.00 ns/op	2.46
mainnet_e81889 - altair processParticipationFlagUpdates	4.2960 us/op	1.9470 us/op	2.21
mainnet_e81889 - altair processSyncCommitteeUpdates	998.00 ns/op	521.00 ns/op	1.92
mainnet_e81889 - altair afterProcessEpoch	195.44 ms/op	195.42 ms/op	1.00
phase0 processEpoch - mainnet_e58758	522.16 ms/op	498.91 ms/op	1.05
mainnet_e58758 - phase0 beforeProcessEpoch	232.61 ms/op	186.50 ms/op	1.25
mainnet_e58758 - phase0 processJustificationAndFinalization	20.678 us/op	17.312 us/op	1.19
mainnet_e58758 - phase0 processRewardsAndPenalties	145.82 ms/op	114.81 ms/op	1.27
mainnet_e58758 - phase0 processRegistryUpdates	9.5090 us/op	7.7600 us/op	1.23
mainnet_e58758 - phase0 processSlashings	748.00 ns/op	565.00 ns/op	1.32
mainnet_e58758 - phase0 processEth1DataReset	817.00 ns/op	623.00 ns/op	1.31
mainnet_e58758 - phase0 processEffectiveBalanceUpdates	2.1471 ms/op	2.1416 ms/op	1.00
mainnet_e58758 - phase0 processSlashingsReset	4.7330 us/op	3.7040 us/op	1.28
mainnet_e58758 - phase0 processRandaoMixesReset	5.6530 us/op	3.9060 us/op	1.45
mainnet_e58758 - phase0 processHistoricalRootsUpdate	1.2400 us/op	603.00 ns/op	2.06
mainnet_e58758 - phase0 processParticipationRecordUpdates	5.6150 us/op	3.4610 us/op	1.62
mainnet_e58758 - phase0 afterProcessEpoch	160.55 ms/op	162.24 ms/op	0.99
phase0 processEffectiveBalanceUpdates - 250000 normalcase	2.8038 ms/op	1.9346 ms/op	1.45
phase0 processEffectiveBalanceUpdates - 250000 worstcase 0.5	2.4747 ms/op	2.2050 ms/op	1.12
altair processInactivityUpdates - 250000 normalcase	40.399 ms/op	41.940 ms/op	0.96
altair processInactivityUpdates - 250000 worstcase	52.015 ms/op	35.062 ms/op	1.48
phase0 processRegistryUpdates - 250000 normalcase	7.3700 us/op	6.0820 us/op	1.21
phase0 processRegistryUpdates - 250000 badcase_full_deposits	425.16 us/op	364.48 us/op	1.17
phase0 processRegistryUpdates - 250000 worstcase 0.5	211.87 ms/op	205.60 ms/op	1.03
altair processRewardsAndPenalties - 250000 normalcase	133.48 ms/op	106.48 ms/op	1.25
altair processRewardsAndPenalties - 250000 worstcase	86.527 ms/op	74.570 ms/op	1.16
phase0 getAttestationDeltas - 250000 normalcase	13.144 ms/op	12.287 ms/op	1.07
phase0 getAttestationDeltas - 250000 worstcase	13.023 ms/op	12.772 ms/op	1.02
phase0 processSlashings - 250000 worstcase	5.3958 ms/op	5.0858 ms/op	1.06
altair processSyncCommitteeUpdates - 250000	271.22 ms/op	297.74 ms/op	0.91
BeaconState.hashTreeRoot - No change	459.00 ns/op	535.00 ns/op	0.86
BeaconState.hashTreeRoot - 1 full validator	55.077 us/op	66.045 us/op	0.83
BeaconState.hashTreeRoot - 32 full validator	548.81 us/op	651.73 us/op	0.84
BeaconState.hashTreeRoot - 512 full validator	6.0976 ms/op	7.1069 ms/op	0.86
BeaconState.hashTreeRoot - 1 validator.effectiveBalance	68.780 us/op	82.029 us/op	0.84
BeaconState.hashTreeRoot - 32 validator.effectiveBalance	1.0197 ms/op	1.1841 ms/op	0.86
BeaconState.hashTreeRoot - 512 validator.effectiveBalance	12.863 ms/op	15.559 ms/op	0.83
BeaconState.hashTreeRoot - 1 balances	53.413 us/op	63.684 us/op	0.84
BeaconState.hashTreeRoot - 32 balances	499.52 us/op	591.56 us/op	0.84
BeaconState.hashTreeRoot - 512 balances	4.5504 ms/op	5.3713 ms/op	0.85
BeaconState.hashTreeRoot - 250000 balances	96.097 ms/op	95.579 ms/op	1.01
aggregationBits - 2048 els - zipIndexesInBitList	28.632 us/op	33.521 us/op	0.85
regular array get 100000 times	67.473 us/op	60.562 us/op	1.11
wrappedArray get 100000 times	67.464 us/op	60.575 us/op	1.11
arrayWithProxy get 100000 times	30.373 ms/op	31.812 ms/op	0.95
ssz.Root.equals	447.00 ns/op	502.00 ns/op	0.89
byteArrayEquals	455.00 ns/op	500.00 ns/op	0.91
shuffle list - 16384 els	11.277 ms/op	11.174 ms/op	1.01
shuffle list - 250000 els	165.60 ms/op	166.34 ms/op	1.00
processSlot - 1 slots	11.554 us/op	13.381 us/op	0.86
processSlot - 32 slots	1.6562 ms/op	1.9198 ms/op	0.86
getEffectiveBalanceIncrementsZeroInactive - 250000 vs - 7PWei	401.92 us/op	350.65 us/op	1.15
getCommitteeAssignments - req 1 vs - 250000 vc	5.3224 ms/op	5.5108 ms/op	0.97
getCommitteeAssignments - req 100 vs - 250000 vc	7.3332 ms/op	7.9073 ms/op	0.93
getCommitteeAssignments - req 1000 vs - 250000 vc	7.7468 ms/op	8.5875 ms/op	0.90
computeProposers - vc 250000	17.378 ms/op	17.603 ms/op	0.99
computeEpochShuffling - vc 250000	169.38 ms/op	169.14 ms/op	1.00
getNextSyncCommittee - vc 250000	262.39 ms/op	295.82 ms/op	0.89

by benchmarkbot/action

[wemeetagain]

Discussion in discord:

• We'll update this to be in line with https://github.com/ChainSafe/engineering-wiki/pull/28
• If someone on the team takes this task, they should assign themselves first so no one else duplicates their work

[dapplion]

Putting as draft until:

• We'll update this to be in line with https://github.com/ChainSafe/engineering-wiki/pull/28

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in 15 days if no further activity occurs. Thank you for your contributions.

[dapplion]

After discussions with the team, we will stick with Lerna for now