IntersectMBO
benchmark TICKF
We need realistic benchmarks of the TICKF rule in order to determine if #3034 is a real problem.
The consensus layer calls TICKF once per slot to determine if a block producing node has been elected leader. When the slot in question is in a new epoch (relative to the last ledger view that the consensus layer has stored), this results in a potentially expensive computation (see #3034).
TICKF is defined here (every ledger era has used the Shelley TICKF):
https://github.com/input-output-hk/cardano-ledger/blob/1176affe9b2a7a3b50c1ceda00cc92f237ea6360/eras/shelley/impl/src/Cardano/Ledger/Shelley/Rules/Tick.hs#L262
In order to get realistic data for our benchmarks, we can use db-analyser to get the ledger state on mainnet just before some of the epoch boundaries. The benchmarks will deserialize the ledger state obtained from db-analyser, and project the NewEpochState for use in the TICKF rule.
Some slots we might consider (this issue was created in the middle of epoch 363):
| slot number | block number | last block of epoch X |
|---|---|---|
| 71452796 | 7749373 | 362 |
| 71020795 | 7728107 | 361 |
| 70588777 | 7706865 | 360 |
| 70156668 | 7685639 | 359 |
| 69724795 | 7664616 | 358 |
Note that the benchmarks will need to use a Slot past the epoch boundary in order to trigger the large calculations.
Tasks:
- [ ] Write a tool that takes a ledger state snapshot from db-analyser and benchmarks the
TICKFrule. - [ ] Create ledger state snapshots for the slots in the table above.
- [ ] Record the times obtained from running the new tool on the snapshots. (The report can be placed in this GitHub issue.)
First task is practically done, we have benchmarks that take mainnet NewEpochState in cbor encoded form and run various benchmarks using that state:
https://github.com/input-output-hk/cardano-ledger/blob/1176affe9b2a7a3b50c1ceda00cc92f237ea6360/libs/ledger-state/bench/Performance.hs#L39-L45
All it needs is a change of CurrentEra to Babbage, since now it is set to Alonzo
I'll link this PR here in case it provides relevant information for y'all https://github.com/input-output-hk/ouroboros-network/pull/4014
The pull request https://github.com/input-output-hk/cardano-ledger/pull/3068 expands on what is started here. Profiling the tickf rule, and also profiling the individual components that make it up. From these components we can see where the time is spent. Here is the output from the benchmark there. Note that the benchmark labels try to convey what is being benchmark as well as what code that snippet is embedded in. In a perfect system the sum of all the "sub-parts" of a benchmark should add up to the time spent in the root.
loads the NewEpochState from CBOR for slot 71452796 then benchmarks the tickf rule
Running 1 benchmarks...
Benchmark bench: RUNNING...
benchmarking Tickf Benchmarks/validatingTickTransitionfunction
time 2.915 s (2.149 s .. 4.514 s)
0.966 R² (0.941 R² .. 1.000 R²)
mean 2.363 s (2.199 s .. 2.660 s)
std dev 289.4 ms (20.99 ms .. 355.5 ms)
variance introduced by outliers: 23% (moderately inflated)
benchmarking Tickf Benchmarks/Tick subparts/adoptGenesisDelegs
time 39.79 ns (38.31 ns .. 42.09 ns)
0.990 R² (0.983 R² .. 0.998 R²)
mean 40.76 ns (39.70 ns .. 42.10 ns)
std dev 4.070 ns (2.833 ns .. 5.666 ns)
variance introduced by outliers: 91% (severely inflated)
benchmarking Tickf Benchmarks/Tick subparts/newEpoch
time 2.624 s (1.910 s .. 4.233 s)
0.956 R² (0.933 R² .. 1.000 R²)
mean 2.108 s (1.963 s .. 2.387 s)
std dev 277.0 ms (4.662 ms .. 329.2 ms)
variance introduced by outliers: 24% (moderately inflated)
benchmarking Tickf Benchmarks/Tick subparts/NewEpoch sub parts/updateRewards
time 1.768 s (1.077 s .. 3.237 s)
0.924 R² (0.882 R² .. 1.000 R²)
mean 1.266 s (1.121 s .. 1.536 s)
std dev 263.6 ms (14.40 ms .. 319.5 ms)
variance introduced by outliers: 48% (moderately inflated)
benchmarking Tickf Benchmarks/Tick subparts/NewEpoch sub parts/updateRewards subparts/sumRewards
time 134.9 ms (128.4 ms .. 140.6 ms)
0.997 R² (0.992 R² .. 1.000 R²)
mean 132.2 ms (129.7 ms .. 134.1 ms)
std dev 3.311 ms (2.039 ms .. 5.198 ms)
variance introduced by outliers: 11% (moderately inflated)
benchmarking Tickf Benchmarks/Tick subparts/NewEpoch sub parts/updateRewards subparts/applyRUpd' Incremental
time 1.017 s (926.0 ms .. 1.208 s)
0.996 R² (0.992 R² .. 1.000 R²)
mean 1.019 s (967.0 ms .. 1.058 s)
std dev 53.14 ms (26.81 ms .. 73.44 ms)
variance introduced by outliers: 19% (moderately inflated)
benchmarking Tickf Benchmarks/Tick subparts/NewEpoch sub parts/updateRewards subparts/applyRUpd subparts/filterAllRewards
time 526.3 ms (490.9 ms .. 548.5 ms)
0.999 R² (0.998 R² .. 1.000 R²)
mean 504.3 ms (494.4 ms .. 514.2 ms)
std dev 11.68 ms (9.220 ms .. 13.60 ms)
variance introduced by outliers: 19% (moderately inflated)
benchmarking Tickf Benchmarks/Tick subparts/NewEpoch sub parts/updateRewards subparts/applyRUpd subparts/aggregateRewards
time 111.7 ms (101.7 ms .. 121.6 ms)
0.992 R² (0.987 R² .. 0.999 R²)
mean 102.6 ms (100.5 ms .. 107.3 ms)
std dev 4.786 ms (2.358 ms .. 7.114 ms)
variance introduced by outliers: 10% (moderately inflated)
benchmarking Tickf Benchmarks/Tick subparts/NewEpoch sub parts/updateRewards subparts/applyRUpd subparts/union+
time 427.1 ms (382.6 ms .. 474.3 ms)
0.998 R² (0.998 R² .. 1.000 R²)
mean 388.2 ms (373.5 ms .. 404.0 ms)
std dev 19.10 ms (9.373 ms .. 24.64 ms)
variance introduced by outliers: 19% (moderately inflated)
benchmarking Tickf Benchmarks/Tick subparts/NewEpoch sub parts/mir rule
time 977.1 ns (902.3 ns .. 1.052 μs)
0.973 R² (0.964 R² .. 0.996 R²)
mean 913.4 ns (882.2 ns .. 963.7 ns)
std dev 125.0 ns (81.15 ns .. 170.0 ns)
variance introduced by outliers: 94% (severely inflated)
benchmarking Tickf Benchmarks/Tick subparts/NewEpoch sub parts/epoch rule
time 116.2 ms (102.7 ms .. 133.5 ms)
0.987 R² (0.970 R² .. 0.999 R²)
mean 131.9 ms (126.0 ms .. 143.6 ms)
std dev 12.18 ms (4.841 ms .. 19.91 ms)
variance introduced by outliers: 24% (moderately inflated)
benchmarking Tickf Benchmarks/Tick subparts/NewEpoch sub parts/calculatePoolDistr, improved (current) version
time 593.9 ms (508.4 ms .. 729.6 ms)
0.993 R² (0.990 R² .. 1.000 R²)
mean 554.2 ms (536.3 ms .. 579.8 ms)
std dev 24.44 ms (8.037 ms .. 31.90 ms)
variance introduced by outliers: 19% (moderately inflated)
benchmarking Tickf Benchmarks/Tick subparts/NewEpoch sub parts/calculatePoolDistr subparts/poolStake
time 547.7 ms (399.0 ms .. 635.4 ms)
0.992 R² (0.975 R² .. 1.000 R²)
mean 562.4 ms (547.2 ms .. 580.1 ms)
std dev 20.44 ms (6.631 ms .. 27.45 ms)
variance introduced by outliers: 19% (moderately inflated)
benchmarking Tickf Benchmarks/Tick subparts/NewEpoch sub parts/calculatePoolDistr subparts/old calculatePoolDistr
time 1.294 s (1.200 s .. 1.471 s)
0.998 R² (0.996 R² .. 1.000 R²)
mean 1.310 s (1.276 s .. 1.349 s)
std dev 45.08 ms (16.56 ms .. 59.22 ms)
variance introduced by outliers: 19% (moderately inflated)
The PR makes a few changes to speed things up. Most notably the calculatePoolDistr function, which was speeded up (timed in the benchmark) from 1.294 seconds (old version) to .5939 seconds (new version)