erigon
erigon copied to clipboard
[WIP] [DO NOT MERGE] Prune algorhythm changing
I've changed prune algo for ii and it was amazing until I spotted the bug))) but now they are pretty curious After reviewing I found that all of allocs happened only because I need to gather CountDups like an alone cgo-call for stat gathering. Without it all is amazing again!
For EthMainnet DB 152G (list of benchs would be updated) Measured pruning of oldest step (2027, 2028 exists) all timings would be updated with more clear stat
with CGo call CountDups:
goos: darwin
goarch: arm64
pkg: mdbx_to_csv/prune
cpu: Apple M3 Max
│ /tmp/old_prune.txt │ /tmp/new_prune.txt │
│ sec/op │ sec/op vs base │
Prune/StorageHistory-16 3.734 ± 4% 2.480 ± 4% -33.57% (p=0.000 n=10)
Prune/TracesFrom-16 4.412 ± 2% 2.460 ± 0% -44.24% (p=0.000 n=10)
Prune/TracesTo-16 5.762 ± 4% 3.254 ± 0% -43.52% (p=0.000 n=10)
Prune/LogAddress-16 1.833 ± 3% 1.255 ± 1% -31.55% (p=0.000 n=10)
Prune/LogTopics-16 5.799 ± 6% 2.733 ± 1% -52.86% (p=0.000 n=10)
Prune/AccountHistory-16 3.154 ± 2% 2.497 ± 0% -20.86% (p=0.000 n=10)
Prune/ReceiptHistory-16 1.392 ± 1% 2.060 ± 2% +47.94% (p=0.000 n=10)
geomean 3.318 2.309 -30.41%
│ /tmp/old_prune.txt │ /tmp/new_prune.txt │
│ B/op │ B/op vs base │
Prune/StorageHistory-16 883.016Mi ± 0% 6.801Mi ± 0% -99.23% (p=0.000 n=10)
Prune/TracesFrom-16 705.672Mi ± 0% 3.292Mi ± 0% -99.53% (p=0.000 n=10)
Prune/TracesTo-16 1104.719Mi ± 0% 2.646Mi ± 0% -99.76% (p=0.000 n=10)
Prune/LogAddress-16 368208.2Ki ± 0% 103.5Ki ± 0% -99.97% (p=0.000 n=10)
Prune/LogTopics-16 1104.719Mi ± 0% 3.367Mi ± 0% -99.70% (p=0.000 n=10)
Prune/AccountHistory-16 883.016Mi ± 0% 3.217Mi ± 0% -99.64% (p=0.000 n=10)
Prune/ReceiptHistory-16 591200416.0 ± 0% 160.0 ± 0% -100.00% (p=0.000 n=10)
geomean 752.1Mi 529.6Ki -99.93%
│ /tmp/old_prune.txt │ /tmp/new_prune.txt │
│ allocs/op │ allocs/op vs base │
Prune/StorageHistory-16 61.50 ± 2% 891446.00 ± 0% +1449405.69% (p=0.000 n=10)
Prune/TracesFrom-16 58.00 ± 2% 431447.00 ± 0% +743774.14% (p=0.000 n=10)
Prune/TracesTo-16 63.00 ± 2% 346824.00 ± 0% +550414.29% (p=0.000 n=10)
Prune/LogAddress-16 52.00 ± 2% 13227.00 ± 0% +25336.54% (p=0.000 n=10)
Prune/LogTopics-16 63.00 ± 2% 441313.00 ± 0% +700396.83% (p=0.000 n=10)
Prune/AccountHistory-16 61.00 ± 2% 421632.00 ± 0% +691100.00% (p=0.000 n=10)
Prune/ReceiptHistory-16 56.000 ± 4% 6.000 ± 0% -89.29% (p=0.000 n=10)
geomean 59.09 57.06k +96470.75%
without CountDups (Amazing!)
goos: darwin
goarch: arm64
pkg: mdbx_to_csv/prune
cpu: Apple M3 Max
│ /tmp/old_prune.txt │ /tmp/new_prune.txt │
│ sec/op │ sec/op vs base │
Prune/StorageHistory-16 3.734 ± 4% 2.485 ± 1% -33.45% (p=0.000 n=10)
Prune/TracesFrom-16 4.412 ± 2% 2.432 ± 1% -44.87% (p=0.000 n=10)
Prune/TracesTo-16 5.762 ± 4% 3.230 ± 2% -43.94% (p=0.000 n=10)
Prune/LogAddress-16 1.833 ± 3% 1.274 ± 2% -30.49% (p=0.000 n=10)
Prune/LogTopics-16 5.799 ± 6% 2.791 ± 0% -51.87% (p=0.000 n=10)
Prune/AccountHistory-16 3.154 ± 2% 2.538 ± 0% -19.55% (p=0.000 n=10)
Prune/ReceiptHistory-16 1.392 ± 1% 2.108 ± 1% +51.42% (p=0.000 n=10)
geomean 3.318 2.329 -29.82%
│ /tmp/old_prune.txt │ /tmp/new_prune.txt │
│ B/op │ B/op vs base │
Prune/StorageHistory-16 925909216.0 ± 0% 160.0 ± 0% -100.00% (p=0.000 n=10)
Prune/TracesFrom-16 739950752.0 ± 0% 160.0 ± 0% -100.00% (p=0.000 n=10)
Prune/TracesTo-16 1158381824.0 ± 0% 160.0 ± 0% -100.00% (p=0.000 n=10)
Prune/LogAddress-16 377045152.0 ± 0% 160.0 ± 0% -100.00% (p=0.000 n=10)
Prune/LogTopics-16 1158381824.0 ± 0% 160.0 ± 0% -100.00% (p=0.000 n=10)
Prune/AccountHistory-16 925909208.0 ± 0% 160.0 ± 0% -100.00% (p=0.000 n=10)
Prune/ReceiptHistory-16 591200416.0 ± 0% 160.0 ± 0% -100.00% (p=0.000 n=10)
geomean 752.1Mi 160.0 -100.00%
│ /tmp/old_prune.txt │ /tmp/new_prune.txt │
│ allocs/op │ allocs/op vs base │
Prune/StorageHistory-16 61.500 ± 2% 6.000 ± 0% -90.24% (p=0.000 n=10)
Prune/TracesFrom-16 58.000 ± 2% 6.000 ± 0% -89.66% (p=0.000 n=10)
Prune/TracesTo-16 63.000 ± 2% 6.000 ± 0% -90.48% (p=0.000 n=10)
Prune/LogAddress-16 52.000 ± 2% 6.000 ± 0% -88.46% (p=0.000 n=10)
Prune/LogTopics-16 63.000 ± 2% 6.000 ± 0% -90.48% (p=0.000 n=10)
Prune/AccountHistory-16 61.000 ± 2% 6.000 ± 0% -90.16% (p=0.000 n=10)
Prune/ReceiptHistory-16 56.000 ± 4% 6.000 ± 0% -89.29% (p=0.000 n=10)
geomean 59.09 6.000 -89.85%
Need to do:
- [x] More tables
- [ ] Add comparison stats of current approach, alldups (PR) and range_del for different tables
- [ ] Think about expanding this to commitment, because now it's ii only (but i think i can't)