erigon icon indicating copy to clipboard operation
erigon copied to clipboard

[WIP] [DO NOT MERGE] Prune algorhythm changing

Open JkLondon opened this issue 3 weeks ago • 4 comments

I've changed prune algo for ii and it was amazing until I spotted the bug))) but now they are pretty curious After reviewing I found that all of allocs happened only because I need to gather CountDups like an alone cgo-call for stat gathering. Without it all is amazing again!

For EthMainnet DB 152G (list of benchs would be updated) Measured pruning of oldest step (2027, 2028 exists) all timings would be updated with more clear stat

with CGo call CountDups:

goos: darwin
goarch: arm64
pkg: mdbx_to_csv/prune
cpu: Apple M3 Max
                        │ /tmp/old_prune.txt │         /tmp/new_prune.txt         │
                        │       sec/op       │   sec/op    vs base                │
Prune/StorageHistory-16           3.734 ± 4%   2.480 ± 4%  -33.57% (p=0.000 n=10)
Prune/TracesFrom-16               4.412 ± 2%   2.460 ± 0%  -44.24% (p=0.000 n=10)
Prune/TracesTo-16                 5.762 ± 4%   3.254 ± 0%  -43.52% (p=0.000 n=10)
Prune/LogAddress-16               1.833 ± 3%   1.255 ± 1%  -31.55% (p=0.000 n=10)
Prune/LogTopics-16                5.799 ± 6%   2.733 ± 1%  -52.86% (p=0.000 n=10)
Prune/AccountHistory-16           3.154 ± 2%   2.497 ± 0%  -20.86% (p=0.000 n=10)
Prune/ReceiptHistory-16           1.392 ± 1%   2.060 ± 2%  +47.94% (p=0.000 n=10)
geomean                           3.318        2.309       -30.41%

                        │ /tmp/old_prune.txt │          /tmp/new_prune.txt           │
                        │        B/op        │     B/op      vs base                 │
Prune/StorageHistory-16       883.016Mi ± 0%   6.801Mi ± 0%   -99.23% (p=0.000 n=10)
Prune/TracesFrom-16           705.672Mi ± 0%   3.292Mi ± 0%   -99.53% (p=0.000 n=10)
Prune/TracesTo-16            1104.719Mi ± 0%   2.646Mi ± 0%   -99.76% (p=0.000 n=10)
Prune/LogAddress-16          368208.2Ki ± 0%   103.5Ki ± 0%   -99.97% (p=0.000 n=10)
Prune/LogTopics-16           1104.719Mi ± 0%   3.367Mi ± 0%   -99.70% (p=0.000 n=10)
Prune/AccountHistory-16       883.016Mi ± 0%   3.217Mi ± 0%   -99.64% (p=0.000 n=10)
Prune/ReceiptHistory-16     591200416.0 ± 0%     160.0 ± 0%  -100.00% (p=0.000 n=10)
geomean                         752.1Mi        529.6Ki        -99.93%

                        │ /tmp/old_prune.txt │             /tmp/new_prune.txt              │
                        │     allocs/op      │   allocs/op     vs base                     │
Prune/StorageHistory-16           61.50 ± 2%   891446.00 ± 0%  +1449405.69% (p=0.000 n=10)
Prune/TracesFrom-16               58.00 ± 2%   431447.00 ± 0%   +743774.14% (p=0.000 n=10)
Prune/TracesTo-16                 63.00 ± 2%   346824.00 ± 0%   +550414.29% (p=0.000 n=10)
Prune/LogAddress-16               52.00 ± 2%    13227.00 ± 0%    +25336.54% (p=0.000 n=10)
Prune/LogTopics-16                63.00 ± 2%   441313.00 ± 0%   +700396.83% (p=0.000 n=10)
Prune/AccountHistory-16           61.00 ± 2%   421632.00 ± 0%   +691100.00% (p=0.000 n=10)
Prune/ReceiptHistory-16          56.000 ± 4%       6.000 ± 0%       -89.29% (p=0.000 n=10)
geomean                           59.09           57.06k         +96470.75%

without CountDups (Amazing!)

goos: darwin
goarch: arm64
pkg: mdbx_to_csv/prune
cpu: Apple M3 Max
                        │ /tmp/old_prune.txt │         /tmp/new_prune.txt         │
                        │       sec/op       │   sec/op    vs base                │
Prune/StorageHistory-16           3.734 ± 4%   2.485 ± 1%  -33.45% (p=0.000 n=10)
Prune/TracesFrom-16               4.412 ± 2%   2.432 ± 1%  -44.87% (p=0.000 n=10)
Prune/TracesTo-16                 5.762 ± 4%   3.230 ± 2%  -43.94% (p=0.000 n=10)
Prune/LogAddress-16               1.833 ± 3%   1.274 ± 2%  -30.49% (p=0.000 n=10)
Prune/LogTopics-16                5.799 ± 6%   2.791 ± 0%  -51.87% (p=0.000 n=10)
Prune/AccountHistory-16           3.154 ± 2%   2.538 ± 0%  -19.55% (p=0.000 n=10)
Prune/ReceiptHistory-16           1.392 ± 1%   2.108 ± 1%  +51.42% (p=0.000 n=10)
geomean                           3.318        2.329       -29.82%

                        │ /tmp/old_prune.txt │         /tmp/new_prune.txt          │
                        │        B/op        │    B/op     vs base                 │
Prune/StorageHistory-16     925909216.0 ± 0%   160.0 ± 0%  -100.00% (p=0.000 n=10)
Prune/TracesFrom-16         739950752.0 ± 0%   160.0 ± 0%  -100.00% (p=0.000 n=10)
Prune/TracesTo-16          1158381824.0 ± 0%   160.0 ± 0%  -100.00% (p=0.000 n=10)
Prune/LogAddress-16         377045152.0 ± 0%   160.0 ± 0%  -100.00% (p=0.000 n=10)
Prune/LogTopics-16         1158381824.0 ± 0%   160.0 ± 0%  -100.00% (p=0.000 n=10)
Prune/AccountHistory-16     925909208.0 ± 0%   160.0 ± 0%  -100.00% (p=0.000 n=10)
Prune/ReceiptHistory-16     591200416.0 ± 0%   160.0 ± 0%  -100.00% (p=0.000 n=10)
geomean                         752.1Mi        160.0       -100.00%

                        │ /tmp/old_prune.txt │         /tmp/new_prune.txt         │
                        │     allocs/op      │ allocs/op   vs base                │
Prune/StorageHistory-16          61.500 ± 2%   6.000 ± 0%  -90.24% (p=0.000 n=10)
Prune/TracesFrom-16              58.000 ± 2%   6.000 ± 0%  -89.66% (p=0.000 n=10)
Prune/TracesTo-16                63.000 ± 2%   6.000 ± 0%  -90.48% (p=0.000 n=10)
Prune/LogAddress-16              52.000 ± 2%   6.000 ± 0%  -88.46% (p=0.000 n=10)
Prune/LogTopics-16               63.000 ± 2%   6.000 ± 0%  -90.48% (p=0.000 n=10)
Prune/AccountHistory-16          61.000 ± 2%   6.000 ± 0%  -90.16% (p=0.000 n=10)
Prune/ReceiptHistory-16          56.000 ± 4%   6.000 ± 0%  -89.29% (p=0.000 n=10)
geomean                           59.09        6.000       -89.85%

Need to do:

  • [x] More tables
  • [ ] Add comparison stats of current approach, alldups (PR) and range_del for different tables
  • [ ] Think about expanding this to commitment, because now it's ii only (but i think i can't)

JkLondon avatar Dec 07 '25 01:12 JkLondon