silkworm icon indicating copy to clipboard operation
silkworm copied to clipboard

Inefficient TxLookup prune

Open AndreaLanfranchi opened this issue 4 years ago • 2 comments

The prune function for tx lookups seems to me quite inefficient.

Basically it traverses all records from BlockTransactionLookup bucket in search of linked block numbers which are below the pruning threshold. That table has hundreds of millions records.

I believe a more efficient approach is to load transaction hashes from to-be-pruned blocks and delete records by transaction hash which is the key of BlockTransactionLookup bucket

AndreaLanfranchi avatar Sep 08 '21 20:09 AndreaLanfranchi

I tested it with Paolo, it is much faster like this.

Giulio2002 avatar Sep 13 '21 14:09 Giulio2002

I tested full bucket vs small bucket and it is much faster in either ways. i don't know if that it is because we avoid to moving the cursor "EthTx" back and forth or if it is because of hashing, but it is actually much faster like this. perhaps on HDD the opposite is true

Giulio2002 avatar Sep 13 '21 14:09 Giulio2002