realtps icon indicating copy to clipboard operation
realtps copied to clipboard

Add optimizations for summing transactions across many blocks

Open brson opened this issue 3 years ago • 1 comments

After just a few days of running it takes over an hour to recalculate tps. This is likely to the enormous amount of networked file system I/O.

I don't have a full design in mind, but I'm thinking some kind of caching layer that stores the number of transactions across a range of blocks. Needs to be simple.

Perhaps something like

  • While calculation is running it stores a vector cache of "runs" where each run is approximately 1 hour of blocks. Each run contains info like:
    • first block number, hash, timestamp
    • last block number, hash, timestamp
    • next block number, hash, timestamp
    • total number of transactions
  • After calculation it stores the whole vector of runs to a file for the next calculation to use
  • During calculation it traverses this cache before hitting the actual blocks, while building up a new cache

brson avatar Apr 14 '22 23:04 brson

My 'blockruns' branch has work on this.

brson avatar May 01 '22 19:05 brson