btcd icon indicating copy to clipboard operation
btcd copied to clipboard

blockchain: rolling merkle root

Open cfromknecht opened this issue 6 years ago • 2 comments

This PR adds a new RollingMerkleTree type for efficiently computing merkle roots. The leaves of the tree are iteratively pushed onto the tree, and any complete substrees are evaluated and stored so that subsequent pushes may consume them when their sibling tree is complete. As a result, this requires O(log n) additional memory be allocated. The current BuildMerkleTreeStore allocates an array of up to 4 times the number of leaves, which is used to store internal nodes and the root.

For 2000 leaves, the benchmarks show that this method is about 7% slower (~90 microseconds), but uses 99.04% less memory and 99.85% fewer allocations. These allocations reduce gc pressure, especially during IBD when the large slices of hashes currently used are almost immediately discarded.

BenchmarkMerkle/rolling-1000-8              2000            694382 ns/op             858 B/op          3 allocs/op
BenchmarkMerkle/rolling-2000-8              1000           1367763 ns/op             923 B/op          3 allocs/op
BenchmarkMerkle/rolling-4000-8               500           2900103 ns/op            1210 B/op          6 allocs/op
BenchmarkMerkle/rolling-8000-8               300           5799806 ns/op            2034 B/op         10 allocs/op
BenchmarkMerkle/rolling-16000-8              100          11572403 ns/op            1578 B/op          3 allocs/op
BenchmarkMerkle/rolling-32000-8               50          22949069 ns/op            1720 B/op          5 allocs/op
BenchmarkMerkle/old-1000-8                  2000            655435 ns/op           48416 B/op       1002 allocs/op
BenchmarkMerkle/old-2000-8                  1000           1277192 ns/op           96800 B/op       2002 allocs/op
BenchmarkMerkle/old-4000-8                   500           2533821 ns/op          193568 B/op       4002 allocs/op
BenchmarkMerkle/old-8000-8                   300           5029310 ns/op          387104 B/op       8002 allocs/op              
BenchmarkMerkle/old-16000-8                  200          10135631 ns/op          774176 B/op      16002 allocs/op                     
BenchmarkMerkle/old-32000-8                  100          20442194 ns/op         1548340 B/op      32002 allocs/op

alternative to #1377

cfromknecht avatar Apr 18 '19 01:04 cfromknecht

@jcvernaleo (as per #1530)

  • Low priority
  • Enhancement
  • Duplicate (of #1377, not sure if we have a tag for this)

jakesylvestre avatar Mar 04 '20 13:03 jakesylvestre

@cfromknecht mind rebasing so the actions can run?

Rjected avatar Jun 08 '20 14:06 Rjected