reth icon indicating copy to clipboard operation
reth copied to clipboard

feat(trie): hash post state in parallel

Open jsvisa opened this issue 1 year ago • 12 comments

implement and close https://github.com/paradigmxyz/reth/issues/7183

jsvisa avatar Mar 18 '24 03:03 jsvisa

needs @rkrasiuk's review

shekhirin avatar Mar 19 '24 14:03 shekhirin

classic piece of code that needs a unit test

@emhane as we already had an test case for this function in https://github.com/paradigmxyz/reth/blob/3c8251b40253e92112e82e209026aa36cde0dc85/crates/storage/provider/src/bundle_state/bundle_state_with_receipts.rs#L1179-L1195

so I add a simple unit test case to check the changes doesn't affect the state root

jsvisa avatar Mar 19 '24 14:03 jsvisa

needs benchmark also! Maybe for small number of items serial is faster

gakonst avatar Mar 19 '24 19:03 gakonst

a mandatory requirement to implement #7183 is to also write a benchmark to determine the threshold for parallelism

rkrasiuk avatar Mar 19 '24 20:03 rkrasiuk

Yeah I'd love if we had a cargo bench that shows serial vs parallel perf for various sizes, and whether other techniques like bucketing 1K hashes in 10 threads x 100 keccaks each perform better?

gakonst avatar Mar 20 '24 19:03 gakonst

I will try to implement a benchmark tool and give a reasonably comparative result based on concurrency and data volume.

jsvisa avatar Mar 21 '24 02:03 jsvisa

Hey @jsvisa! Let me know if you are blocked on something here 😊

onbjerg avatar Apr 18 '24 11:04 onbjerg

Hey @jsvisa! Let me know if you are blocked on something here 😊

Sorry for the long and late response, I'm not familiar with Rust, I think I can have another try and update this pr the day before this weekend

jsvisa avatar Apr 18 '24 11:04 jsvisa

No worries! Let us know if you have questions/need help getting this over the line 😊

onbjerg avatar Apr 18 '24 11:04 onbjerg

Looks good from my side, all we need now is a benchmark, you can see some examples here: https://github.com/paradigmxyz/reth/tree/main/crates/trie/benches

onbjerg avatar Apr 25 '24 18:04 onbjerg

Looks good from my side, all we need now is a benchmark, you can see some examples here: https://github.com/paradigmxyz/reth/tree/main/crates/trie/benches

thank you, try to implement it this weekend

jsvisa avatar Apr 26 '24 00:04 jsvisa

@mattsse @onbjerg Sorry for the delay of updating this PR. I've added the benchmark, and the result below:

image image

jsvisa avatar May 11 '24 13:05 jsvisa