snarkOS icon indicating copy to clipboard operation
snarkOS copied to clipboard

Use more cores with rayon while syncing with the CDN

Open ljedrz opened this issue 1 year ago • 7 comments

Since we perform the CDN sync before starting the node, we can use a rayon threadpool with all the cores dedicated to syncing without sacrificing any other functionality.

I tested it back in testnet3 and saw higher CPU utilization.

ljedrz avatar Apr 09 '24 11:04 ljedrz

How much faster is the rate of syncing? Can you share numbers for pre and post syncing for this PR?

howardwu avatar Apr 09 '24 22:04 howardwu

Back when I was testing on testnet3 I noted that it doubled my CPU utilization; the exact increase in speed would depend on the number and power of cores, but this results in both faster deserialization and ledger insertion.

ljedrz avatar Apr 10 '24 05:04 ljedrz

Do you have empirical data that you can share to confirm this finding?

howardwu avatar Apr 11 '24 20:04 howardwu

Unfortunately no, not unless @zosorock has performed some measurements back when this was requested.

In any case, it's a pretty trivial change, so it can wait until the next time we're using a CDN if we'd like to collect some solid figures before merging.

ljedrz avatar Apr 22 '24 11:04 ljedrz

Closing pending testing of newer optimizations for syncing logic.

apruden2008 avatar Jun 09 '24 21:06 apruden2008

@vicsn @ljedrz Please keep this PR in draft state as it's a valid optimization for snarkOS.

howardwu avatar Jun 09 '24 22:06 howardwu

gm

Lanckneht avatar Jul 19 '24 16:07 Lanckneht