client: create snap sync fetcher to sync accounts

[am1r021]

AccountFetcher Capture data now done/part of

• https://github.com/ethereumjs/ethereumjs-monorepo/pull/2157
  • getAccountRange
  • getStorageRanges
  • getByteCodes
  • getTrieNodes

My understanding is that there needs to be one fetcher per each get call defined in the snap specs. Open to suggestions on this design plan.

[codecov[bot]]

Codecov Report

Merging #2107 (420d087) into master (0cf9654) will decrease coverage by 0.72%. The diff coverage is 100.00%.

Additional details and impacted files

Flag	Coverage Δ
block	?
blockchain	90.19% <ø> (ø)
client	?
common	95.82% <ø> (ø)
devp2p	91.62% <ø> (-0.09%)	:arrow_down:
ethash	?
evm	83.36% <ø> (ø)
rlp	∅ <ø> (?)
statemanager	?
trie	?
tx	?
util	84.50% <100.00%> (+0.01%)	:arrow_up:
vm	?

Flags with carried forward coverage won't be shown. Click here to find out more.

[am1r021]

When calling getAccountRange, the proof and accounts arrays contain data elements that look like this:

{
  account: {
    hash: <Buffer 59 41 33 56 5c 9e 83 0b 70 3d d8 a3 24 55 45 73 d2 16 e1 60 a0 83 28 db 62 f7 85 c9 ff 90 61 0d>,
    body: [
      <Buffer 05>,
      <Buffer 16 48 4e a6 bc cd 3f>,
      Uint8Array(0) [],
      Uint8Array(0) []
    ]
  },
  proof: <Buffer f8 66 9d 3b a0 c8 17 db 1f e9 ac 90 64 49 6c 76 75 81 3b 54 3d c9 ec 10 5c 1c 24 4e 43 3b c8 9c b8 46 f8 44 01 80 a0 56 e8 1f 17 1b cc 55 a6 ff 83 45 ... 54 more bytes>
}

Sometimes, the proof is undefined:

  account: {
    hash: <Buffer 59 41 34 ee 04 72 a6 7a b4 0e 94 e5 aa bb 00 74 02 a7 74 fb c9 d8 bd 2c 76 bc 36 36 9c b5 fa f1>,
    body: [
      <Buffer 01>,
      Uint8Array(0) [],
      <Buffer 8f b4 c3 c2 7f 69 21 c6 44 dd fc 94 24 97 5b 91 b4 58 82 47 ca 2a 1d ec 36 96 da 65 a3 97 bb f5>,
      <Buffer f1 b5 74 43 1f 38 38 d9 cd ff 6e 70 1a fd 5a 05 86 52 da b5 ae 55 23 28 8a 83 d5 fa d7 69 61 39>
    ]
  },
  proof: undefined

Will need to also finish implementing StorageRanges, ByteCodes, and TrieNodes and collecting data for them too.

[am1r021]

Now that there is reference data for getAccountRange, next steps are to figure out how to divide the fetching process into smaller tasks, if necessary. When fetching accounts, we need to verify the integrity of the account data using the included proof.

[g11tech]

When calling getAccountRange, the proof and accounts arrays contain data elements that look like this:

{
  account: {
    hash: <Buffer 59 41 33 56 5c 9e 83 0b 70 3d d8 a3 24 55 45 73 d2 16 e1 60 a0 83 28 db 62 f7 85 c9 ff 90 61 0d>,
.....
Sometimes, the proof is `undefined`:

account: { hash: <Buffer 59 41 34 ee 04 72 a6 7a b4 0e 94 e5 aa bb 00 74 02 a7 74 fb c9 d8 bd 2c 76 bc 36 36 9c b5 fa f1>, .... proof: undefined

Will need to also finish implementing `StorageRanges`, `ByteCodes`, and `TrieNodes` and collecting data for them too.

awesome glad to see you got it working!

[g11tech]

yes we need separate fetchers for fetching each of the ranges :+1:

[am1r021]

@g11tech I wasn't able to get forceSnapSync option to work. I would set it like this:

DEBUG=devp2p:* npm run client:start:dev2 -- --forceSnapSync=true ...

I ended up hardcoding it to true. Am I setting it incorrectly?

[g11tech]

@g11tech I wasn't able to get forceSnapSync option to work. I would set it like this:

DEBUG=devp2p:* npm run client:start:dev2 -- --forceSnapSync=true ...

I ended up hardcoding it to true. Am I setting it incorrectly?

looks alright to me the way you set, i will check and add a fix in this PR

[g11tech]

:tada: the fetcher syncs range on mainnet! (and then create new tasks and sync them further)

[gitpoap-bot[bot]]

Congrats, your important contribution to this open-source project has earned you a GitPOAP!

GitPOAP: 2023 EthereumJS Contributor:

Head to gitpoap.io & connect your GitHub account to mint!

Learn more about GitPOAPs here.