erigon [DO-NOT-MERGE] .ef optimization experiments

Nov 28 '24 23:11 wmitsuda

note: code quality is "hacky", pls do not consider for review yet, only for evaluation of the overall solution + validating the strategy

Nov 29 '24 00:11 wmitsuda

now that the prototype 1 is out, let me rebase it to main as it is 20 days outdated...

Nov 29 '24 23:11 wmitsuda

Current status:

Implemented collate/merge support in addition to read from snapshots. In theory it is feature complete, now need to do a full validation of correctness + optimizations, polish the code, etc...
Hid everything behind a feature flag: --experimental.ef-optimization
- That means if someone runs this branch as-is against an existing node, it should behave like this PR is not applied and the data should be written/read like regular E3.
- If the feature flag is activated, new data moved from mdbx to snapshots will be written in the new optimized format (simple sequences/rebased elias-fano)
- New merged files will be converted during the merge process.
- Existing snapshots will not be touched, read support is backwards compatible.
- New data is NOT backwards compatible, can't disable the flag and go back to regular E3. DO BACKUP YOUR NODE BEFORE TRYING THIS PR.
Did some simple tests so far on holesky:
- Ran this PR with the flag DISABLED, waited for 1 collation, compared with regular node in order to validate feature flag disabled == working as before.
- Ran this PR with the flag ENABLED, waited for 1 collation, 1 step was moved to new format.

Next steps:

Do longer tests (to validate multiple merges) + data deep comparison with regular E3.
Bootstrapping a new holesky node with --no-downloader in other to simulate a snapshotter + full comparison with regular E3 node. Not sure if my hardware will do it in a sane timeframe, but I'll try it.

Dec 10 '24 04:12 wmitsuda

@wmitsuda FYI: maybe next commands may help to test files erigon snapshots integrity --check=InvertedIndex and erigon snapshots integrity --check=HistoryNoSystemTxs

Dec 10 '24 04:12 AskAlexSharov

Current status:

Regenerated snapshots on holesky, there is now a idx_verify tool for comparing two node's .ef file integrity:

time build/bin/integration idx_verify --sourcedir /var/lib/ethereum/erigon3-holesky/erig
on --targetdir /var/lib/ethereum/erigon3-holesky-experimental/erigon
2024/12/16 04:53:56 Comparing idx files:
2024/12/16 04:53:56 Deep checking file v1-accounts.0-64.ef...
2024/12/16 04:59:56 Deep checking file v1-accounts.64-68.ef...
2024/12/16 05:00:05 Deep checking file v1-accounts.68-70.ef...
2024/12/16 05:00:10 Deep checking file v1-accounts.70-71.ef...
2024/12/16 05:00:13 Deep checking file v1-code.0-64.ef...
2024/12/16 05:01:47 Deep checking file v1-code.64-68.ef...
2024/12/16 05:01:48 Deep checking file v1-code.68-70.ef...
2024/12/16 05:01:48 Deep checking file v1-code.70-71.ef...
2024/12/16 05:01:48 Deep checking file v1-logaddrs.0-64.ef...
2024/12/16 05:02:03 Deep checking file v1-logaddrs.64-68.ef...
2024/12/16 05:02:06 Deep checking file v1-logaddrs.68-70.ef...
2024/12/16 05:02:07 Deep checking file v1-logaddrs.70-71.ef...
2024/12/16 05:02:07 Deep checking file v1-logtopics.0-64.ef...
2024/12/16 05:13:25 Deep checking file v1-logtopics.64-68.ef...
2024/12/16 05:13:55 Deep checking file v1-logtopics.68-70.ef...
2024/12/16 05:14:03 Deep checking file v1-logtopics.70-71.ef...
2024/12/16 05:14:07 Deep checking file v1-receipt.0-64.ef...
2024/12/16 05:14:07 Optimized eliasfano is longer
2024/12/16 05:14:07 key=0x00
2024/12/16 05:14:07 source min=0 max=99999999 count=67244080
2024/12/16 05:14:07 target min=0 max=99999999 count=97250285
2024/12/16 05:14:07 value mismatch!
2024/12/16 05:14:07 skipping to next file...
2024/12/16 05:14:07 Deep checking file v1-receipt.64-68.ef...
2024/12/16 05:14:08 Deep checking file v1-receipt.68-70.ef...
2024/12/16 05:14:08 Deep checking file v1-receipt.70-71.ef...
2024/12/16 05:14:08 values mismatch: source=109375000 target=109375019
2024/12/16 05:14:08 key=0x01
2024/12/16 05:14:08 source min=109375000 max=110937468 count=97974
2024/12/16 05:14:08 target min=109375019 max=110937468 count=97974
2024/12/16 05:14:08 value mismatch!
2024/12/16 05:14:08 skipping to next file...
2024/12/16 05:14:08 Deep checking file v1-storage.0-64.ef...
2024/12/16 05:34:50 Deep checking file v1-storage.64-68.ef...
2024/12/16 05:38:57 Deep checking file v1-storage.68-70.ef...
2024/12/16 05:41:05 Deep checking file v1-storage.70-71.ef...
2024/12/16 05:41:44 Deep checking file v1-tracesfrom.0-64.ef...
2024/12/16 05:45:46 Deep checking file v1-tracesfrom.64-68.ef...
2024/12/16 05:45:54 Deep checking file v1-tracesfrom.68-70.ef...
2024/12/16 05:45:57 Deep checking file v1-tracesfrom.70-71.ef...
2024/12/16 05:45:59 Deep checking file v1-tracesto.0-64.ef...
2024/12/16 05:50:10 Deep checking file v1-tracesto.64-68.ef...
2024/12/16 05:50:19 Deep checking file v1-tracesto.68-70.ef...
2024/12/16 05:50:23 Deep checking file v1-tracesto.70-71.ef...
build/bin/integration idx_verify --sourcedir  --targetdir   1242.30s user 302.65s system 45% cpu 56:28.69 total

Except for v1-receipt-*.ef, deep comparison between a node synced from current snapshots vs genesis sync + this PR optimizations showed data is equivalent.
Receipts seems buggy according to our conversation in discord, so let's consider this test was successful as everything else worked as expected.
The /idx size was reduced by 29GB as forecasted by the initial calculations:

du -hd1 /var/lib/ethereum/erigon3-holesky/erigon/snapshots
11G     /var/lib/ethereum/erigon3-holesky/erigon/snapshots/history
4.0K    /var/lib/ethereum/erigon3-holesky/erigon/snapshots/caplin
7.1G    /var/lib/ethereum/erigon3-holesky/erigon/snapshots/accessor
36G     /var/lib/ethereum/erigon3-holesky/erigon/snapshots/domain
51G     /var/lib/ethereum/erigon3-holesky/erigon/snapshots/idx
164G    /var/lib/ethereum/erigon3-holesky/erigon/snapshots

du -hd1 /var/lib/ethereum/erigon3-holesky-experimental/erigon/snapshots
11G     /var/lib/ethereum/erigon3-holesky-experimental/erigon/snapshots/history
4.0K    /var/lib/ethereum/erigon3-holesky-experimental/erigon/snapshots/caplin
7.2G    /var/lib/ethereum/erigon3-holesky-experimental/erigon/snapshots/accessor
36G     /var/lib/ethereum/erigon3-holesky-experimental/erigon/snapshots/domain
22G     /var/lib/ethereum/erigon3-holesky-experimental/erigon/snapshots/idx
136G    /var/lib/ethereum/erigon3-holesky-experimental/erigon/snapshots

so, we can conclude data is correct + size reduction is the expected one so far.

Dec 16 '24 04:12 wmitsuda

from now on, I think we are ready to advance one more step towards making it stable:

We could use some preliminary review to fix obvious design decisions (@AskAlexSharov when you have some time), but not merge yet.
Do a genesis sync on a real chain, i.e. mainnet and observe the results. Let it run for a longer period.

Dec 16 '24 04:12 wmitsuda

Current status:

Converter (idx_optimize) ran on polygon took +20 hours. It looks ok-ish for something that will run only once. Won't invest in optiming it unless there is some obvious optimization.

real    1250m42.555s
user    1097m13.994s
sys     127m1.218s

Verifier (idx_verify) took +11 hours and didn't find any errors.

real    687m25.619s
user    557m49.088s
sys     107m10.684s

Difference between pre/post conversion confirms over ~800GB reduction.

erigon@dev-bm-e3-bormainnet-n1:/erigon-data/e3-polygon-sync-mainnet$ du -hd1 snapshots/
790G    snapshots/idx
766G    snapshots/history
835G    snapshots/domain
327G    snapshots/accessor
4.0K    snapshots/caplin
4.0T    snapshots/
erigon@dev-bm-e3-bormainnet-n1:/erigon-data/e3-polygon-sync-mainnet$ du -hd1 /erigon-data/snapshots-backup/
1.6T    /erigon-data/snapshots-backup/idx
766G    /erigon-data/snapshots-backup/history
835G    /erigon-data/snapshots-backup/domain
328G    /erigon-data/snapshots-backup/accessor
4.0K    /erigon-data/snapshots-backup/caplin
4.8T    /erigon-data/snapshots-backup/

Jan 19 '25 20:01 wmitsuda

regarding syncing from scratch:

analyzed the run on mainnet that Giulio made, seems ok, but I'm not 100% confident because:
- tracesfrom/tracesto/storage/logaddr/logtopic indexes all matched perfectly, which makes me confident that code is correct
- there were differences in most receipts idx files, but I my inclined to ignore those since my mainnet node is from 12/2024 while Giulio's run was from the end of 12/2024 and there were changes in receipts code that may have led to those differences. Also, it seems receipts code may still have some bugs, so that part of the test is still inconclusive.
- there were differences all most accounts idxs and in 1 code idx. But I'm almost sure this has occurred because of a bug I fixed starting of 01/2025 (hence, after Giulio has started his test) which affected range queries which may have generated incomplete domain data (hence different indexes).
- there were differences in the domain files which I didn't notice when I ran my first test on holesky, on mainnet test the size differences were biggers than expected, raising this issue. That could be explained by the bug I mentioned in the previous point.
- TLDR; the bug in range indexes should be making the indexes return less data than expected, since domain construction apparently relies on those, there were less domain data being generated.

In order to solve those last points, I'm syncing holesky again from scratch and will do another comparison with holesky converted vs. holesky from scratch from this PR.

If both snapshots are identical, then the code is correct.

Jan 19 '25 20:01 wmitsuda

updated the PR to consolidate the rollout tools:

make integration -> integration tool is required

Convert an existing node:

Stop erigon
BACKUP snapshots/accessor and snapshots/idx directories
Run: build/bin/integration idx_optimize --datadir <DATA_DIR>
That'll convert:
- snapshots/idx/*.ef -> snapshots/idx/*.ef.new files
- snapshots/accessor/*.efi -> snapshots/accessor/*.efi.new files
- Old files will NOT be replaced
(optional) Run the verifier tool: build/bin/integration idx_verify --sourcedir <DATA_DIR> --targetdir <DATA_DIR> --manuallyOptimized
Delete old .ef and .efi files:
- rm snapshots/idx/*.ef
- rm snapshots/accessor/*.efi
Rename new optimized files to the expected ones (use the rename tool: sudo apt install rename):
- In snapshots/idx dir: rename 's/\.ef.new\z/.ef/' *.ef.new
- In snapshots/accessor dir: rename 's/\.efi.new\z/.efi/' *.efi.new
Start Erigon using this PR branch. New data should be collated/merged in the new optimized format.

The idx_verifiy and idx_optimize tools can be safely deleted in the future after this file format is successfully deployed in all networks.

Jan 19 '25 20:01 wmitsuda

Status: finished another round of tests on holesky, now comparing a fresh synced node vs. this branch PR from scratch.

Methodology:

Deep compare 2 directories:
- Snapshot sync holesky + apply idx_optimize converter
- Sync this branch PR from scratch
Expected:
- .ef/.efi files should differ
- all other files MUST be the same, implying we reproduced the same snapshots deterministically except for the .ef/efi

Results:

Expected:

There are diffs on .ef/.efi files
idx_verify says contents are correct

NOT expected:

Diffs on .idx/.vi/.kvi files -> this is probably because of the salt.
Diffs on .bt/.kvei -> ???
Diffs on all receipts.v -> probably due to known receipts bugs, will ignore for now.
Diffs on exactly 1 .seg file: v1-000000-000100-bodies.seg -> ???

diff -rq eth-nodes/erigon3-holesky-experimental/snapshots eth-nodes/erigon3-holesky-converted/snapshots | grep -v torrent | grep -v \.efi
Files eth-nodes/erigon3-holesky-experimental/snapshots/accessor/v1-receipt.0-64.vi and eth-nodes/erigon3-holesky-converted/snapshots/accessor/v1-receipt.0-64.vi differ
Files eth-nodes/erigon3-holesky-experimental/snapshots/accessor/v1-receipt.72-74.vi and eth-nodes/erigon3-holesky-converted/snapshots/accessor/v1-receipt.72-74.vi differ
Files eth-nodes/erigon3-holesky-experimental/snapshots/domain/v1-accounts.0-64.bt and eth-nodes/erigon3-holesky-converted/snapshots/domain/v1-accounts.0-64.bt differ
Files eth-nodes/erigon3-holesky-experimental/snapshots/domain/v1-accounts.0-64.kvei and eth-nodes/erigon3-holesky-converted/snapshots/domain/v1-accounts.0-64.kvei differ
Files eth-nodes/erigon3-holesky-experimental/snapshots/domain/v1-accounts.64-72.kvei and eth-nodes/erigon3-holesky-converted/snapshots/domain/v1-accounts.64-72.kvei differ
Files eth-nodes/erigon3-holesky-experimental/snapshots/domain/v1-accounts.72-74.kvei and eth-nodes/erigon3-holesky-converted/snapshots/domain/v1-accounts.72-74.kvei differ
Files eth-nodes/erigon3-holesky-experimental/snapshots/domain/v1-code.0-64.bt and eth-nodes/erigon3-holesky-converted/snapshots/domain/v1-code.0-64.bt differ
Files eth-nodes/erigon3-holesky-experimental/snapshots/domain/v1-code.0-64.kvei and eth-nodes/erigon3-holesky-converted/snapshots/domain/v1-code.0-64.kvei differ
Files eth-nodes/erigon3-holesky-experimental/snapshots/domain/v1-code.64-72.kvei and eth-nodes/erigon3-holesky-converted/snapshots/domain/v1-code.64-72.kvei differ
Files eth-nodes/erigon3-holesky-experimental/snapshots/domain/v1-code.72-74.kvei and eth-nodes/erigon3-holesky-converted/snapshots/domain/v1-code.72-74.kvei differ
Files eth-nodes/erigon3-holesky-experimental/snapshots/domain/v1-commitment.0-64.kv and eth-nodes/erigon3-holesky-converted/snapshots/domain/v1-commitment.0-64.kv differ
Files eth-nodes/erigon3-holesky-experimental/snapshots/domain/v1-commitment.0-64.kvi and eth-nodes/erigon3-holesky-converted/snapshots/domain/v1-commitment.0-64.kvi differ
Files eth-nodes/erigon3-holesky-experimental/snapshots/domain/v1-commitment.64-72.kv and eth-nodes/erigon3-holesky-converted/snapshots/domain/v1-commitment.64-72.kv differ
Files eth-nodes/erigon3-holesky-experimental/snapshots/domain/v1-commitment.64-72.kvi and eth-nodes/erigon3-holesky-converted/snapshots/domain/v1-commitment.64-72.kvi differ
Files eth-nodes/erigon3-holesky-experimental/snapshots/domain/v1-commitment.72-74.kv and eth-nodes/erigon3-holesky-converted/snapshots/domain/v1-commitment.72-74.kv differ
Files eth-nodes/erigon3-holesky-experimental/snapshots/domain/v1-commitment.72-74.kvi and eth-nodes/erigon3-holesky-converted/snapshots/domain/v1-commitment.72-74.kvi differ
Files eth-nodes/erigon3-holesky-experimental/snapshots/domain/v1-receipt.0-64.kvei and eth-nodes/erigon3-holesky-converted/snapshots/domain/v1-receipt.0-64.kvei differ
Files eth-nodes/erigon3-holesky-experimental/snapshots/domain/v1-receipt.64-72.kvei and eth-nodes/erigon3-holesky-converted/snapshots/domain/v1-receipt.64-72.kvei differ
Files eth-nodes/erigon3-holesky-experimental/snapshots/domain/v1-receipt.72-74.kvei and eth-nodes/erigon3-holesky-converted/snapshots/domain/v1-receipt.72-74.kvei differ
Files eth-nodes/erigon3-holesky-experimental/snapshots/domain/v1-storage.0-64.bt and eth-nodes/erigon3-holesky-converted/snapshots/domain/v1-storage.0-64.bt differ
Files eth-nodes/erigon3-holesky-experimental/snapshots/domain/v1-storage.0-64.kvei and eth-nodes/erigon3-holesky-converted/snapshots/domain/v1-storage.0-64.kvei differ
Files eth-nodes/erigon3-holesky-experimental/snapshots/domain/v1-storage.64-72.kvei and eth-nodes/erigon3-holesky-converted/snapshots/domain/v1-storage.64-72.kvei differ
Files eth-nodes/erigon3-holesky-experimental/snapshots/domain/v1-storage.72-74.kvei and eth-nodes/erigon3-holesky-converted/snapshots/domain/v1-storage.72-74.kvei differ
Files eth-nodes/erigon3-holesky-experimental/snapshots/history/v1-receipt.0-64.v and eth-nodes/erigon3-holesky-converted/snapshots/history/v1-receipt.0-64.v differ
Files eth-nodes/erigon3-holesky-experimental/snapshots/history/v1-receipt.72-74.v and eth-nodes/erigon3-holesky-converted/snapshots/history/v1-receipt.72-74.v differ
Files eth-nodes/erigon3-holesky-experimental/snapshots/idx/v1-receipt.0-64.ef and eth-nodes/erigon3-holesky-converted/snapshots/idx/v1-receipt.0-64.ef differ
Files eth-nodes/erigon3-holesky-experimental/snapshots/idx/v1-receipt.72-74.ef and eth-nodes/erigon3-holesky-converted/snapshots/idx/v1-receipt.72-74.ef differ
Only in eth-nodes/erigon3-holesky-converted/snapshots: prohibit_new_downloads.lock
Files eth-nodes/erigon3-holesky-experimental/snapshots/salt-blocks.txt and eth-nodes/erigon3-holesky-converted/snapshots/salt-blocks.txt differ
Files eth-nodes/erigon3-holesky-experimental/snapshots/v1-000000-000100-bodies.idx and eth-nodes/erigon3-holesky-converted/snapshots/v1-000000-000100-bodies.idx differ
Files eth-nodes/erigon3-holesky-experimental/snapshots/v1-000000-000100-bodies.seg and eth-nodes/erigon3-holesky-converted/snapshots/v1-000000-000100-bodies.seg differ
Files eth-nodes/erigon3-holesky-experimental/snapshots/v1-000000-000100-headers.idx and eth-nodes/erigon3-holesky-converted/snapshots/v1-000000-000100-headers.idx differ
Files eth-nodes/erigon3-holesky-experimental/snapshots/v1-000000-000100-transactions-to-block.idx and eth-nodes/erigon3-holesky-converted/snapshots/v1-000000-000100-transactions-to-block.idx differ
Files eth-nodes/erigon3-holesky-experimental/snapshots/v1-000000-000100-transactions.idx and eth-nodes/erigon3-holesky-converted/snapshots/v1-000000-000100-transactions.idx differ
Files eth-nodes/erigon3-holesky-experimental/snapshots/v1-000100-000200-bodies.idx and eth-nodes/erigon3-holesky-converted/snapshots/v1-000100-000200-bodies.idx differ
Files eth-nodes/erigon3-holesky-experimental/snapshots/v1-000100-000200-headers.idx and eth-nodes/erigon3-holesky-converted/snapshots/v1-000100-000200-headers.idx differ
Files eth-nodes/erigon3-holesky-experimental/snapshots/v1-000100-000200-transactions-to-block.idx and eth-nodes/erigon3-holesky-converted/snapshots/v1-000100-000200-transactions-to-block.idx differ
Files eth-nodes/erigon3-holesky-experimental/snapshots/v1-000100-000200-transactions.idx and eth-nodes/erigon3-holesky-converted/snapshots/v1-000100-000200-transactions.idx differ
Files eth-nodes/erigon3-holesky-experimental/snapshots/v1-000200-000300-bodies.idx and eth-nodes/erigon3-holesky-converted/snapshots/v1-000200-000300-bodies.idx differ
Files eth-nodes/erigon3-holesky-experimental/snapshots/v1-000200-000300-headers.idx and eth-nodes/erigon3-holesky-converted/snapshots/v1-000200-000300-headers.idx differ
Files eth-nodes/erigon3-holesky-experimental/snapshots/v1-000200-000300-transactions-to-block.idx and eth-nodes/erigon3-holesky-converted/snapshots/v1-000200-000300-transactions-to-block.idx differ
Files eth-nodes/erigon3-holesky-experimental/snapshots/v1-000200-000300-transactions.idx and eth-nodes/erigon3-holesky-converted/snapshots/v1-000200-000300-transactions.idx differ
Files eth-nodes/erigon3-holesky-experimental/snapshots/v1-000300-000400-bodies.idx and eth-nodes/erigon3-holesky-converted/snapshots/v1-000300-000400-bodies.idx differ
Files eth-nodes/erigon3-holesky-experimental/snapshots/v1-000300-000400-headers.idx and eth-nodes/erigon3-holesky-converted/snapshots/v1-000300-000400-headers.idx differ
Files eth-nodes/erigon3-holesky-experimental/snapshots/v1-000300-000400-transactions-to-block.idx and eth-nodes/erigon3-holesky-converted/snapshots/v1-000300-000400-transactions-to-block.idx differ
Files eth-nodes/erigon3-holesky-experimental/snapshots/v1-000300-000400-transactions.idx and eth-nodes/erigon3-holesky-converted/snapshots/v1-000300-000400-transactions.idx differ
Files eth-nodes/erigon3-holesky-experimental/snapshots/v1-000400-000500-bodies.idx and eth-nodes/erigon3-holesky-converted/snapshots/v1-000400-000500-bodies.idx differ
Files eth-nodes/erigon3-holesky-experimental/snapshots/v1-000400-000500-headers.idx and eth-nodes/erigon3-holesky-converted/snapshots/v1-000400-000500-headers.idx differ
Files eth-nodes/erigon3-holesky-experimental/snapshots/v1-000400-000500-transactions-to-block.idx and eth-nodes/erigon3-holesky-converted/snapshots/v1-000400-000500-transactions-to-block.idx differ
Files eth-nodes/erigon3-holesky-experimental/snapshots/v1-000400-000500-transactions.idx and eth-nodes/erigon3-holesky-converted/snapshots/v1-000400-000500-transactions.idx differ
Files eth-nodes/erigon3-holesky-experimental/snapshots/v1-000500-000600-bodies.idx and eth-nodes/erigon3-holesky-converted/snapshots/v1-000500-000600-bodies.idx differ
Files eth-nodes/erigon3-holesky-experimental/snapshots/v1-000500-000600-headers.idx and eth-nodes/erigon3-holesky-converted/snapshots/v1-000500-000600-headers.idx differ
Files eth-nodes/erigon3-holesky-experimental/snapshots/v1-000500-000600-transactions-to-block.idx and eth-nodes/erigon3-holesky-converted/snapshots/v1-000500-000600-transactions-to-block.idx differ
Files eth-nodes/erigon3-holesky-experimental/snapshots/v1-000500-000600-transactions.idx and eth-nodes/erigon3-holesky-converted/snapshots/v1-000500-000600-transactions.idx differ
Files eth-nodes/erigon3-holesky-experimental/snapshots/v1-000600-000700-bodies.idx and eth-nodes/erigon3-holesky-converted/snapshots/v1-000600-000700-bodies.idx differ
Files eth-nodes/erigon3-holesky-experimental/snapshots/v1-000600-000700-headers.idx and eth-nodes/erigon3-holesky-converted/snapshots/v1-000600-000700-headers.idx differ
Files eth-nodes/erigon3-holesky-experimental/snapshots/v1-000600-000700-transactions-to-block.idx and eth-nodes/erigon3-holesky-converted/snapshots/v1-000600-000700-transactions-to-block.idx differ
Files eth-nodes/erigon3-holesky-experimental/snapshots/v1-000600-000700-transactions.idx and eth-nodes/erigon3-holesky-converted/snapshots/v1-000600-000700-transactions.idx differ
Files eth-nodes/erigon3-holesky-experimental/snapshots/v1-000700-000800-bodies.idx and eth-nodes/erigon3-holesky-converted/snapshots/v1-000700-000800-bodies.idx differ
Files eth-nodes/erigon3-holesky-experimental/snapshots/v1-000700-000800-headers.idx and eth-nodes/erigon3-holesky-converted/snapshots/v1-000700-000800-headers.idx differ
Files eth-nodes/erigon3-holesky-experimental/snapshots/v1-000700-000800-transactions-to-block.idx and eth-nodes/erigon3-holesky-converted/snapshots/v1-000700-000800-transactions-to-block.idx differ
Files eth-nodes/erigon3-holesky-experimental/snapshots/v1-000700-000800-transactions.idx and eth-nodes/erigon3-holesky-converted/snapshots/v1-000700-000800-transactions.idx differ
Files eth-nodes/erigon3-holesky-experimental/snapshots/v1-000800-000900-bodies.idx and eth-nodes/erigon3-holesky-converted/snapshots/v1-000800-000900-bodies.idx differ
Files eth-nodes/erigon3-holesky-experimental/snapshots/v1-000800-000900-headers.idx and eth-nodes/erigon3-holesky-converted/snapshots/v1-000800-000900-headers.idx differ
Files eth-nodes/erigon3-holesky-experimental/snapshots/v1-000800-000900-transactions-to-block.idx and eth-nodes/erigon3-holesky-converted/snapshots/v1-000800-000900-transactions-to-block.idx differ
Files eth-nodes/erigon3-holesky-experimental/snapshots/v1-000800-000900-transactions.idx and eth-nodes/erigon3-holesky-converted/snapshots/v1-000800-000900-transactions.idx differ
Files eth-nodes/erigon3-holesky-experimental/snapshots/v1-000900-001000-bodies.idx and eth-nodes/erigon3-holesky-converted/snapshots/v1-000900-001000-bodies.idx differ
Files eth-nodes/erigon3-holesky-experimental/snapshots/v1-000900-001000-headers.idx and eth-nodes/erigon3-holesky-converted/snapshots/v1-000900-001000-headers.idx differ
Files eth-nodes/erigon3-holesky-experimental/snapshots/v1-000900-001000-transactions-to-block.idx and eth-nodes/erigon3-holesky-converted/snapshots/v1-000900-001000-transactions-to-block.idx differ
Files eth-nodes/erigon3-holesky-experimental/snapshots/v1-000900-001000-transactions.idx and eth-nodes/erigon3-holesky-converted/snapshots/v1-000900-001000-transactions.idx differ
Files eth-nodes/erigon3-holesky-experimental/snapshots/v1-001000-001100-bodies.idx and eth-nodes/erigon3-holesky-converted/snapshots/v1-001000-001100-bodies.idx differ
Files eth-nodes/erigon3-holesky-experimental/snapshots/v1-001000-001100-headers.idx and eth-nodes/erigon3-holesky-converted/snapshots/v1-001000-001100-headers.idx differ
Files eth-nodes/erigon3-holesky-experimental/snapshots/v1-001000-001100-transactions-to-block.idx and eth-nodes/erigon3-holesky-converted/snapshots/v1-001000-001100-transactions-to-block.idx differ
Files eth-nodes/erigon3-holesky-experimental/snapshots/v1-001000-001100-transactions.idx and eth-nodes/erigon3-holesky-converted/snapshots/v1-001000-001100-transactions.idx differ
Files eth-nodes/erigon3-holesky-experimental/snapshots/v1-001100-001200-bodies.idx and eth-nodes/erigon3-holesky-converted/snapshots/v1-001100-001200-bodies.idx differ
Files eth-nodes/erigon3-holesky-experimental/snapshots/v1-001100-001200-headers.idx and eth-nodes/erigon3-holesky-converted/snapshots/v1-001100-001200-headers.idx differ
Files eth-nodes/erigon3-holesky-experimental/snapshots/v1-001100-001200-transactions-to-block.idx and eth-nodes/erigon3-holesky-converted/snapshots/v1-001100-001200-transactions-to-block.idx differ
Files eth-nodes/erigon3-holesky-experimental/snapshots/v1-001100-001200-transactions.idx and eth-nodes/erigon3-holesky-converted/snapshots/v1-001100-001200-transactions.idx differ
Files eth-nodes/erigon3-holesky-experimental/snapshots/v1-001200-001300-bodies.idx and eth-nodes/erigon3-holesky-converted/snapshots/v1-001200-001300-bodies.idx differ
Files eth-nodes/erigon3-holesky-experimental/snapshots/v1-001200-001300-headers.idx and eth-nodes/erigon3-holesky-converted/snapshots/v1-001200-001300-headers.idx differ
Files eth-nodes/erigon3-holesky-experimental/snapshots/v1-001200-001300-transactions-to-block.idx and eth-nodes/erigon3-holesky-converted/snapshots/v1-001200-001300-transactions-to-block.idx differ
Files eth-nodes/erigon3-holesky-experimental/snapshots/v1-001200-001300-transactions.idx and eth-nodes/erigon3-holesky-converted/snapshots/v1-001200-001300-transactions.idx differ
Files eth-nodes/erigon3-holesky-experimental/snapshots/v1-001300-001400-bodies.idx and eth-nodes/erigon3-holesky-converted/snapshots/v1-001300-001400-bodies.idx differ
Files eth-nodes/erigon3-holesky-experimental/snapshots/v1-001300-001400-headers.idx and eth-nodes/erigon3-holesky-converted/snapshots/v1-001300-001400-headers.idx differ
Files eth-nodes/erigon3-holesky-experimental/snapshots/v1-001300-001400-transactions-to-block.idx and eth-nodes/erigon3-holesky-converted/snapshots/v1-001300-001400-transactions-to-block.idx differ
Files eth-nodes/erigon3-holesky-experimental/snapshots/v1-001300-001400-transactions.idx and eth-nodes/erigon3-holesky-converted/snapshots/v1-001300-001400-transactions.idx differ
Files eth-nodes/erigon3-holesky-experimental/snapshots/v1-001400-001500-bodies.idx and eth-nodes/erigon3-holesky-converted/snapshots/v1-001400-001500-bodies.idx differ
Files eth-nodes/erigon3-holesky-experimental/snapshots/v1-001400-001500-headers.idx and eth-nodes/erigon3-holesky-converted/snapshots/v1-001400-001500-headers.idx differ
Files eth-nodes/erigon3-holesky-experimental/snapshots/v1-001400-001500-transactions-to-block.idx and eth-nodes/erigon3-holesky-converted/snapshots/v1-001400-001500-transactions-to-block.idx differ
Files eth-nodes/erigon3-holesky-experimental/snapshots/v1-001400-001500-transactions.idx and eth-nodes/erigon3-holesky-converted/snapshots/v1-001400-001500-transactions.idx differ
Files eth-nodes/erigon3-holesky-experimental/snapshots/v1-001500-001600-bodies.idx and eth-nodes/erigon3-holesky-converted/snapshots/v1-001500-001600-bodies.idx differ
Files eth-nodes/erigon3-holesky-experimental/snapshots/v1-001500-001600-headers.idx and eth-nodes/erigon3-holesky-converted/snapshots/v1-001500-001600-headers.idx differ
Files eth-nodes/erigon3-holesky-experimental/snapshots/v1-001500-001600-transactions-to-block.idx and eth-nodes/erigon3-holesky-converted/snapshots/v1-001500-001600-transactions-to-block.idx differ
Files eth-nodes/erigon3-holesky-experimental/snapshots/v1-001500-001600-transactions.idx and eth-nodes/erigon3-holesky-converted/snapshots/v1-001500-001600-transactions.idx differ
Files eth-nodes/erigon3-holesky-experimental/snapshots/v1-001600-001700-bodies.idx and eth-nodes/erigon3-holesky-converted/snapshots/v1-001600-001700-bodies.idx differ
Files eth-nodes/erigon3-holesky-experimental/snapshots/v1-001600-001700-headers.idx and eth-nodes/erigon3-holesky-converted/snapshots/v1-001600-001700-headers.idx differ
Files eth-nodes/erigon3-holesky-experimental/snapshots/v1-001600-001700-transactions-to-block.idx and eth-nodes/erigon3-holesky-converted/snapshots/v1-001600-001700-transactions-to-block.idx differ
Files eth-nodes/erigon3-holesky-experimental/snapshots/v1-001600-001700-transactions.idx and eth-nodes/erigon3-holesky-converted/snapshots/v1-001600-001700-transactions.idx differ
Files eth-nodes/erigon3-holesky-experimental/snapshots/v1-001700-001800-bodies.idx and eth-nodes/erigon3-holesky-converted/snapshots/v1-001700-001800-bodies.idx differ
Files eth-nodes/erigon3-holesky-experimental/snapshots/v1-001700-001800-headers.idx and eth-nodes/erigon3-holesky-converted/snapshots/v1-001700-001800-headers.idx differ
Files eth-nodes/erigon3-holesky-experimental/snapshots/v1-001700-001800-transactions-to-block.idx and eth-nodes/erigon3-holesky-converted/snapshots/v1-001700-001800-transactions-to-block.idx differ
Files eth-nodes/erigon3-holesky-experimental/snapshots/v1-001700-001800-transactions.idx and eth-nodes/erigon3-holesky-converted/snapshots/v1-001700-001800-transactions.idx differ
Files eth-nodes/erigon3-holesky-experimental/snapshots/v1-001800-001900-bodies.idx and eth-nodes/erigon3-holesky-converted/snapshots/v1-001800-001900-bodies.idx differ
Files eth-nodes/erigon3-holesky-experimental/snapshots/v1-001800-001900-headers.idx and eth-nodes/erigon3-holesky-converted/snapshots/v1-001800-001900-headers.idx differ
Files eth-nodes/erigon3-holesky-experimental/snapshots/v1-001800-001900-transactions-to-block.idx and eth-nodes/erigon3-holesky-converted/snapshots/v1-001800-001900-transactions-to-block.idx differ
Files eth-nodes/erigon3-holesky-experimental/snapshots/v1-001800-001900-transactions.idx and eth-nodes/erigon3-holesky-converted/snapshots/v1-001800-001900-transactions.idx differ
Files eth-nodes/erigon3-holesky-experimental/snapshots/v1-001900-002000-bodies.idx and eth-nodes/erigon3-holesky-converted/snapshots/v1-001900-002000-bodies.idx differ
Files eth-nodes/erigon3-holesky-experimental/snapshots/v1-001900-002000-headers.idx and eth-nodes/erigon3-holesky-converted/snapshots/v1-001900-002000-headers.idx differ
Files eth-nodes/erigon3-holesky-experimental/snapshots/v1-001900-002000-transactions-to-block.idx and eth-nodes/erigon3-holesky-converted/snapshots/v1-001900-002000-transactions-to-block.idx differ
Files eth-nodes/erigon3-holesky-experimental/snapshots/v1-001900-002000-transactions.idx and eth-nodes/erigon3-holesky-converted/snapshots/v1-001900-002000-transactions.idx differ
Files eth-nodes/erigon3-holesky-experimental/snapshots/v1-002000-002100-bodies.idx and eth-nodes/erigon3-holesky-converted/snapshots/v1-002000-002100-bodies.idx differ
Files eth-nodes/erigon3-holesky-experimental/snapshots/v1-002000-002100-headers.idx and eth-nodes/erigon3-holesky-converted/snapshots/v1-002000-002100-headers.idx differ
Files eth-nodes/erigon3-holesky-experimental/snapshots/v1-002000-002100-transactions-to-block.idx and eth-nodes/erigon3-holesky-converted/snapshots/v1-002000-002100-transactions-to-block.idx differ
Files eth-nodes/erigon3-holesky-experimental/snapshots/v1-002000-002100-transactions.idx and eth-nodes/erigon3-holesky-converted/snapshots/v1-002000-002100-transactions.idx differ
Files eth-nodes/erigon3-holesky-experimental/snapshots/v1-002100-002200-bodies.idx and eth-nodes/erigon3-holesky-converted/snapshots/v1-002100-002200-bodies.idx differ
Files eth-nodes/erigon3-holesky-experimental/snapshots/v1-002100-002200-headers.idx and eth-nodes/erigon3-holesky-converted/snapshots/v1-002100-002200-headers.idx differ
Files eth-nodes/erigon3-holesky-experimental/snapshots/v1-002100-002200-transactions-to-block.idx and eth-nodes/erigon3-holesky-converted/snapshots/v1-002100-002200-transactions-to-block.idx differ
Files eth-nodes/erigon3-holesky-experimental/snapshots/v1-002100-002200-transactions.idx and eth-nodes/erigon3-holesky-converted/snapshots/v1-002100-002200-transactions.idx differ
Files eth-nodes/erigon3-holesky-experimental/snapshots/v1-002200-002300-bodies.idx and eth-nodes/erigon3-holesky-converted/snapshots/v1-002200-002300-bodies.idx differ
Files eth-nodes/erigon3-holesky-experimental/snapshots/v1-002200-002300-headers.idx and eth-nodes/erigon3-holesky-converted/snapshots/v1-002200-002300-headers.idx differ
Files eth-nodes/erigon3-holesky-experimental/snapshots/v1-002200-002300-transactions-to-block.idx and eth-nodes/erigon3-holesky-converted/snapshots/v1-002200-002300-transactions-to-block.idx differ
Files eth-nodes/erigon3-holesky-experimental/snapshots/v1-002200-002300-transactions.idx and eth-nodes/erigon3-holesky-converted/snapshots/v1-002200-002300-transactions.idx differ
Files eth-nodes/erigon3-holesky-experimental/snapshots/v1-002300-002400-bodies.idx and eth-nodes/erigon3-holesky-converted/snapshots/v1-002300-002400-bodies.idx differ
Files eth-nodes/erigon3-holesky-experimental/snapshots/v1-002300-002400-headers.idx and eth-nodes/erigon3-holesky-converted/snapshots/v1-002300-002400-headers.idx differ
Files eth-nodes/erigon3-holesky-experimental/snapshots/v1-002300-002400-transactions-to-block.idx and eth-nodes/erigon3-holesky-converted/snapshots/v1-002300-002400-transactions-to-block.idx differ
Files eth-nodes/erigon3-holesky-experimental/snapshots/v1-002300-002400-transactions.idx and eth-nodes/erigon3-holesky-converted/snapshots/v1-002300-002400-transactions.idx differ
Files eth-nodes/erigon3-holesky-experimental/snapshots/v1-002400-002500-bodies.idx and eth-nodes/erigon3-holesky-converted/snapshots/v1-002400-002500-bodies.idx differ
Files eth-nodes/erigon3-holesky-experimental/snapshots/v1-002400-002500-headers.idx and eth-nodes/erigon3-holesky-converted/snapshots/v1-002400-002500-headers.idx differ
Files eth-nodes/erigon3-holesky-experimental/snapshots/v1-002400-002500-transactions-to-block.idx and eth-nodes/erigon3-holesky-converted/snapshots/v1-002400-002500-transactions-to-block.idx differ
Files eth-nodes/erigon3-holesky-experimental/snapshots/v1-002400-002500-transactions.idx and eth-nodes/erigon3-holesky-converted/snapshots/v1-002400-002500-transactions.idx differ
Files eth-nodes/erigon3-holesky-experimental/snapshots/v1-002500-002600-bodies.idx and eth-nodes/erigon3-holesky-converted/snapshots/v1-002500-002600-bodies.idx differ
Files eth-nodes/erigon3-holesky-experimental/snapshots/v1-002500-002600-headers.idx and eth-nodes/erigon3-holesky-converted/snapshots/v1-002500-002600-headers.idx differ
Files eth-nodes/erigon3-holesky-experimental/snapshots/v1-002500-002600-transactions-to-block.idx and eth-nodes/erigon3-holesky-converted/snapshots/v1-002500-002600-transactions-to-block.idx differ
Files eth-nodes/erigon3-holesky-experimental/snapshots/v1-002500-002600-transactions.idx and eth-nodes/erigon3-holesky-converted/snapshots/v1-002500-002600-transactions.idx differ
Files eth-nodes/erigon3-holesky-experimental/snapshots/v1-002600-002700-bodies.idx and eth-nodes/erigon3-holesky-converted/snapshots/v1-002600-002700-bodies.idx differ
Files eth-nodes/erigon3-holesky-experimental/snapshots/v1-002600-002700-headers.idx and eth-nodes/erigon3-holesky-converted/snapshots/v1-002600-002700-headers.idx differ
Files eth-nodes/erigon3-holesky-experimental/snapshots/v1-002600-002700-transactions-to-block.idx and eth-nodes/erigon3-holesky-converted/snapshots/v1-002600-002700-transactions-to-block.idx differ
Files eth-nodes/erigon3-holesky-experimental/snapshots/v1-002600-002700-transactions.idx and eth-nodes/erigon3-holesky-converted/snapshots/v1-002600-002700-transactions.idx differ
Files eth-nodes/erigon3-holesky-experimental/snapshots/v1-002700-002800-bodies.idx and eth-nodes/erigon3-holesky-converted/snapshots/v1-002700-002800-bodies.idx differ
Files eth-nodes/erigon3-holesky-experimental/snapshots/v1-002700-002800-headers.idx and eth-nodes/erigon3-holesky-converted/snapshots/v1-002700-002800-headers.idx differ
Files eth-nodes/erigon3-holesky-experimental/snapshots/v1-002700-002800-transactions-to-block.idx and eth-nodes/erigon3-holesky-converted/snapshots/v1-002700-002800-transactions-to-block.idx differ
Files eth-nodes/erigon3-holesky-experimental/snapshots/v1-002700-002800-transactions.idx and eth-nodes/erigon3-holesky-converted/snapshots/v1-002700-002800-transactions.idx differ
Files eth-nodes/erigon3-holesky-experimental/snapshots/v1-002800-002900-bodies.idx and eth-nodes/erigon3-holesky-converted/snapshots/v1-002800-002900-bodies.idx differ
Files eth-nodes/erigon3-holesky-experimental/snapshots/v1-002800-002900-headers.idx and eth-nodes/erigon3-holesky-converted/snapshots/v1-002800-002900-headers.idx differ
Files eth-nodes/erigon3-holesky-experimental/snapshots/v1-002800-002900-transactions-to-block.idx and eth-nodes/erigon3-holesky-converted/snapshots/v1-002800-002900-transactions-to-block.idx differ
Files eth-nodes/erigon3-holesky-experimental/snapshots/v1-002800-002900-transactions.idx and eth-nodes/erigon3-holesky-converted/snapshots/v1-002800-002900-transactions.idx differ
Files eth-nodes/erigon3-holesky-experimental/snapshots/v1-002900-003000-bodies.idx and eth-nodes/erigon3-holesky-converted/snapshots/v1-002900-003000-bodies.idx differ
Files eth-nodes/erigon3-holesky-experimental/snapshots/v1-002900-003000-headers.idx and eth-nodes/erigon3-holesky-converted/snapshots/v1-002900-003000-headers.idx differ
Files eth-nodes/erigon3-holesky-experimental/snapshots/v1-002900-003000-transactions-to-block.idx and eth-nodes/erigon3-holesky-converted/snapshots/v1-002900-003000-transactions-to-block.idx differ
Files eth-nodes/erigon3-holesky-experimental/snapshots/v1-002900-003000-transactions.idx and eth-nodes/erigon3-holesky-converted/snapshots/v1-002900-003000-transactions.idx differ
Files eth-nodes/erigon3-holesky-experimental/snapshots/v1-003000-003100-bodies.idx and eth-nodes/erigon3-holesky-converted/snapshots/v1-003000-003100-bodies.idx differ
Files eth-nodes/erigon3-holesky-experimental/snapshots/v1-003000-003100-headers.idx and eth-nodes/erigon3-holesky-converted/snapshots/v1-003000-003100-headers.idx differ
Files eth-nodes/erigon3-holesky-experimental/snapshots/v1-003000-003100-transactions-to-block.idx and eth-nodes/erigon3-holesky-converted/snapshots/v1-003000-003100-transactions-to-block.idx differ
Files eth-nodes/erigon3-holesky-experimental/snapshots/v1-003000-003100-transactions.idx and eth-nodes/erigon3-holesky-converted/snapshots/v1-003000-003100-transactions.idx differ
Files eth-nodes/erigon3-holesky-experimental/snapshots/v1-003100-003110-bodies.idx and eth-nodes/erigon3-holesky-converted/snapshots/v1-003100-003110-bodies.idx differ
Files eth-nodes/erigon3-holesky-experimental/snapshots/v1-003100-003110-headers.idx and eth-nodes/erigon3-holesky-converted/snapshots/v1-003100-003110-headers.idx differ
Files eth-nodes/erigon3-holesky-experimental/snapshots/v1-003100-003110-transactions-to-block.idx and eth-nodes/erigon3-holesky-converted/snapshots/v1-003100-003110-transactions-to-block.idx differ
Files eth-nodes/erigon3-holesky-experimental/snapshots/v1-003100-003110-transactions.idx and eth-nodes/erigon3-holesky-converted/snapshots/v1-003100-003110-transactions.idx differ
Files eth-nodes/erigon3-holesky-experimental/snapshots/v1-003110-003120-bodies.idx and eth-nodes/erigon3-holesky-converted/snapshots/v1-003110-003120-bodies.idx differ
Files eth-nodes/erigon3-holesky-experimental/snapshots/v1-003110-003120-headers.idx and eth-nodes/erigon3-holesky-converted/snapshots/v1-003110-003120-headers.idx differ
Files eth-nodes/erigon3-holesky-experimental/snapshots/v1-003110-003120-transactions-to-block.idx and eth-nodes/erigon3-holesky-converted/snapshots/v1-003110-003120-transactions-to-block.idx differ
Files eth-nodes/erigon3-holesky-experimental/snapshots/v1-003110-003120-transactions.idx and eth-nodes/erigon3-holesky-converted/snapshots/v1-003110-003120-transactions.idx differ
Files eth-nodes/erigon3-holesky-experimental/snapshots/v1-003120-003130-bodies.idx and eth-nodes/erigon3-holesky-converted/snapshots/v1-003120-003130-bodies.idx differ
Files eth-nodes/erigon3-holesky-experimental/snapshots/v1-003120-003130-headers.idx and eth-nodes/erigon3-holesky-converted/snapshots/v1-003120-003130-headers.idx differ
Files eth-nodes/erigon3-holesky-experimental/snapshots/v1-003120-003130-transactions-to-block.idx and eth-nodes/erigon3-holesky-converted/snapshots/v1-003120-003130-transactions-to-block.idx differ
Files eth-nodes/erigon3-holesky-experimental/snapshots/v1-003120-003130-transactions.idx and eth-nodes/erigon3-holesky-converted/snapshots/v1-003120-003130-transactions.idx differ
Files eth-nodes/erigon3-holesky-experimental/snapshots/v1-003130-003140-bodies.idx and eth-nodes/erigon3-holesky-converted/snapshots/v1-003130-003140-bodies.idx differ
Files eth-nodes/erigon3-holesky-experimental/snapshots/v1-003130-003140-headers.idx and eth-nodes/erigon3-holesky-converted/snapshots/v1-003130-003140-headers.idx differ
Files eth-nodes/erigon3-holesky-experimental/snapshots/v1-003130-003140-transactions-to-block.idx and eth-nodes/erigon3-holesky-converted/snapshots/v1-003130-003140-transactions-to-block.idx differ
Files eth-nodes/erigon3-holesky-experimental/snapshots/v1-003130-003140-transactions.idx and eth-nodes/erigon3-holesky-converted/snapshots/v1-003130-003140-transactions.idx differ
Files eth-nodes/erigon3-holesky-experimental/snapshots/v1-003140-003150-bodies.idx and eth-nodes/erigon3-holesky-converted/snapshots/v1-003140-003150-bodies.idx differ
Files eth-nodes/erigon3-holesky-experimental/snapshots/v1-003140-003150-headers.idx and eth-nodes/erigon3-holesky-converted/snapshots/v1-003140-003150-headers.idx differ
Files eth-nodes/erigon3-holesky-experimental/snapshots/v1-003140-003150-transactions-to-block.idx and eth-nodes/erigon3-holesky-converted/snapshots/v1-003140-003150-transactions-to-block.idx differ
Files eth-nodes/erigon3-holesky-experimental/snapshots/v1-003140-003150-transactions.idx and eth-nodes/erigon3-holesky-converted/snapshots/v1-003140-003150-transactions.idx differ
Files eth-nodes/erigon3-holesky-experimental/snapshots/v1-003150-003160-bodies.idx and eth-nodes/erigon3-holesky-converted/snapshots/v1-003150-003160-bodies.idx differ
Files eth-nodes/erigon3-holesky-experimental/snapshots/v1-003150-003160-headers.idx and eth-nodes/erigon3-holesky-converted/snapshots/v1-003150-003160-headers.idx differ
Files eth-nodes/erigon3-holesky-experimental/snapshots/v1-003150-003160-transactions-to-block.idx and eth-nodes/erigon3-holesky-converted/snapshots/v1-003150-003160-transactions-to-block.idx differ
Files eth-nodes/erigon3-holesky-experimental/snapshots/v1-003150-003160-transactions.idx and eth-nodes/erigon3-holesky-converted/snapshots/v1-003150-003160-transactions.idx differ
Files eth-nodes/erigon3-holesky-experimental/snapshots/v1-003160-003170-bodies.idx and eth-nodes/erigon3-holesky-converted/snapshots/v1-003160-003170-bodies.idx differ
Files eth-nodes/erigon3-holesky-experimental/snapshots/v1-003160-003170-headers.idx and eth-nodes/erigon3-holesky-converted/snapshots/v1-003160-003170-headers.idx differ
Files eth-nodes/erigon3-holesky-experimental/snapshots/v1-003160-003170-transactions-to-block.idx and eth-nodes/erigon3-holesky-converted/snapshots/v1-003160-003170-transactions-to-block.idx differ
Files eth-nodes/erigon3-holesky-experimental/snapshots/v1-003160-003170-transactions.idx and eth-nodes/erigon3-holesky-converted/snapshots/v1-003160-003170-transactions.idx differ
Files eth-nodes/erigon3-holesky-experimental/snapshots/v1-003170-003180-bodies.idx and eth-nodes/erigon3-holesky-converted/snapshots/v1-003170-003180-bodies.idx differ
Files eth-nodes/erigon3-holesky-experimental/snapshots/v1-003170-003180-headers.idx and eth-nodes/erigon3-holesky-converted/snapshots/v1-003170-003180-headers.idx differ
Files eth-nodes/erigon3-holesky-experimental/snapshots/v1-003170-003180-transactions-to-block.idx and eth-nodes/erigon3-holesky-converted/snapshots/v1-003170-003180-transactions-to-block.idx differ
Files eth-nodes/erigon3-holesky-experimental/snapshots/v1-003170-003180-transactions.idx and eth-nodes/erigon3-holesky-converted/snapshots/v1-003170-003180-transactions.idx differ
Files eth-nodes/erigon3-holesky-experimental/snapshots/v1-003180-003181-bodies.idx and eth-nodes/erigon3-holesky-converted/snapshots/v1-003180-003181-bodies.idx differ
Files eth-nodes/erigon3-holesky-experimental/snapshots/v1-003180-003181-headers.idx and eth-nodes/erigon3-holesky-converted/snapshots/v1-003180-003181-headers.idx differ
Files eth-nodes/erigon3-holesky-experimental/snapshots/v1-003180-003181-transactions-to-block.idx and eth-nodes/erigon3-holesky-converted/snapshots/v1-003180-003181-transactions-to-block.idx differ
Files eth-nodes/erigon3-holesky-experimental/snapshots/v1-003180-003181-transactions.idx and eth-nodes/erigon3-holesky-converted/snapshots/v1-003180-003181-transactions.idx differ

Jan 20 '25 19:01 wmitsuda

Next steps: another round of sync from scratch, now I'll try to force use the same salt as the snapshots served by Erigon in order to reproduce the same indexes.

@AskAlexSharov @awskii I could use some advice here if there is something obvious I'm not seeing:

Ignoring the fact that the salt is different:
Why does the v1-000000-000100-bodies.seg file (and only this .seg) was generated differently
Why does all v1-commitment-*.kv were generated differently
Why does v1-code.0-64.bt (and only this .bt) was generated differently

Is there any other moving part other than the salt that does not make the generation deterministic?

Jan 20 '25 19:01 wmitsuda

Commitment is not enough determenistic. - it’s known issue. other things - i’m not sure, but probably ok to ignore.

Jan 21 '25 00:01 AskAlexSharov

Commitment is not enough determenistic. - it’s known issue. other things - i’m not sure, but probably ok to ignore.

FYI: according to Giulio on Discord, it is probably due to purification process

Jan 21 '25 22:01 wmitsuda

Status, another round on holesky completed, comparing holesky beta 1 vs. holesky from scratch + PR

Regarding .ef contents, everything matches, except v1-receipt.*. It looks like there are still some bugs on beta 1 that is making contents differ or the published files on beta 1 are outdated (FYI: @JkLondon @AskAlexSharov , pls ignore if it is known issue)

time build/bin/integration idx_verify --sourcedir ~/eth-nodes/erigon3-holesky --targetdir ~/eth-nodes/erigon3-
holesky-experimental
2025/01/22 16:42:36 Comparing idx files:
2025/01/22 16:42:36 Deep checking files /Users/wmitsuda/eth-nodes/erigon3-holesky/snapshots/idx/v1-accounts.0-64.ef -> /Users/wmitsuda/eth-nodes/erigon3-holesky-experimental/snapshots/idx/v1-accounts.0-64.ef, /U
sers/wmitsuda/eth-nodes/erigon3-holesky-experimental/snapshots/accessor/v1-accounts.0-64.efi...
2025/01/22 16:43:39 Deep checking files /Users/wmitsuda/eth-nodes/erigon3-holesky/snapshots/idx/v1-accounts.64-72.ef -> /Users/wmitsuda/eth-nodes/erigon3-holesky-experimental/snapshots/idx/v1-accounts.64-72.ef,
/Users/wmitsuda/eth-nodes/erigon3-holesky-experimental/snapshots/accessor/v1-accounts.64-72.efi...
2025/01/22 16:43:42 Deep checking files /Users/wmitsuda/eth-nodes/erigon3-holesky/snapshots/idx/v1-accounts.72-74.ef -> /Users/wmitsuda/eth-nodes/erigon3-holesky-experimental/snapshots/idx/v1-accounts.72-74.ef,
/Users/wmitsuda/eth-nodes/erigon3-holesky-experimental/snapshots/accessor/v1-accounts.72-74.efi...
2025/01/22 16:43:45 Deep checking files /Users/wmitsuda/eth-nodes/erigon3-holesky/snapshots/idx/v1-code.0-64.ef -> /Users/wmitsuda/eth-nodes/erigon3-holesky-experimental/snapshots/idx/v1-code.0-64.ef, /Users/wmi
tsuda/eth-nodes/erigon3-holesky-experimental/snapshots/accessor/v1-code.0-64.efi...
2025/01/22 16:44:11 Deep checking files /Users/wmitsuda/eth-nodes/erigon3-holesky/snapshots/idx/v1-code.64-72.ef -> /Users/wmitsuda/eth-nodes/erigon3-holesky-experimental/snapshots/idx/v1-code.64-72.ef, /Users/w
mitsuda/eth-nodes/erigon3-holesky-experimental/snapshots/accessor/v1-code.64-72.efi...
2025/01/22 16:44:11 Deep checking files /Users/wmitsuda/eth-nodes/erigon3-holesky/snapshots/idx/v1-code.72-74.ef -> /Users/wmitsuda/eth-nodes/erigon3-holesky-experimental/snapshots/idx/v1-code.72-74.ef, /Users/w
mitsuda/eth-nodes/erigon3-holesky-experimental/snapshots/accessor/v1-code.72-74.efi...
2025/01/22 16:44:14 Deep checking files /Users/wmitsuda/eth-nodes/erigon3-holesky/snapshots/idx/v1-logaddrs.0-64.ef -> /Users/wmitsuda/eth-nodes/erigon3-holesky-experimental/snapshots/idx/v1-logaddrs.0-64.ef, /U
sers/wmitsuda/eth-nodes/erigon3-holesky-experimental/snapshots/accessor/v1-logaddrs.0-64.efi...
2025/01/22 16:44:16 Deep checking files /Users/wmitsuda/eth-nodes/erigon3-holesky/snapshots/idx/v1-logaddrs.64-72.ef -> /Users/wmitsuda/eth-nodes/erigon3-holesky-experimental/snapshots/idx/v1-logaddrs.64-72.ef,
/Users/wmitsuda/eth-nodes/erigon3-holesky-experimental/snapshots/accessor/v1-logaddrs.64-72.efi...
2025/01/22 16:44:17 Deep checking files /Users/wmitsuda/eth-nodes/erigon3-holesky/snapshots/idx/v1-logaddrs.72-74.ef -> /Users/wmitsuda/eth-nodes/erigon3-holesky-experimental/snapshots/idx/v1-logaddrs.72-74.ef,
/Users/wmitsuda/eth-nodes/erigon3-holesky-experimental/snapshots/accessor/v1-logaddrs.72-74.efi...
2025/01/22 16:44:17 Deep checking files /Users/wmitsuda/eth-nodes/erigon3-holesky/snapshots/idx/v1-logtopics.0-64.ef -> /Users/wmitsuda/eth-nodes/erigon3-holesky-experimental/snapshots/idx/v1-logtopics.0-64.ef,
/Users/wmitsuda/eth-nodes/erigon3-holesky-experimental/snapshots/accessor/v1-logtopics.0-64.efi...
2025/01/22 16:45:38 Deep checking files /Users/wmitsuda/eth-nodes/erigon3-holesky/snapshots/idx/v1-logtopics.64-72.ef -> /Users/wmitsuda/eth-nodes/erigon3-holesky-experimental/snapshots/idx/v1-logtopics.64-72.ef
, /Users/wmitsuda/eth-nodes/erigon3-holesky-experimental/snapshots/accessor/v1-logtopics.64-72.efi...
2025/01/22 16:45:43 Deep checking files /Users/wmitsuda/eth-nodes/erigon3-holesky/snapshots/idx/v1-logtopics.72-74.ef -> /Users/wmitsuda/eth-nodes/erigon3-holesky-experimental/snapshots/idx/v1-logtopics.72-74.ef
, /Users/wmitsuda/eth-nodes/erigon3-holesky-experimental/snapshots/accessor/v1-logtopics.72-74.efi...
2025/01/22 16:45:47 Deep checking files /Users/wmitsuda/eth-nodes/erigon3-holesky/snapshots/idx/v1-receipt.0-64.ef -> /Users/wmitsuda/eth-nodes/erigon3-holesky-experimental/snapshots/idx/v1-receipt.0-64.ef, /Use
rs/wmitsuda/eth-nodes/erigon3-holesky-experimental/snapshots/accessor/v1-receipt.0-64.efi...
2025/01/22 16:45:47 Optimized eliasfano is longer
2025/01/22 16:45:47 key=0x00
2025/01/22 16:45:47 source min=0 max=99999999 count=97248454
2025/01/22 16:45:47 target min=0 max=99999999 count=97250285
2025/01/22 16:45:47 value mismatch!
2025/01/22 16:45:47 skipping to next file...
2025/01/22 16:45:47 Deep checking files /Users/wmitsuda/eth-nodes/erigon3-holesky/snapshots/idx/v1-receipt.64-72.ef -> /Users/wmitsuda/eth-nodes/erigon3-holesky-experimental/snapshots/idx/v1-receipt.64-72.ef, /U
sers/wmitsuda/eth-nodes/erigon3-holesky-experimental/snapshots/accessor/v1-receipt.64-72.efi...
2025/01/22 16:45:48 Deep checking files /Users/wmitsuda/eth-nodes/erigon3-holesky/snapshots/idx/v1-receipt.72-74.ef -> /Users/wmitsuda/eth-nodes/erigon3-holesky-experimental/snapshots/idx/v1-receipt.72-74.ef, /U
sers/wmitsuda/eth-nodes/erigon3-holesky-experimental/snapshots/accessor/v1-receipt.72-74.efi...
2025/01/22 16:45:48 values mismatch: source=112500000 target=112500002
2025/01/22 16:45:48 key=0x01
2025/01/22 16:45:48 source min=112500000 max=115624999 count=435158
2025/01/22 16:45:48 target min=112500002 max=115624999 count=435158
2025/01/22 16:45:48 value mismatch!
2025/01/22 16:45:48 skipping to next file...
2025/01/22 16:45:48 Deep checking files /Users/wmitsuda/eth-nodes/erigon3-holesky/snapshots/idx/v1-storage.0-64.ef -> /Users/wmitsuda/eth-nodes/erigon3-holesky-experimental/snapshots/idx/v1-storage.0-64.ef, /Use
rs/wmitsuda/eth-nodes/erigon3-holesky-experimental/snapshots/accessor/v1-storage.0-64.efi...
2025/01/22 16:48:08 Deep checking files /Users/wmitsuda/eth-nodes/erigon3-holesky/snapshots/idx/v1-storage.64-72.ef -> /Users/wmitsuda/eth-nodes/erigon3-holesky-experimental/snapshots/idx/v1-storage.64-72.ef, /U
sers/wmitsuda/eth-nodes/erigon3-holesky-experimental/snapshots/accessor/v1-storage.64-72.efi...
2025/01/22 16:49:02 Deep checking files /Users/wmitsuda/eth-nodes/erigon3-holesky/snapshots/idx/v1-storage.72-74.ef -> /Users/wmitsuda/eth-nodes/erigon3-holesky-experimental/snapshots/idx/v1-storage.72-74.ef, /U
sers/wmitsuda/eth-nodes/erigon3-holesky-experimental/snapshots/accessor/v1-storage.72-74.efi...
2025/01/22 16:49:14 Deep checking files /Users/wmitsuda/eth-nodes/erigon3-holesky/snapshots/idx/v1-tracesfrom.0-64.ef -> /Users/wmitsuda/eth-nodes/erigon3-holesky-experimental/snapshots/idx/v1-tracesfrom.0-64.ef
, /Users/wmitsuda/eth-nodes/erigon3-holesky-experimental/snapshots/accessor/v1-tracesfrom.0-64.efi...
2025/01/22 16:50:08 Deep checking files /Users/wmitsuda/eth-nodes/erigon3-holesky/snapshots/idx/v1-tracesfrom.64-72.ef -> /Users/wmitsuda/eth-nodes/erigon3-holesky-experimental/snapshots/idx/v1-tracesfrom.64-72.
ef, /Users/wmitsuda/eth-nodes/erigon3-holesky-experimental/snapshots/accessor/v1-tracesfrom.64-72.efi...
2025/01/22 16:50:10 Deep checking files /Users/wmitsuda/eth-nodes/erigon3-holesky/snapshots/idx/v1-tracesfrom.72-74.ef -> /Users/wmitsuda/eth-nodes/erigon3-holesky-experimental/snapshots/idx/v1-tracesfrom.72-74.
ef, /Users/wmitsuda/eth-nodes/erigon3-holesky-experimental/snapshots/accessor/v1-tracesfrom.72-74.efi...
2025/01/22 16:50:14 Deep checking files /Users/wmitsuda/eth-nodes/erigon3-holesky/snapshots/idx/v1-tracesto.0-64.ef -> /Users/wmitsuda/eth-nodes/erigon3-holesky-experimental/snapshots/idx/v1-tracesto.0-64.ef, /U
sers/wmitsuda/eth-nodes/erigon3-holesky-experimental/snapshots/accessor/v1-tracesto.0-64.efi...
2025/01/22 16:51:21 Deep checking files /Users/wmitsuda/eth-nodes/erigon3-holesky/snapshots/idx/v1-tracesto.64-72.ef -> /Users/wmitsuda/eth-nodes/erigon3-holesky-experimental/snapshots/idx/v1-tracesto.64-72.ef,
/Users/wmitsuda/eth-nodes/erigon3-holesky-experimental/snapshots/accessor/v1-tracesto.64-72.efi...
2025/01/22 16:51:23 Deep checking files /Users/wmitsuda/eth-nodes/erigon3-holesky/snapshots/idx/v1-tracesto.72-74.ef -> /Users/wmitsuda/eth-nodes/erigon3-holesky-experimental/snapshots/idx/v1-tracesto.72-74.ef,
/Users/wmitsuda/eth-nodes/erigon3-holesky-experimental/snapshots/accessor/v1-tracesto.72-74.efi...
build/bin/integration idx_verify --sourcedir ~/eth-nodes/erigon3-holesky    519.57s user 55.97s system 108% cpu 8:49.97 total

Jan 22 '25 22:01 wmitsuda

Regarding comparing converted holesky vs. generated holesky (only snapshots dir), it seems now everything matches, except:

v1-receipt.* -> probably due to concerns I mentioned in the previous comment
v1-commitment.* -> will ignore, known as not deterministic
domains/v1-*.0-64.bt -> strange, all domains, only the 0-64 range .bt doesn't match
*.kvei -> not sure it is not deterministic, this run I made sure to copy salts before the execution, maybe there is something else that is making it not deterministic.
There are a few transactions.idx (only the latest ones) which are not being deterministically regenerated. No idea why, salt files were copied, and all < 3100- files were regenerated correctly. The change in this PR has nothing to do with transactions .idx, apparently safe to ignore, but not sure why the diff...

diff -rq ~/eth-nodes/erigon3-holesky-converted/snapshots ~/eth-nodes/erigon3-holesky-experimental/snapshots |
grep -v torrent | grep -v \.efi | grep -v \.ef
Files /Users/wmitsuda/eth-nodes/erigon3-holesky-converted/snapshots/accessor/v1-receipt.0-64.vi and /Users/wmitsuda/eth-nodes/erigon3-holesky-experimental/snapshots/accessor/v1-receipt.0-64.vi differ
Files /Users/wmitsuda/eth-nodes/erigon3-holesky-converted/snapshots/accessor/v1-receipt.72-74.vi and /Users/wmitsuda/eth-nodes/erigon3-holesky-experimental/snapshots/accessor/v1-receipt.72-74.vi differ
Files /Users/wmitsuda/eth-nodes/erigon3-holesky-converted/snapshots/domain/v1-accounts.0-64.bt and /Users/wmitsuda/eth-nodes/erigon3-holesky-experimental/snapshots/domain/v1-accounts.0-64.bt differ
Files /Users/wmitsuda/eth-nodes/erigon3-holesky-converted/snapshots/domain/v1-accounts.0-64.kvei and /Users/wmitsuda/eth-nodes/erigon3-holesky-experimental/snapshots/domain/v1-accounts.0-64.kvei differ
Files /Users/wmitsuda/eth-nodes/erigon3-holesky-converted/snapshots/domain/v1-accounts.64-72.kvei and /Users/wmitsuda/eth-nodes/erigon3-holesky-experimental/snapshots/domain/v1-accounts.64-72.kvei differ
Files /Users/wmitsuda/eth-nodes/erigon3-holesky-converted/snapshots/domain/v1-accounts.72-74.kvei and /Users/wmitsuda/eth-nodes/erigon3-holesky-experimental/snapshots/domain/v1-accounts.72-74.kvei differ
Files /Users/wmitsuda/eth-nodes/erigon3-holesky-converted/snapshots/domain/v1-code.0-64.bt and /Users/wmitsuda/eth-nodes/erigon3-holesky-experimental/snapshots/domain/v1-code.0-64.bt differ
Files /Users/wmitsuda/eth-nodes/erigon3-holesky-converted/snapshots/domain/v1-code.0-64.kvei and /Users/wmitsuda/eth-nodes/erigon3-holesky-experimental/snapshots/domain/v1-code.0-64.kvei differ
Files /Users/wmitsuda/eth-nodes/erigon3-holesky-converted/snapshots/domain/v1-code.64-72.kvei and /Users/wmitsuda/eth-nodes/erigon3-holesky-experimental/snapshots/domain/v1-code.64-72.kvei differ
Files /Users/wmitsuda/eth-nodes/erigon3-holesky-converted/snapshots/domain/v1-code.72-74.kvei and /Users/wmitsuda/eth-nodes/erigon3-holesky-experimental/snapshots/domain/v1-code.72-74.kvei differ
Files /Users/wmitsuda/eth-nodes/erigon3-holesky-converted/snapshots/domain/v1-commitment.0-64.kv and /Users/wmitsuda/eth-nodes/erigon3-holesky-experimental/snapshots/domain/v1-commitment.0-64.kv differ
Files /Users/wmitsuda/eth-nodes/erigon3-holesky-converted/snapshots/domain/v1-commitment.0-64.kvi and /Users/wmitsuda/eth-nodes/erigon3-holesky-experimental/snapshots/domain/v1-commitment.0-64.kvi differ
Files /Users/wmitsuda/eth-nodes/erigon3-holesky-converted/snapshots/domain/v1-commitment.64-72.kv and /Users/wmitsuda/eth-nodes/erigon3-holesky-experimental/snapshots/domain/v1-commitment.64-72.kv differ
Files /Users/wmitsuda/eth-nodes/erigon3-holesky-converted/snapshots/domain/v1-commitment.64-72.kvi and /Users/wmitsuda/eth-nodes/erigon3-holesky-experimental/snapshots/domain/v1-commitment.64-72.kvi differ
Files /Users/wmitsuda/eth-nodes/erigon3-holesky-converted/snapshots/domain/v1-commitment.72-74.kv and /Users/wmitsuda/eth-nodes/erigon3-holesky-experimental/snapshots/domain/v1-commitment.72-74.kv differ
Files /Users/wmitsuda/eth-nodes/erigon3-holesky-converted/snapshots/domain/v1-commitment.72-74.kvi and /Users/wmitsuda/eth-nodes/erigon3-holesky-experimental/snapshots/domain/v1-commitment.72-74.kvi differ
Files /Users/wmitsuda/eth-nodes/erigon3-holesky-converted/snapshots/domain/v1-receipt.0-64.kvei and /Users/wmitsuda/eth-nodes/erigon3-holesky-experimental/snapshots/domain/v1-receipt.0-64.kvei differ
Files /Users/wmitsuda/eth-nodes/erigon3-holesky-converted/snapshots/domain/v1-receipt.64-72.kvei and /Users/wmitsuda/eth-nodes/erigon3-holesky-experimental/snapshots/domain/v1-receipt.64-72.kvei differ
Files /Users/wmitsuda/eth-nodes/erigon3-holesky-converted/snapshots/domain/v1-receipt.72-74.kvei and /Users/wmitsuda/eth-nodes/erigon3-holesky-experimental/snapshots/domain/v1-receipt.72-74.kvei differ
Files /Users/wmitsuda/eth-nodes/erigon3-holesky-converted/snapshots/domain/v1-storage.0-64.bt and /Users/wmitsuda/eth-nodes/erigon3-holesky-experimental/snapshots/domain/v1-storage.0-64.bt differ
Files /Users/wmitsuda/eth-nodes/erigon3-holesky-converted/snapshots/domain/v1-storage.0-64.kvei and /Users/wmitsuda/eth-nodes/erigon3-holesky-experimental/snapshots/domain/v1-storage.0-64.kvei differ
Files /Users/wmitsuda/eth-nodes/erigon3-holesky-converted/snapshots/domain/v1-storage.64-72.kvei and /Users/wmitsuda/eth-nodes/erigon3-holesky-experimental/snapshots/domain/v1-storage.64-72.kvei differ
Files /Users/wmitsuda/eth-nodes/erigon3-holesky-converted/snapshots/domain/v1-storage.72-74.kvei and /Users/wmitsuda/eth-nodes/erigon3-holesky-experimental/snapshots/domain/v1-storage.72-74.kvei differ
Files /Users/wmitsuda/eth-nodes/erigon3-holesky-converted/snapshots/history/v1-receipt.0-64.v and /Users/wmitsuda/eth-nodes/erigon3-holesky-experimental/snapshots/history/v1-receipt.0-64.v differ
Files /Users/wmitsuda/eth-nodes/erigon3-holesky-converted/snapshots/history/v1-receipt.72-74.v and /Users/wmitsuda/eth-nodes/erigon3-holesky-experimental/snapshots/history/v1-receipt.72-74.v differ
Files /Users/wmitsuda/eth-nodes/erigon3-holesky-converted/snapshots/v1-003180-003190-transactions-to-block.idx and /Users/wmitsuda/eth-nodes/erigon3-holesky-experimental/snapshots/v1-003180-003190-transactions-t
o-block.idx differ
Files /Users/wmitsuda/eth-nodes/erigon3-holesky-converted/snapshots/v1-003180-003190-transactions.idx and /Users/wmitsuda/eth-nodes/erigon3-holesky-experimental/snapshots/v1-003180-003190-transactions.idx differ
Files /Users/wmitsuda/eth-nodes/erigon3-holesky-converted/snapshots/v1-003190-003191-transactions-to-block.idx and /Users/wmitsuda/eth-nodes/erigon3-holesky-experimental/snapshots/v1-003190-003191-transactions-t
o-block.idx differ
Files /Users/wmitsuda/eth-nodes/erigon3-holesky-converted/snapshots/v1-003190-003191-transactions.idx and /Users/wmitsuda/eth-nodes/erigon3-holesky-experimental/snapshots/v1-003190-003191-transactions.idx differ
Files /Users/wmitsuda/eth-nodes/erigon3-holesky-converted/snapshots/v1-003191-003192-transactions-to-block.idx and /Users/wmitsuda/eth-nodes/erigon3-holesky-experimental/snapshots/v1-003191-003192-transactions-t
o-block.idx differ
Files /Users/wmitsuda/eth-nodes/erigon3-holesky-converted/snapshots/v1-003191-003192-transactions.idx and /Users/wmitsuda/eth-nodes/erigon3-holesky-experimental/snapshots/v1-003191-003192-transactions.idx differ
Files /Users/wmitsuda/eth-nodes/erigon3-holesky-converted/snapshots/v1-003192-003193-transactions-to-block.idx and /Users/wmitsuda/eth-nodes/erigon3-holesky-experimental/snapshots/v1-003192-003193-transactions-t
o-block.idx differ
Files /Users/wmitsuda/eth-nodes/erigon3-holesky-converted/snapshots/v1-003192-003193-transactions.idx and /Users/wmitsuda/eth-nodes/erigon3-holesky-experimental/snapshots/v1-003192-003193-transactions.idx differ
Files /Users/wmitsuda/eth-nodes/erigon3-holesky-converted/snapshots/v1-003193-003194-transactions-to-block.idx and /Users/wmitsuda/eth-nodes/erigon3-holesky-experimental/snapshots/v1-003193-003194-transactions-t
o-block.idx differ
Files /Users/wmitsuda/eth-nodes/erigon3-holesky-converted/snapshots/v1-003193-003194-transactions.idx and /Users/wmitsuda/eth-nodes/erigon3-holesky-experimental/snapshots/v1-003193-003194-transactions.idx differ
Files /Users/wmitsuda/eth-nodes/erigon3-holesky-converted/snapshots/v1-003194-003195-transactions-to-block.idx and /Users/wmitsuda/eth-nodes/erigon3-holesky-experimental/snapshots/v1-003194-003195-transactions-t
o-block.idx differ
Files /Users/wmitsuda/eth-nodes/erigon3-holesky-converted/snapshots/v1-003194-003195-transactions.idx and /Users/wmitsuda/eth-nodes/erigon3-holesky-experimental/snapshots/v1-003194-003195-transactions.idx differ
Files /Users/wmitsuda/eth-nodes/erigon3-holesky-converted/snapshots/v1-003195-003196-transactions-to-block.idx and /Users/wmitsuda/eth-nodes/erigon3-holesky-experimental/snapshots/v1-003195-003196-transactions-t
o-block.idx differ
Files /Users/wmitsuda/eth-nodes/erigon3-holesky-converted/snapshots/v1-003195-003196-transactions.idx and /Users/wmitsuda/eth-nodes/erigon3-holesky-experimental/snapshots/v1-003195-003196-transactions.idx differ
Files /Users/wmitsuda/eth-nodes/erigon3-holesky-converted/snapshots/v1-003196-003197-transactions-to-block.idx and /Users/wmitsuda/eth-nodes/erigon3-holesky-experimental/snapshots/v1-003196-003197-transactions-t
o-block.idx differ
Files /Users/wmitsuda/eth-nodes/erigon3-holesky-converted/snapshots/v1-003196-003197-transactions.idx and /Users/wmitsuda/eth-nodes/erigon3-holesky-experimental/snapshots/v1-003196-003197-transactions.idx differ

Jan 22 '25 23:01 wmitsuda

Given the overall results on holesky now seems satisfactory, now triggering a regeneration run on mainnet, should be done in < 1 week

Jan 22 '25 23:01 wmitsuda

Status: mainnet sync from scratch still running after 1 week, at block 17.9M. note: mainnet sync performance noticeably drops after block 16M, going from +700Mgas/s -> 400-500Mgas/s.

Jan 29 '25 17:01 wmitsuda

Status: mainnet sync from scratch done after 11 days, now proceeding to check data integrity.

FYI: a few hours after reaching the tip I got: https://github.com/erigontech/erigon/issues/13676 , but I think it has nothing to do with this change, so ignoring in this context.

Feb 03 '25 08:02 wmitsuda

data seems good, diffs are similar to what I found on holesky.

I have another node that is not converted, but new data is being generated in the new format, and new .ef files are equivalent to plain E3 ones.

Feb 03 '25 19:02 wmitsuda

given our previous conversation about leaving this one post-E3 final, I'll keep merging main and running multiple rounds of sync from scratch on mainnet from now on in order to catch any regressions.

Feb 03 '25 19:02 wmitsuda

Status:

last mainnet sync test resulted in .ef data being semantically correct compared to plain E3.
however the data generated by this PR also differs in some other few files that were not expected.
so the question is: is this PR also generating incorrect data for let's say a few domains? or this PR is correct and for some unknown reason the data coming from published snapshots is different? (it may be incorrect due to some past bug, but also it could be just format slightly changed over time but it is still semantically equivalent, so can't assume snapshots are incorrect).

in order to answer that question I conducted another experiment:

regenerated the state snapshots using the same base branch of this PR (beta2)
then I ended up with the following 3 equivalent mainnet instances:
- (1) mainnet plain -> just E3 beta 2 as-is
- (2) mainnet experimental -> this PR regenerated locally on top of beta 2
- (3) mainnet regen -> beta 2 regenerated locally

The expected result is:

diffing (2) and (3) must be the same except:
- idx/*.ef -> because of this PR's optimizations
- accessor/*.efi -> because .ef contents changed, hence offsets changed
- we are ignoring domains/commitment-* -> because it is known it is not deterministic
- we are ignoring domains?receipts-* -> because of recent bugs at the time of beta2

if the above is true, then it means there are not errors introduced in other files by this PR and the currently published snapshots are not 100% correspondent to what should be produced by the current code (but it doesn't mean they are incorrect).

the following comments will contain the result of this experiment for future reference.

Feb 28 '25 08:02 wmitsuda

Results: (2) vs (3)

diff -r -x '*.torrent' -x 'preverified.toml' -x '*.seg' -x '*.idx' erigon3-mainnet-regen/snapshots/ erigon3-mainnet-experimental/snapshots/ > experimental-vs-regen.txt

the attached diff shows the result is what we expected, hence validating our hypothesis this PR is correct.

the only thing not expected was all the .kvei differs, but we'll assume it is not deterministic or it uses some seed/salt/etc we didn't properly configured at chain generation.

experimental-vs-regen.txt

Feb 28 '25 09:02 wmitsuda

Results: (1) vs (3)

diff -r -x '*.torrent' -x 'preverified.toml' -x '*.seg' -x '*.idx' erigon3-mainnet-regen/snapshots/ erigon3-mainnet/snapshots/ > original-vs-regen.txt

noticed how many data files seems different, like regenerated .ef and .v files, making a direct comparison between published snapshots and regenerated ones unreliable, so I'll rely on (2) vs (3) to validate this PR.

original-vs-regen.txt

Feb 28 '25 09:02 wmitsuda

I'm taking this PR out of draft state in order to trigger the CI pipeline and because after the latest round of tests on mainnet I'm now confident about it's correctness.

I'm leaving the do-not-merge tags because it is a one-way operation and that should be properly planned with the team.

Feb 28 '25 09:02 wmitsuda

also, next step is merge recent main as current test was run against beta2.

Feb 28 '25 09:02 wmitsuda

@AskAlexSharov I think I could use some preliminary review now if you have some time.

We are not going to merge it yet, I've proposed an in-person session to discuss how the rollout will be coordinated for 3.1, but I think the code is good enough to be reviewed now so we don't have to rush last minute modifications later.

Mar 02 '25 08:03 wmitsuda

commitment .kvei and .kvi - is index of .kv - also expected to be different

Mar 02 '25 10:03 AskAlexSharov

If your PR changing .ef then .efi will also changed

Mar 02 '25 10:03 AskAlexSharov

Status: need some small adjustments before merging

[x] Merge latest main
[ ] Change converter to fabricate .ef v2.0
[ ] Change converter to fabricate .efi v1.1
[ ] Change snapshot to read/write .ef v2.0
[ ] Change snapshot to read/write .efi v1.1
[ ] Regenerate holesky as a fast sanity check
[ ] Add Alex's suggestions

Mar 25 '25 19:03 wmitsuda