ckb
ckb copied to clipboard
Enhance `ckb export/import` for flexible ranges, faster import, and JSONL output
Feature Request
Is your feature request related to a problem? Please describe.
-
ckb exportonly supports exporting from genesis up to a target block height. -
ckb importfully validates all data usingSwitch::NONE, which is slow. - Exported data is in JSON format, making it hard to merge splits like
a.json (1–1,000,000)andb.json (1,000,001–1,000,100).
Describe the solution you'd like
-
Support arbitrary height ranges in
ckb export(e.g.,--from <start>…--to <end>). -
Add optional flags to
ckb importto skip certain checks, speeding up large imports. - Support JSONL output format ? in export for easy merging and processing of segmented data files.
On my desktop: i9-14900K 32Core, nvme ssd, export ckb mainnet 16,846,713 blocks need 5 minutes.
The exported jsonl file size is 222G, I think we need to streaming compress it when exporting.