ckb icon indicating copy to clipboard operation
ckb copied to clipboard

Enhance `ckb export/import` for flexible ranges, faster import, and JSONL output

Open eval-exec opened this issue 7 months ago • 1 comments

Feature Request

Is your feature request related to a problem? Please describe.

  • ckb export only supports exporting from genesis up to a target block height.
  • ckb import fully validates all data using Switch::NONE, which is slow.
  • Exported data is in JSON format, making it hard to merge splits like a.json (1–1,000,000) and b.json (1,000,001–1,000,100).

Describe the solution you'd like

  1. Support arbitrary height ranges in ckb export (e.g., --from <start>--to <end>).
  2. Add optional flags to ckb import to skip certain checks, speeding up large imports.
  3. Support JSONL output format ? in export for easy merging and processing of segmented data files.

eval-exec avatar Jul 04 '25 01:07 eval-exec

On my desktop: i9-14900K 32Core, nvme ssd, export ckb mainnet 16,846,713 blocks need 5 minutes. The exported jsonl file size is 222G, I think we need to streaming compress it when exporting. Image

eval-exec avatar Jul 29 '25 05:07 eval-exec