fast-sqlite3-inserts icon indicating copy to clipboard operation
fast-sqlite3-inserts copied to clipboard

serialize sqlite pages

Open milahu opened this issue 5 months ago • 0 comments

learn about SQLite file format then generate the pages and write to disk directly.

now that kaitai has serialization we can use it to write sqlite pages

this is an early draft... it only works to write the database header, no pages further progress requires serializing pages and writing them to the correct byte offsets in the database file

use case: convert a 5 GB jsonl file to sqlite torrents_byteoffsets_parse_jsonl_zst.py with my script i reach only 5 MiB/s the bottleneck are the sqlite insert queries my system (under load) can read and decompress at 20 MiB/s

milahu avatar Jul 31 '25 12:07 milahu