fast-sqlite3-inserts
fast-sqlite3-inserts copied to clipboard
serialize sqlite pages
learn about SQLite file format then generate the pages and write to disk directly.
now that kaitai has serialization we can use it to write sqlite pages
this is an early draft... it only works to write the database header, no pages further progress requires serializing pages and writing them to the correct byte offsets in the database file
use case: convert a 5 GB jsonl file to sqlite torrents_byteoffsets_parse_jsonl_zst.py with my script i reach only 5 MiB/s the bottleneck are the sqlite insert queries my system (under load) can read and decompress at 20 MiB/s