metawarc
metawarc copied to clipboard
Indexer: Improve perf by deferring index DB commit
Since we're writing to a local sqlite, we're not worried about losing connection to a remote DB and losing consistency, so we can defer the commit until the end.
This speeds up indexing by about 30x.
My editor removed some trailing whitespace too.
@martin-kokos thanks, I considered to replace sqlite with duckdb to speedup analytics after data processing.
That's a separate topic, I think.