metawarc icon indicating copy to clipboard operation
metawarc copied to clipboard

Indexer: Improve perf by deferring index DB commit

Open martin-kokos opened this issue 1 year ago • 3 comments

Since we're writing to a local sqlite, we're not worried about losing connection to a remote DB and losing consistency, so we can defer the commit until the end.

This speeds up indexing by about 30x.

martin-kokos avatar Aug 19 '24 09:08 martin-kokos

My editor removed some trailing whitespace too.

martin-kokos avatar Aug 19 '24 09:08 martin-kokos

@martin-kokos thanks, I considered to replace sqlite with duckdb to speedup analytics after data processing.

ivbeg avatar Aug 19 '24 10:08 ivbeg

That's a separate topic, I think.

martin-kokos avatar Aug 19 '24 12:08 martin-kokos