go-threads icon indicating copy to clipboard operation
go-threads copied to clipboard

Published 20 hours ago • verified publisher icon textileio

Backfill index on creation

Open asutula opened this issue 4 years ago • 4 comments

Currently we start building the index from the time it was created, but we should go though all existing data and index it as well.

asutula avatar Mar 26 '20 17:03 asutula

Feeling like this might be quite complex. Indexing can take an unbounded amount of time, so that has me thinking it will have to be backed by a durable, persistent, job queue kinda setup.

asutula avatar Mar 27 '20 20:03 asutula

You'd also want it to be non-blocking

asutula avatar Mar 27 '20 20:03 asutula

I think most DBs block while doing an initial index (or back filling), e.g., (https://docs.mongodb.com/manual/core/index-creation/), but some of the more modern one (Rethink?) do not. The simpler solution might be a good first step.

sanderpick avatar Mar 27 '20 20:03 sanderpick

Our situation might actually be a bit more dangerous though, since we're handing full admin control over to a dev with a client lib. At least they can only mess up their own DB (once ACLs are a thing).

sanderpick avatar Mar 27 '20 21:03 sanderpick