go-threads
go-threads copied to clipboard
Backfill index on creation
Currently we start building the index from the time it was created, but we should go though all existing data and index it as well.
Feeling like this might be quite complex. Indexing can take an unbounded amount of time, so that has me thinking it will have to be backed by a durable, persistent, job queue kinda setup.
You'd also want it to be non-blocking
I think most DBs block while doing an initial index (or back filling), e.g., (https://docs.mongodb.com/manual/core/index-creation/), but some of the more modern one (Rethink?) do not. The simpler solution might be a good first step.
Our situation might actually be a bit more dangerous though, since we're handing full admin control over to a dev with a client lib. At least they can only mess up their own DB (once ACLs are a thing).