fdb-document-layer
fdb-document-layer copied to clipboard
Parallel index rebuilds
This is a follow-up feature request for #4.
Once we have fault-tolerant index builds we should look into parallel builds. This can follow task bucket pattern. Tasks should be small enough to fit in one transaction. If the task doesn't finish in a single transaction, it should be split. This makes each task atomic. Maintaining status could be as simple as tracking pending tasks. Reading shard keys between index bounds would give us an estimation of the index size.
It would have been ideal to use the task buckets here. But, the task buckets live in fdbclient and use ReadYourWritesTransaction. They don't go through fdb_flow. Also, index rebuild is a very specific case we are better off having the custom code.