fdb-document-layer icon indicating copy to clipboard operation
fdb-document-layer copied to clipboard

Parallel index rebuilds

Open apkar opened this issue 6 years ago • 0 comments

This is a follow-up feature request for #4.

Once we have fault-tolerant index builds we should look into parallel builds. This can follow task bucket pattern. Tasks should be small enough to fit in one transaction. If the task doesn't finish in a single transaction, it should be split. This makes each task atomic. Maintaining status could be as simple as tracking pending tasks. Reading shard keys between index bounds would give us an estimation of the index size.

It would have been ideal to use the task buckets here. But, the task buckets live in fdbclient and use ReadYourWritesTransaction. They don't go through fdb_flow. Also, index rebuild is a very specific case we are better off having the custom code.

apkar avatar Nov 29 '18 02:11 apkar