couchdb-couch icon indicating copy to clipboard operation
couchdb-couch copied to clipboard

Improve compaction task status updates

Open davisp opened this issue 8 years ago • 3 comments

Previous the emsort related operations did not update the compaction task status. For large databases this leads to some very long waits while the compaction task stays at 100%. This change adds progress reports to the steps for sorting and copying document ids back into the database file.

davisp avatar Mar 30 '17 15:03 davisp

Also I need to change my use of DocCount to be TotalChanges. I noticed while testing the optimizations I tried that its currently not correct during compaction retries.

davisp avatar Mar 30 '17 20:03 davisp

Updated to use define's for batch sizes and fixed the changes/doc count mixup.

For the {merge_start, forward}, thats only there for completeness. It felt a bit weird to not include that event even though I don't necessarily need it for this work.

Updates during the second phase happen here:

https://github.com/apache/couchdb-couch/pull/241/files#diff-f6f654ab26b490bab95be6f502c49d89R1376

This is a bit subtle but it seemed like the best I could do without starting to modify the on-disk contents of files which I like to avoid when at all possible.

Its a bit funky but without starting to store the total number of rows in emsort the best approach I had was to guess with the input being the total number of changes processed in this run and then in the first phase of the merge sort just count how many rows we have. Once that first pass is over we can calculate the total number of rows that will be copied and then just update progress in the normal fashion. Its a bit awkward but that emsort_cb kind of switches modes once it sees that the first phase has ended and then its purely just listening for row_copy events.

davisp avatar Mar 31 '17 16:03 davisp

And I hesitate to store the total number of rows because there'd then be upgrade things to worry about and generally speaking this will just sort itself out while still giving an approximate progress update.

davisp avatar Mar 31 '17 16:03 davisp