brainlit icon indicating copy to clipboard operation
brainlit copied to clipboard

Allow upload_volumes to start from a specfic mip/chunk_idx

Open bvarjavand opened this issue 4 years ago • 1 comments

CloudVolume will sometimes hang randomly, interrupting large uploads.

Currently, if an error happens, we only print a short error message, and the upload must be restarted from scratch.

It would be far superior to print the mip and chunk the error was on along with the error, enabling continuation of a large, multi-day upload.

The previous data upload looked like the following, using code from the old-upload branch.

Screen Shot 2020-08-07 at 12 58 11 PM

Current progress to make this feature usable to any user is in the continue_upload branch. Specifically,

  • upload.py
    • Adding parameter for upload_volumes method on line 258
    • Type checking on line 280
    • Adding functionality to upload_volumes on lines [309-315] (https://github.com/neurodata/brainlit/blob/continue_upload/brainlit/utils/upload.py#L309-L315), 324, and 333.

Adding tests and debugging still needs to be done. Tests would be added in tests/test_upload.py, adding the continue_upload parameter to the type checking test for upload_volumes, as well as adding a test exclusively checking validity of continuing uploads from the middle and end indices.

bvarjavand avatar Aug 06 '20 00:08 bvarjavand

provide an example.

jovo avatar Aug 07 '20 19:08 jovo