brainlit
brainlit copied to clipboard
Allow upload_volumes to start from a specfic mip/chunk_idx
CloudVolume will sometimes hang randomly, interrupting large uploads.
Currently, if an error happens, we only print a short error message, and the upload must be restarted from scratch.
It would be far superior to print the mip and chunk the error was on along with the error, enabling continuation of a large, multi-day upload.
The previous data upload looked like the following, using code from the old-upload
branch.

Current progress to make this feature usable to any user is in the continue_upload
branch.
Specifically,
-
upload.py
Adding tests and debugging still needs to be done. Tests would be added in tests/test_upload.py
, adding the continue_upload
parameter to the type checking test for upload_volumes
, as well as adding a test exclusively checking validity of continuing uploads from the middle and end indices.
provide an example.