internetarchive icon indicating copy to clipboard operation
internetarchive copied to clipboard

Upload: Retry on exceptions instead of crashing

Open imerr opened this issue 1 year ago • 2 comments

Hi I've been trying to upload some things to IA, unfortunately I would come back to the ia upload command having crashed due to connection timeouts very often It'd be good if those would be handled more gracefully and just retried unless there's a reason they are not currently? I have patched this in for myself, but that's more of a duct tape fix to get things working (infinite retries with 1s sleep inbetween on exception) and I figured I'd reach out for feedback how this should be handled properly

I see there is already a retry flag, but that only applies to 503 responses. I would suggest to expand that to also include connection errors and the like? Although personally I'd like it to just retry indefinitely on network errors (with proper backoff between attempts?), so maybe expanding the flag with a special value to do that would be good (but that's an issue for another time)

Let me know what the preferred way of handling this is and I'm happy to get that coded out and submit a pr (or if someone else wants to that's fine too)

imerr avatar Jun 25 '23 15:06 imerr

Exponential backoff. Also it should probably only retry if the server responded with a "100 Continue" for the same action before.

maxz avatar Jun 27 '23 00:06 maxz

I don't think 100s are relevant; requests doesn't support Expect: 100-continue anyway.

JustAnotherArchivist avatar Jun 27 '23 01:06 JustAnotherArchivist