ArchiveBot icon indicating copy to clipboard operation
ArchiveBot copied to clipboard

The uploader should try the next file on upload failure

Open falconkirtaran opened this issue 7 years ago • 0 comments

Sometimes the uploader will keep trying and failing to upload a single WARC. When this happens, it retries the same WARC constantly and does not advance to others, even if the reason the WARC upload failed is not ephemeral (bad filename, some bucket size limit reached...); this causes full disk and exacerbates the issue.

It should instead try to skip failed files by keeping a list of them, and print the list of failing files on the console, and periodically retry them when it has nothing else to do. Later, notifications.

falconkirtaran avatar Aug 01 '17 11:08 falconkirtaran