Amazon is flakey

[kousu]

We've been having sporadic failures due to Amazon rejecting us:

https://github.com/spine-generic/data-multi-subject/runs/4226501836?check_suite_focus=true#step:11:711

 get derivatives/labels/sub-tokyoSkyra03/anat/sub-tokyoSkyra03_T2star_rms_gmseg-manual.nii.gz (from amazon...) 
  download failed: Internal Server Error

  Unable to access these remotes: amazon

  Try making some of these repositories available:
  	5a5447a8-a9b8-49bc-8276-01a62632b502 -- [amazon]
   	5cdba4fc-8d50-4e89-bb0c-a3a4f9449666 -- [email protected]:~/code/spine-generic/data-multi-subject
   	9e4d13f3-30e1-4a29-8b86-670879928606 -- [email protected]:~/data/data-multi-subject

  (Note that these git remotes have annex-ignore set: origin)
failed

https://github.com/spine-generic/data-multi-subject/runs/4038349050?check_suite_focus=true#step:11:867

 get sub-beijingVerio04/dwi/sub-beijingVerio04_acq-b0_dwi.nii.gz (from amazon...) 
  download failed: HandshakeFailed (Error_Misc "Network.Socket.recvBuf: resource vanished (Connection reset by peer)")

  Unable to access these remotes: amazon

  Try making some of these repositories available:
  	5a5447a8-a9b8-49bc-8276-01a62632b502 -- [amazon]
   	5cdba4fc-8d50-4e89-bb0c-a3a4f9449666 -- [email protected]:~/code/spine-generic/data-multi-subject
   	9e4d13f3-30e1-4a29-8b86-670879928606 -- [email protected]:~/data/data-multi-subject

  (Note that these git remotes have annex-ignore set: origin)
failed

git-annex doesn't have any kind of backoff/retry mechanism that I know of, so a single bad connection during the download fails the whole thing.

For CI, we can click "re-run". For human users they can run git annex get . again, so maybe this isn't a big deal, but we should keep an eye on it.

[alexfoias]

We had the issue on 2022-03-02 . See screenshot: But I managed to get it on my computer :

~/data/data-multi-subject ╱ af/fix_defaced_images  git annex get sub-tokyo750w07/anat/sub-tokyo750w07_acq-MToff_MTS.nii.gz