Implement ECR API retry/backoff logic when throttling is detected

[geezyx]

Hello,

We have been running into throttling limits with AWS ECR, and AWS has requested that we do the following:

Implement a retry exception or exponential backoff on the API calls that you suspect that you are being throttled.

Since the majority of our ECR usage is due to concourse (many of our custom resources are hosted there, as well as the final build artifacts from our pipelines), we don't have much control over this behavior. AWS has requested that we implement the above before they will increase our API limits.

Is it possible to introduce some sort of retry logic with increasing wait timers when the throttling errors are detected? It seems like a 400 status with a specific message about throttling is returned from the API. Here is a sample error we found in another tool we use (image and tag removed):

Failed to query image details "<image>" "<tag>" ThrottlingException: Rate exceeded
status code: 400

[concourse-bot]

Hi there!

We use Pivotal Tracker to provide visibility into what our team is working on. A story for this issue has been automatically created.

The current status is as follows:

• [ ] #143514049 Implement ECR API retry/backoff logic when throttling is detected

This comment, as well as the labels on the issue, will be automatically updated as the status in Tracker changes.

[geezyx]

Due to https://github.com/concourse/docker-image-resource/issues/114 we can't see the actual error coming back from ECR, but in addition to the 400 error I mentioned above, there is this throttling response from AWS that could be helpful: http://docs.aws.amazon.com/AmazonECR/latest/userguide/common-errors.html#error-429-too-many-requests

[chendrix]

This is something that would be a wonderful PR to accept.

If you're interested, please start by looking at our Contributing Guide

I've also attached this to the Help Wanted milestone.