Example of how to add a custom retryer for IMDS requests when EC2 role credential requests are throttled?

Open e-gineer opened this issue 2 years ago • 0 comments

Describe the issue

For the AWS plugin of Steampipe, we manage AWS client connections to many different accounts in parallel. This works great .... unless there is a very large number of clients all slamming the same IMDS endpoint on a single EC2 instance with a role.

In that case we start receiving throttling errors like this:

EC2: DescribeVpcs, failed to sign request: failed to retrieve credentials: failed to refresh cached credentials, no EC2 IMDS role found, operation error ec2imds: GetMetadata, http response error StatusCode: 429, request to EC2 IMDS failed

We've optimized everything down to a single client per account (shared across regions as needed through .Copy()) to ensure there is a single credentials setup per account. (Side note: a single credentials setup is 3 calls to the metadata API.)

As a next step, we'd like to add retry / throttling to the IMDS API requests from within the EC2 role credentials. I believe this needs to be done through configuration, because the actual IMDS API calls don't happen until the client is actually used - so there is no error at instantiation time.

Unfortunately - I can't work out, or find any examples, for how to do this.

Here are the best links I found:

IDMS service - https://pkg.go.dev/github.com/aws/aws-sdk-go-v2/feature/ec2/imds#pkg-overview
EC2 IAM Role Credentials examples - https://aws.github.io/aws-sdk-go-v2/docs/migrating/#amazon-ec2-iam-role-credentials
(Broken) example with ec2rolecreds - https://pkg.go.dev/github.com/aws/aws-sdk-go-v2/credentials/ec2rolecreds#Provider
(Fixed) issue that WithEC2RoleCredentialOptions was not honored - https://github.com/aws/aws-sdk-go-v2/issues/1296

Unfortunately it's difficult to reproduce this through a basic example script - you need to do hundreds of IMDS requests in a row.

Here is (roughly) the code I used. It ran, and the debugHTTPClient took effect / showed the requests, but the other imds.Options{...} did not seem to apply at all?

Should this approach work? Any pointers you can give for how to add a custom retryer to the IMDS requests inside EC2 role credentials?

// This code ran, but didn't seem to respect the idms.Options{...}
// The debugHTTPClient did work to show requests / throttling.
retryer := retry.NewStandard(func(o *retry.StandardOptions) {
  // reseting state of rand to generate different random values
  rand.Seed(time.Now().UnixNano())
  o.MaxAttempts = 50
  o.MaxBackoff = 5 * time.Minute
  o.RateLimiter = NoOpRateLimit{} // With no rate limiter
  o.Backoff = NewExponentialJitterBackoff(100*time.Millisecond, 5)
  log.Printf("[WARN] retryer!")
})
configOptions = append(configOptions, config.WithEC2RoleCredentialOptions(func(opts *ec2rolecreds.Options) {
  // debugHTTPClient per https://github.com/aws/aws-sdk-go-v2/issues/1296
  opts.Client = imds.New(imds.Options{Retryer: retryer, ClientLogMode: aws.LogRetries | aws.LogRequest}, withDebugHTTPClient())
}))

AWS Go SDK V2 Module Versions Used

No response

Jan 26 '23 19:01 e-gineer

Example of how to add a custom retryer for IMDS requests when EC2 role credential requests are throttled?

Describe the issue

Links

AWS Go SDK V2 Module Versions Used