kitchen-ec2 icon indicating copy to clipboard operation
kitchen-ec2 copied to clipboard

Polling fails due "Request limit exceeded" and orphans instances

Open sinneduy opened this issue 6 years ago • 3 comments

It looks like (PR) added retry functionality to handle "Request limit exceeded" for instances starting, and verifying volumes are up, but it doesn't have that for polling for an instance to exist.

Waited 0/300s for spot request <redacted> to become fulfilled.
--
  | Waited 5/300s for spot request <redacted> to become fulfilled.
  | Instance <redacted> requested.
  | Polling AWS for existence, attempt 0...
  | -----> Destroying <redacted>...
  | Finished destroying <redacted> (0m0.00s).

rake aborted!
--
  | Kitchen::ActionFailed: Failed to complete #create action: [stopped waiting due to an unexpected error: Request limit exceeded.]

Perhaps the wait_with_destroy call needs to be used for this?

Also, it looks like the instance doesn't actually get destroyed after this failure.

sinneduy avatar Mar 20 '18 22:03 sinneduy

Have you tried "retry_limit"? https://github.com/test-kitchen/kitchen-ec2/blob/65796302b738152eb775ab88bae0531b42986b88/lib/kitchen/driver/aws/client.rb. This is not documented but appears to be passed directly to the AWS SDK.

ghost avatar Mar 21 '18 19:03 ghost

that can reduce the frequency of this error, but it can still result in orphaned instances, if there are any failures...?

sinneduy avatar Mar 21 '18 19:03 sinneduy

Im also running into this issue... any plans to fix?

xblitz avatar Oct 30 '18 19:10 xblitz