terragrunt icon indicating copy to clipboard operation
terragrunt copied to clipboard

Terragrunt multiple accounts with assume role getting connection refused from aws

Open wiseelf opened this issue 2 years ago • 0 comments

Terrafrom version: 1.2.8 Terragrunt version: 0.38.9 AWS provider version: 4.28.0

My config:

iam_role = local.iam_role

# Generate an AWS provider block
generate "provider" {
  path = "provider.tf"
  if_exists = "overwrite_terragrunt"
  contents = <<EOF
provider "aws" {
  region  = "${local.aws_region}"

  # Only these AWS Account IDs may be operated on by this template
  allowed_account_ids = ["${local.account_id}"]
}
EOF
}

# Configure Terragrunt to automatically store tfstate files in an S3 bucket
remote_state {
  backend = "s3"
  config = {
    encrypt        = true
    bucket         = "terraform-st-acc-${local.account_id}-${local.aws_region}"
    key            = "${path_relative_to_include()}/terraform.tfstate"
    region         = local.aws_region
    dynamodb_table = "terraform-locks"
  }
  generate = {
    path      = "backend.tf"
    if_exists = "overwrite_terragrunt"
  }
}

Having multiple accounts and switching between them with assume-role. During terragrunt run-all apply --terragrunt-non-interactive --terragrunt-parallelism 5 getting connection refused errors from sts, dynamodb, s3, etc. Example:

ERRO[0040] Create S3 bucket with retry iris-terraform-st-acc-1234567890-us-east-1 returned an error: RequestError: send request failed
caused by: Put "https://terraform-st-acc-1234567890-us-east-1.s3.amazonaws.com/": dial tcp 52.217.133.121:443: connect: connection refused. Sleeping for 10s and will try again.
ERRO[0044] Create S3 bucket with retry terraform-st-acc-1234567890-us-east-1 returned an error: RequestError: send request failed
caused by: Put "https://terraform-st-acc-1234567890-us-east-1.s3.amazonaws.com/?versioning=": dial tcp 52.217.133.121:443: connect: connection refused. Sleeping for 10s and will try again. 

Initializing the backend...
╷
│ Error: error configuring S3 Backend: error validating provider credentials: error calling sts:GetCallerIdentity: RequestError: send request failed
│ caused by: Post "https://sts.amazonaws.com/": dial tcp 209.54.180.124:443: connect: connection refused
│
│

│ Error: Error releasing the state lock
│
│ Error message: RequestError: send request failed
│ caused by: Post "https://dynamodb.us-east-1.amazonaws.com/": dial tcp
│ 3.218.182.10:443: connect: connection refused
│ Lock Info:

Looks like terragrunt is doing sts:assume for each operation such as: to read info from backend, to put a lock, to remove a lock. So AWS basically starts throttling my requests and i'm getting connection refused errors. Am I missing something? Is there a way to avoid these errors and keep using assume-roles?

wiseelf avatar Aug 30 '22 06:08 wiseelf