terraform Allow references to instances of the same resource

Allow references to instances of the same resource

Open Clete2 opened this issue 4 years ago • 12 comments

Terraform Version

Terraform v0.12.18
+ provider.archive v1.3.0
+ provider.aws v2.43.0
+ provider.null v2.1.2

Terraform Configuration Files

resource "aws_sqs_queue" "queue" {
  count = length(var.queue_definitions)
  name  = "${var.name}${count.index}"

  delay_seconds              = var.queue_definitions[count.index].delaySeconds
  redrive_policy             = count.index == 0 ? null : "{\"deadLetterTargetArn\":\"${aws_sqs_queue.queue[count.index - 1].arn}\",\"maxReceiveCount\":${var.queue_definitions[count.index].maxReceiveCount}}"
  visibility_timeout_seconds = var.visibility_timeout
  kms_master_key_id          = var.kms_key_id
}

...

Module declaration:
module "sqs-backoff-sample" {
  source     = "./modules/sqs-exponential-backoff"
  name       = "clete-sqs-test-"
  kms_key_id = var.kms_key_id
  account    = var.account
  region     = var.region
  queue_definitions = [
    { maxReceiveCount : 2, delaySeconds : 0 },
    { maxReceiveCount : 4, delaySeconds : 5 },
    { maxReceiveCount : 6, delaySeconds : 15 },
    { maxReceiveCount : 8, delaySeconds : 60 }
  ]
}

Debug Output

Nothing interesting other than: Error: Cycle: module.sqs-backoff-sample.aws_sqs_queue.queue[3], module.sqs-backoff-sample.aws_sqs_queue.queue[2], module.sqs-backoff-sample.aws_sqs_queue.queue[1], module.sqs-backoff-sample.aws_sqs_queue.queue[0]

Crash Output

Expected Behavior

I am trying to design a loop such that each queue cascades into the queue below it. The goal is to create an exponential backoff scenario where retries are slowed down as the failed messages fall into the lower queues (see queue_definitions above).

The key issue is this line: redrive_policy = count.index == 0 ? null : "{\"deadLetterTargetArn\":\"${aws_sqs_queue.queue[count.index - 1].arn}\",\"maxReceiveCount\":${var.queue_definitions[count.index].maxReceiveCount}}"

For the redrive policy, I need to specify the previous queue's ARN such that queue 4 dumps into queue 3, 3 into 2, 2 into 1, and 1 is the final queue (count.index == 0 thus no redrive).

Actual Behavior

Terraform incorrectly identifies a cycle between 3->2->1->0, but in reality index 0 does not depend on any other queue.

Steps to Reproduce

Dynamically create multiple queues by putting the "aws_sqs_queue" into a module and then initializing it with the inputs given. Try to apply.

Additional Context

If you can think of another way to implement dynamically defined n number of queues that overflow into each other in case of failures, I am all ears.

References

Dec 20 '19 16:12 Clete2

Hi @Clete2,

Thanks for the example config and use case.

This currently isn't supported in terraform. For various reasons dependencies are tracked at the resource level rather than individual instances, and references to resources need to first evaluate the entire resource as whole. This means that references to other instance indexes within the resource config is the same as a self reference and hence a cycle.

You're correct that there isn't a way to do this for any N resource at the moment, and the individual resources would have to be written out separately.

Dec 20 '19 19:12 jbardin

@jbardin Thanks for your reply; I appreciate it. I'd like to see more advanced use cases supported in Terraform like conditionals being supported on all attributes and loops being more robust / less limitations. I hope this can come in the future.

Dec 20 '19 19:12 Clete2

@jbardin is there something in the roadmap that will address this? Not sure if enabling depends_on for modules will help or address this.

Dec 23 '19 14:12 karl-cardenas-coding

While this is something we are going to look into in the future, it's going to take some careful consideration because it's not how terraform was originally intended to operate.

Jan 02 '20 14:01 jbardin

I've run into the same thing trying to setup a chain of (nearly identical) incoming SES rules where there "after" entry should refer to the previous aws_ses_receipt_rule. This is another vote to support such dependency at the instance level.

Jul 19 '20 02:07 mtesch-um

After the deeper analysis done through the 0.13 development cycle, it's clear that terraform as it exists now cannot support this type of self-reference. We can keep this open for future enhancements, but this is not something that can be added to the roadmap in the foreseeable future.

Jul 20 '20 17:07 jbardin

Thanks for the update @jbardin. For now I have abandoned attempting this sort of approach. For those looking for a quick fix on this issue, you should either utilize copy/paste to chain resources together, or look at something like CDK or Pulumi.

Jul 20 '20 17:07 Clete2

@jbardin I'm also voting for this feature to be added. :+1:

I have a resource for GitHub teams, which can have parent teams and there is currently no real good solution to address this. If I was able to reference my resource inside the resource, I would be able to accomplish something like this (this would pretty much give us recursion):

terraform.tfvars:

teams = [
  {
    name        = "DevOps"
    description = "The DevOps Team"
    privacy     = "closed"
    parent_name = null
    members = ["cytopia"]
  },
  {
    name        = "Engineerng"
    description = "The Engineering Team"
    privacy     = "closed"
    parent_name = "DevOps"
    members = ["cytopia"]
  },
]

main.tf:

locals {
  teams = { for index, team in var.teams : team.name => team }
}

resource "github_team" "team" {
  for_each = local.teams

  name           = each.value.name
  description    = each.value.description
  privacy        = each.value.privacy
  # The following fetches the parent team id of this resource, hence Terraform should ensure to
  # create this before executing the current iteration (recursively, in case the parent team also has a parent team)
  parent_team_id = each.value.parent_name != null ? github_team.team[each.value.parent_name]["id"] : null
}

Jan 18 '22 14:01 cytopia

Another use-case: creation of Keycloak roles via a declaractive approach (from YAML), using mrparkers/terraform-provider-keycloak:

roles:
  - id: vehicle_viewer
    description: Can view vehicles.
    composite: null
  - id: vehicle_manager
    description: Can manage vehicles.
    composite:
      - vehicle_viewer
  - id: &vehicle_admin vehicle_admin
    description: Can administer vehicles.
    composite:
      - vehicle_viewer
      - vehicle_manager

resource "keycloak_role" "eos_roles" {
    for_each = [ for i in yamldecode(file("${path.module}/eos-authorization-model.yaml")).roles : i.id ]

    realm_id    = keycloak_realm.realm.internal_id
    client_id   = keycloak_openid_client.connectivity_api_client.id
    name        = each.key
    description = each.value.description

    composite_roles = each.value.composite == null ? null : [ for i in each.value.composite : keycloak_role.eos_roles[i].id ]
}