resource-instance-scoped named local values

[thegedge]

Sometimes I find that I repeat myself in resources. For example:

resource "aws_subnet" "foo" {
  count = 3
  availability_zone = "${var.region}${element(split(",", var.availability_zones), count.index)}"
  tags = {
    Name = "private-${var.region}${element(split(",", var.availability_zones), count.index)}-subnet"
  }
}

I was thinking it would be great to have self references here, as sometimes these interpolations can be somewhat complex and I end up having to copy and paste. For example:

resource "aws_subnet" "foo" {
  count = 3
  availability_zone = "${var.region}${element(split(",", var.availability_zones), count.index)}"
  tags = {
    Name = "private-${self.availability_zone}-subnet"
  }
}

Not sure if it's possible to do this in a nice manner at the moment?

[apparentlymart]

Implementing something like what you described could potentially be allowed as long as the attribute in question is not computed, which is to say that its value is a known constant at plan time. Variables and interpolation functions applied to variables would fit into that rule, as could count.index since repetition of resources is also handled at plan time.

Currently the only place where there is special handling for plan-time-available values is in the special count argument itself, where only variables are allowed. Perhaps this concept could be generalized so that self can use it too. I know I've wanted to do stuff like this as well, rather than duplicating complex expressions all over the place.

A tricky part of this is that it creates dependencies within the attributes of the resource themselves. While it's not true in your example, you could easily create a situation where there is a cycle of dependencies between attributes, which Terraform would then need to detect and report. This would likely require the interpolator or config processor to retain some more state than it does today.

[radeksimko]

Maybe having self references which would work in the context of the resource itself and would be interpreted in the Update phase of each resource may be the way forward...

Ain't saying it makes things easier for the actual implementation as I can imagine the amount of changes we'd have to make in the lifecycle & interpolation, but it's just some food :hamburger: for thought :thought_balloon: .

[jleclanche]

This would really be nice. Here's an example with aws_s3_bucket resources:

resource "aws_s3_bucket" "example_org" {
    bucket = "blog.example.org"
    logging {
        target_bucket = "logs_bucket"
        target_prefix = "logs/${self.bucket}/"
    }
}

resource "aws_s3_bucket" "www_example_org" {
    bucket = "www.example.org"
    website {
        redirect_all_requests_to = "${aws_s3_bucket.example_org.bucket}"
    }
}

[tkellen]

This is related to #5805

[stormbeta]

This would make a lot of our stuff cleaner - I don't need full-blown self access, but I'd at least like to be able to refer back to the resource name within that resource's block, otherwise it makes for a ton of error-prone duplication across tags, descriptions, and collections of related resources.

Variables only help a little with that, because you can't interpolate resource names either.

[leighmhart]

yes, definitely, ${self.tags.Name} would be a common and useful case (in a remote-exec provisioner, for example, to netdom renamecomputer).

[dimisjim]

@apparentlymart Are there plans to include this in 0.12 ? It would be quite useful.

[apparentlymart]

There are no plans to include this in Terraform v0.12. The scope of Terraform v0.12 is now closed, because its release process is already underway.

As I described in my earlier comment, there are some design challenges to solve before this could be implemented. So far we've not had time to work on those, so work here is blocked until there is time to think through what restrictions are needed to make this behave in a deterministic way while still being useful, and how best to implement it since today the arguments inside a block can be evaluated in any order while implementing this feature would effectively make the arguments themselves a dependency graph, which is a considerably more complex design.

We implemented local values several versions ago to address use-cases like this in a different way, by allowing repeated expressions to be hoisted out into a named symbol that can then be used in various locations:

locals {
  bucket_name = "blog.example.org"
}

resource "aws_s3_bucket" "example_org" {
  bucket = "${local.bucket_name}"
  logging {
    target_bucket = "logs_bucket"
    target_prefix = "logs/${local.bucket_name}/"
  }
}

The above is the recommended way to address this use-case for now.

One thing that we cannot do with local values as currently implemented is instance-specific local values that can refer to count.index, within a resource block that has count set. For 0.12 we've reserved the name locals when used inside resource blocks. It doesn't do anything yet because there's a lot more implementation work to do to make it work, but our intent was to eventually support resource-scoped local values, which might look something like this:

resource "aws_s3_bucket" "example_org" {
  count = 5
  locals {
    bucket_name = "example${count.index}"
  }

  bucket = "${local.bucket_name}"
  logging {
    target_bucket = "logs_bucket"
    target_prefix = "logs/${local.bucket_name}/"
  }
}

The idea here would be that because this bucket_name local is defined inside the resource block we can evaluate it separately for each instance of the resource, and thus allow count.index to be used to generate a different value each time. This would be considerably simpler from an implementation standpoint because it separates the idea of re-usable values (which need to be evaluated in a dependency-aware fashion) from resource attributes (which can then still be evaluated in any order). I think (subjectively) it's also a simpler user model too, because it doesn't require any unusual restrictions on what can and cannot be used inside that block but rather behaves the same way as expressions elsewhere in the resource block.

Reserving the locals name as a "meta-argument" is the first step down this path for 0.12. I don't know yet when we'll be able to attack the rest of this, since once 0.12 is done we will need to devote some time to non-language-related parts of Terraform that have been more neglected during the 0.12 development cycle, but this scoped-locals design seems more promising (in terms of us knowing what an implementation of it might look like) and something we want to investigate further in a later release.

[wyardley]

Any new news on this one? A good example use case:

Let's say you have a bunch of these:

resource "google_cloudbuild_trigger" "foo_bar" {
  name               = "foo-bar"
  substitutions = {
    _APP_NAME = "foo-bar"
  }
}

it's not really worth making a separate local var for each reference, but it would be very nice to be able to do self.name or google_cloudbuild_trigger.foo_bar.name, neither of which works now.

[apparentlymart]

Hi @wyardley!

I think we must have a duplicate of this issue somewhere, because I know I've already posted what I'm about to write somewhere before, but since I can't find it I'll try to summarize it here.

With the later addition of the for_each argument for resource blocks, it became possible in principle to use the for_each data structure along with each.value to get an effect which seems functionally equivalent to local values on a per-instance basis, because the each object is already scoped to a particular resource instance and can contain arbitrary data:

resource "aws_subnet" "foo" {
  for_each = {
    for az in var.availability_zones : az => {
      name = "private-${az}-subnet"
    }
  }

  availability_zone = each.key
  tags = {
    Name = each.value.name
  }
}

For comparison, here's a hypothetical example using the unimplemented "local locals" design I mentioned in my earlier comment:

# INVALID: This is a hypothetical example of an unimplemented language feature
resource "aws_subnet" "foo" {
  for_each = toset(var.availability_zones)
  locals {
    name = "private-${each.key}-subnet"
  }

  availability_zone = each.key
  tags = {
    Name = local.name
  }
}

When I asked about it in whatever other issue we were discussing this in, the consensus seemed to be that using each.value as a per-instance set of local values was close enough to local locals to significantly reduce the impact of implementing local locals, and thus we put it on the back-burner and refocused attention elsewhere.

---

Of course, this is addressing the original request for local values that vary on a per-instance basis, whereas your example here is a singleton resource and therefore doesn't really need per-instance data, since you can already factor out any data that is either entirely constant or only relies on global objects into a top-level locals block:

locals {
  app_name = "foo-bar"
}

resource "google_cloudbuild_trigger" "foo_bar" {
  name = local.app_name
  substitutions = {
    _APP_NAME = local.app_name
  }
}

I think then what you have in mind is just that you'd find it stylistically preferable to nest this local value inside its resource block because you know it's not useful in any resource other than this one. I can certainly see where you're coming from on that one, and indeed if we did have local locals implemented I'd probably write it this way too:

# INVALID: This is a hypothetical example of an unimplemented language feature

resource "google_cloudbuild_trigger" "foo_bar" {
  locals {
    app_name = "foo-bar"
  }

  name = local.app_name
  substitutions = {
    _APP_NAME = local.app_name
  }
}

The translation of this one to the for_each-based solution is counter-intuitive because there would of course only be exactly one instance always:

resource "google_cloudbuild_trigger" "foo_bar" {
  for_each = toset(["foo-bar"])

  name = each.key
  substitutions = {
    _APP_NAME = each.key
  }
}

I'm not suggesting that this last example is desirable or a good idea, but of course the fact that there are already two different ways to write the same thing with existing language features naturally leads to de-prioritizing any new solutions to these problems in relation to other problems that don't have existing solutions.

---

I should of course also address the other design idea that we were discussing in this issue (indeed, perhaps this other approach is why this one remains open even though we alerady have another for local locals) of allowing self-references inside the main body of a resource block, like these examples:

# INVALID

resource "aws_subnet" "foo" {
  for_each = toset(var.availability_zones)

  availability_zone = each.key
  tags = {
    Name = "private-${self.availability_zone}-subnet"
  }
}


resource "google_cloudbuild_trigger" "foo_bar" {
  name = "foo-bar"
  substitutions = {
    _APP_NAME = self.name
  }
}

Although I can see the appeal of this language design, I don't think it's really practical to implement it because it would effectively require Terraform to build a dependency graph of all of the individual arguments inside a resource block and resolve them in a particular order, whereas today all of the arguments are independent of one another and Terraform Core's implementation exploits that by just visiting them all in whatever order comes naturally from traversing the underlying data structure, and all of that work is outsourced to HCL so Terraform Core just sees the final object and doesn't actually know exactly how it was constructed.

I think the "local locals" design is more likely, because it has a clearer execution model: evaluate the special locals block first, and then evaluate everything else in the usual way but with some extra values in the symbol table.

Terraform Core's design doesn't really make either of these easy today, because it treats count and each both as special-cases which are the only symbols that can vary on a per-instance basis; everything else gets resolved globally, including local. But while local locals are not easy, I can at least imagine a viable path to them which is localized within the Terraform Core codebase, without the need for also changing the HCL API.

[wyardley]

I think I did probably come across some other threads about this.

I guess on an intuitive level, it feels like it should be possible to refer to a (non-computed) attribute of the current object, but I can understand that seeming like it should be easy doesn't make it so, and I think I understand what you're saying about why it would be challenging. So, to me, the self reference reads the cleanest / is the easiest to understand.

I like the locally scoped locals approach ok, and would use something like that for this situation if it existed, though I imagine it might be confusing to people who are used to terraform's current implementation of locals, which is maybe too flexible.

[wyardley]

into a top-level locals block:

locals {
  app_name = "foo-bar"
}

Ok, so my example was intentionally oversimplified, and yes, in that case, locals=> app_name would work well enough, but now imagine there were 30 of these blocks that, for whatever reason, I wasn't creating via iteration (for example, there are some other differences between them).

So now I don't need locals => app_name but rather a full map, which I have no way to reference without using the name once. That is, imagine the following block, but let's say x 20 / 30?

resource "google_cloudbuild_trigger" "app1" {
  name = "app1"
  substitutions = {
    _APP_NAME = "app1" // what I'm wanting to use `self.name` or `local.app_name` for
    _FOO            = "bar"
  }
}

resource "google_cloudbuild_trigger" "app2" {
  name          = "app2"
  substitutions = {
    _APP_NAME = "app2" // ditto
    _QUX            = "widget"
  }
  some_other    = true
}

If I'm not mistaken, there's no convenient way when not using iteration to use a top level local var to do the thing I'm proposing here, since if I built an array, I'd have to access it by numeric index, and if I built a hash, I'd have to use the name to refer to the name, which wouldn't help.

locals {
  app_names = {
    app1 = "app1"
    app2 = "app2"
  }
}

obviously, this is still an oversimplified and somewhat contrived example, but does that help explain how a scoped local var would be useful in a scenario like this in a way that a top level one wouldn't be?