Allow pure (side effect free) variables in terraform blocks

[nwmcsween]

Current Terraform Version

Terraform v1.2.6
on windows_amd64

Use-cases

One of the key features of terraform wrappers.

Attempted Solutions

Terragrunt

Proposal

Allow pure (side effect free) sources within terraform blocks due to the chicken and egg issue of querying state requiring state.

The hard part is what can be defined as pure.

References

Dozens, this is a commonly asked feature.

[apparentlymart]

Hi @nwmcsween! Thanks for sharing this feature request.

The major design question that needs to be answered to move forward with a design like this is:

Backend initialization currently happens during terraform init, which isn't a Terraform "run" (plan+apply) and so doesn't have access to anything that would normally be available during Terraform evaluation. Although this distinction is not clear from a UI perspective, there is an architectural boundary between "Terraform CLI" which is responsible for the concept of backends and their management, and "Terraform Core" which is responsible for the main Terraform language.

Therefore we'd need to figure out what would be an appropriate architectural change to meet this request. There have been various early ideas in the past and I don't have a full manifest of them to hand, but here are some I remember off the top of my head:

• Remove the idea of "backend initialization" altogether and dynamically configure state storage along with all of the other things Terraform Core does. This would in principle allow the backend configuration to access anything visible in the root module, but it has several drawbacks:

  • There would be no opportunity to propose automatically "migrating" state from one backend to another when the configuration changes. Instead, a configuration change would require the user to run an explicit migration step first or otherwise possibly end up creating an entirely new state instead of updating the old one.
  • There is some question about what happens if there are side-effects that happen before the state storage initialization occurs but then the state storage initialization fails. In your proposal you specifically called out "side-effect-free", which is indeed one way to avoid this drawback if we can find a sufficiently tight definition of "side-effect-free" that we can decide during the planning phase. If we decided to take that compromise, it would require a precise definition of what is allowed and what is not, and how Terraform Core will enforce that.

• Move the idea of input variables to the terraform init command and make that command "remember" somehow the variable values that were chosen during initialization as part of the working directory state, and remove the ability to set variables on an individual plan basis.

  This one is essentially off the table now since the v1.0 compatibility promises have frozen the current modelling of input variables. It could potentially come in a hypothetical breaking Terraform v2.0 release, but we have no current plans to do that and the overhead of planning and executing such a breaking release would, I think, be significantly larger than implementing the design change.

• Change the modelling of "workspaces" so that each workspace has a state storage rather than each state storage having a set of workspaces. This would mean that there would either no longer be a mechanism for listing available workspaces or the list of available workspaces would need to be specified within the configuration itself, rather than in the remote system. The state storage configuration could therefore at least include references to terraform.workspace to dynamically vary based on the selected workspace, if nothing else. If there were a local configuration file specifying which workspaces are available then we could potentially expand the idea of terraform.workspace to be a bag of arbitrary key/value pairs, rather than just a single string.

  This is more-or-less what wrapping tools like Terragrunt are doing, albeit doing it in a kinda clunky way as a result of having to achieve it through terraform init's existing design rather than changing terraform init to handle it directly.

  However, doing this withing Terraform itself risks running into similar compatibility challenges as the previous point. We did carve out the terraform workspace ... family of subcommands from the compatibility promises specifically to leave a narrow avenue open for this, but it remains to be seen whether a detailed design in this direction would fit through that narrow avenue in practice.

At the current time it's both unclear which of these paths (or some other path) is the best one to take, and all of them have design challenges that need to be resolved before moving forward. This enhancement request is therefore blocked on their being a viable detailed proposal to move forward with, even though I'd agree that the current design has various undesirable limitations.

---

In the meantime, the standing working approach is to use terraform init with the -backend-config option to dynamically specify systematic backend configuration settings. Because that's clearly inconvenient to use when running terraform init directly, I would recommend anyone using that approach to use some kind of wrapping automation to do it. Such wrapping automation doesn't necessarily need to have such a large scope as community solutions like Terragrunt; if this particular problem is your only focus then it is viable to wrap only terraform init, and then use the Terraform commands directly for other operations. (If you have other desires for automation then of course that might benefit from wrapping other commands too; Running Terraform in Automation has some guidance for those who want to implement more thorough automation that covers the entire end-to-end flow.

[nwmcsween]

As I don't know terraform internals my input could be considered noise, so treat it as such and ignore (but if there is docs on this let me know!) if it takes too much effort to reiterate on this from previous discussions.

Could terraform introduce a constant resource that is required to be resolved at init or at runtime with a data source assuming data sources are side effect free?