hashicorp
Allow `replace_triggered_by` to exist in a cycle
Terraform Version
Terraform v1.2.7
on linux_amd64
Use Cases
I am in the situation where I use Terraform to create a one-time secret to help in provisioning a VM. This secret needs to always be recreated when the VM is recreated, will not change otherwise, and the VM depends on the value of this secret.
Attempted Solutions
A naive implementation would be to have the VM consume the secret (creating a dependency), and then add replacement_triggered_by to the secret. However, this creates a cycle error.
The only workaround I can find is as follows:
- Add a provisioner to the secret resource, storing it on disk
- Add a timer that will always be recreated using a trigger
- Have the VM
depends_onthe timer - Read out the secrets file during VM creation/provisioning
- Have the secret be
replace_triggered_bythe VM
Proposal
Currently, this scenario causes a cycle error: VM depends on secret, which depends on the VM, etc. However, I would say that in the particular case of replace_triggered_by, there is a logical argument to be made that this is not strictly a cycle: the secret only needs to be replaced when the VM needs to be replaced for other reasons. So I could envision the code handling the replace_triggered_by to work recursively: first find all resources that need replacement for other reasons, disregarding r_t_b; then for each resource marked for recreation check if other resources depend on it and mark those for replacement, etc etc. This "solves" the cycle in this case.
Another solution would be a more general way to deal with these "ephemeral" resources, like secrets and other things that only need to be used once and after which Terraform doesn't really have to care about their state.
References
No response
Hi @FWest98! Thanks for starting this discussion.
I think a crucial thing we'd need to figure out to decide the feasibility of this proposal is: can we define the replace_triggered_by behavior such that it's independent of normal action planning behavior.
A key challenge is that Terraform needs to know whether there's a change pending for the VM in order to decide whether it needs to replace the secret, but Terraform also needs to know the planned new state for the secret in order to decide whether there's a change pending for the VM.
One potential avenue for breaking this cycle is to see if we can get any benefit from noticing that replace_triggered_by only needs to know the planned action of the other resource (create, update, no-op, etc), whereas a normal reference-based dependency needs to know the full planned new state of he other object. If we can find some way to define it so that the planned action of a resource can be decided separately from evaluating its planned new state then it may be possible to break the cycle by splitting the resource planning actions into two parts:
- Plan secret
- Plan VM
- Decide final action for VM
- Decide final action for secret
Since we know that replace_triggered_by effectively forces a particular action for the resource it's attached to, in theory we could initially plan "secret" and see that it is a no-op, but then separately later decide to override that action with a "replace" once we've seen what action was chosen for the VM.
That's a pretty significant change to the way Terraform does its planning work though, so I think it would take some prototyping to see if it's really possible to break that dependency and to see how significantly it would change the internal architecture of Terraform. If it requires non-trivial redesign then the benefit would need to be pretty high to justify that.
Thanks again!
Thank you for the quick and extensive reply! I certainly understand that this is not easily solvable, and would possibly require significant work in the core planning logic.
For now, is there any way to more neatly "circumvent" this issue? All online tutorials I find solve this specific issue by providing the one-time-use secret in some other way (through chef, puppet, ansible, or some other tool), so that Terraform is just used for the "static" configuration. In my setup, adding another tool would be overkill as most provisioning happens after the VM has authenticated.
I think I'd need to see a more complete example of what you have tried so far in order to suggest potential changes, but that would risk turning this issue into a discussion about how to use Terraform as implemented today rather than a feature request for a future change, so it'd probably be best to start a topic in the community forum and then we can discuss over there some potential different ways to solve the problem you have with Terraform's current design.
Valid point. I will expand this issue with a few demo use cases for future reference.
Deploying Rancher Server using Terraform
Deploying Rancher Server requires a so-called "bootstrap password" to be set, which will serve as a one-time token to access the configuration interface to set up proper authentication for the system. The rancher2_bootstrap resource will then use this password to perform the bootstrapping. This is a one-time use password, so ideally we would like to use a random_password to serve this role: we create it in Terraform, send it to our machine as an option in our Helm chart, and later use it to bootstrap everything.
However, when we now need to replace the machine for whatever reason, this password does not get rotated. Instead, we either need to manually taint or pass -replace for the password resource. This MR is about allowing this password resource to be replace_triggered_by-dependent on the machine resource (while the machine resource is content-dependent on the password).
Provisioning a server with Vault AppRole
This one goes a bit against the "official" tutorials. Following the tutorials, Terraform finds the AppRole RoleID and sends it to the machine during provisioning, after which a third-party tool (Chef in the tutorials) sends over a wrapped SecretID so the machine can authenticate and obtain a token for whatever it needs to do. My particular setup is not compatible with this procedure (for example: I sign my SSH host keys automatically at startup, but that needs Vault, and without signed host keys no other provisioning tool should want to connect since the host keys can't be verified).
In my case, I have my RoleID embedded in the machine template/image that is stored outside of Terraform, and I would like Terraform to generate a SecretID and send it to the machine while provisioning. Probably not perfectly safe, but together with strict CIDR settings on the token it is sufficient for me. The situation is now similar to before: I have to regenerate the SecretID token every time the machine is recreated, but I would not even want to regenerate one when the machine is untouched. So the machine is content-dependent on the SecretID token, while ideally, the token is replace_triggered_by-dependent on the machine.
(take this section with a grain of salt, just some thoughts floating in my head while writing the previous section and thinking more about it)
I think the common pattern in these examples is that there really seem to be multiple "types" of dependencies: some resources depend on each other in terms of content (a machine needs the output of a config transpiler, for example). On the other hand, some resources are only dependent on others in terms of their lifecycle (e.g. they need to be recreated together; which is why the replace_triggered_by attribute exists). And even other resources might depend on others in a temporal sense (e.g. a dependency on a time_sleep resource).
In practice, these dependencies are mostly interchangeable in terms of the dependency graph and whatever planning TF does: e.g. for planning the order of operations it is not relevant whether resources are content-wise or temporally dependent. However, in this MR I think we would need to introduce some distinction in a "type" of dependency, which I think will give a more natural interpretation on how to tackle the scenarios above (and there are probably many more to think of).
Other thought: both of these examples revolve around some kind of "ephemeral" resource that only assists/aids in the creation of another (set of) resource(s), while we don't truly care about the lifecycle of the resource after the deployment is complete (ideally, in these cases, the resource should even be destroyed!). Maybe it would be easier to add some dedicated functionality to support these patterns better, instead of rethinking the dependency structure? (like; add a key to a resource needed_for_creation that will only create and then destroy the resource when the listed resource(s) is/are created?) Maybe this issue/idea of rethinking the dependencies is the wrong solution for the problem of "ephemeral assistive resources".
I think this "types of dependency" line of thinking is a good way to summarize why it might be possible to reorder these operations so that there isn't a cycle.
In practice though there can only be one (partial) order of operations, and defining that partial order is what Terraform's execution graph represents; we do ultimately need to be able to describe only one graph which Terraform can then walk to perform actions in a correct order.
For that reason, this sort of goal tends to lead to splitting existing single graph nodes that do multiple operations into "smaller" nodes that all represent the same object but perform different actions against it.
"Replacing" is actually already an example of that during the apply step: Terraform splits that into two graph nodes where one creates the new object and the other destroys the existing object. This then allows the destroying order to differ from the creating order.
That's the sort of thought process I was starting when I listed out four actions in my earlier comment: "decide the final action for..." could potentially be a separate graph node with different dependencies. Thinking down that path a little more, some potential graph building rules could be:
- Any resource with
replace_triggered_bygets this extra node created for it to handle the final decision about action.- The new node depends on the main planning node for the resource, causing it to happen after it.
- For each resource mentioned in a
replace_triggered_by, also create for it a "decide action" node if it doesn't already have one. - Create dependency edges between those new nodes based on the resources listed in
replace_triggered_byarguments. (And we would no longer considerreplace_triggered_byfor the normal dependency analysis, because we know that can never consume data from what it refers to, only the action decision.)
An important implication of the above (assuming those rules actually work; I've not tried it with a real thought experiment yet) is that it would become possible to "strengthen" the action for a resource as a separate step after its initial planning: it could change from no-op to replace or from update to replace.
I think the wildcard here we would need to investigate further is that changing action to "replace" is in practice actually a matter of throwing the original plan away altogether and making a new "create" plan for the object, so that the provider gets the opportunity to reset any attributes that won't survive recreation. That means that any object with a data dependency on a resource will need to also depend on its "decide final action" node because that can potentially change the planned new state data for the object.
I think therefore the main question we need to answer here, either by thought experiment or by prototyping, is whether this restructuring actually does end up eliminating the dependency cycles, or whether that requirement to decide the final action before visiting any dependencies just puts us back in a dependency cycle again.
Thanks @FWest98,
This type of "cycle" has come up a lot recently, I think because replace_triggered_by is so close to solving these types of problems, but can't look back to a previously planned resource, or the resource itself.
To give some more details about why this type of handling was not possible, let's take the example from above:
Plan secret
Plan VM
Decide final action for VM
Decide final action for secret
Which sounds possible when seeing it in a linear form, but this can be linked in a multitude of ways into a much more complex graph. once we get to the final Decide final action for secret step, it would require re-planning the secret, however any number of other resources could have already been planned based on the initial result of secret, which would then be invalidated.
Another observation I have after thinking down this path is that our existing logic for changing an update into a replace if the provider reports that it cannot make a change in-place could in theory happen in this separate graph node too, whereas today it's just built in to the resource's main graph node. That would mean that there's still only one place where that sort of re-planning would happen, but it would be ordered independently from the original planning call.
That would effectively mean that every resource would have two nodes in the graph though, which will roughly double the size of the graph and may adversely affect those who are using Terraform with very large configurations (despite our recommendations not to).
(continuing the last comment, which I cut short to read the concurrent replies 😉)
There's a few options that I've looked into here
- Back-propagating the information through the graph, invalidating prior plans as we go. The complexity here is quite high for such infrequent edge cases, and would require a huge amount of refactoring to achieve.
- Re-running the entire plan once new information has been decided. This is essentially the same as above, but we just assume the whole graph is affected so we don't need to try and carry the information back through the dependencies, though we would however still need to carry some extra data forward to prevent cycles of re-planning. This could have a very high overhead for large configs, and may be difficult to communicate problems causing long planning processes to users.
- Find a new mechanism to isolate discrete sections of the graph which are interdependent, so that we can be sure no side effects can take place until the group of nodes is complete. This could be interesting to research, and might be usable in other contexts, but again would require significant changes within core.
I'm running into the same issue, in my case it is an EC2 instance, on which I want to install Tailscale, and use a one time API key to set it up. I have the option to set up a reusable key, but I don't like that idea at all since it would then be retrievable to any ssh user on that box by looking at the userdata. As things stand I guess I will add a note to the code telling people to taint the API key resource if they want to replace the box, but I don't like that idea either, since the patching process will most likely be to scrap and recreate the instance, so will happen fairly frequently.
@jbardin , @apparentlymart Are there any plans to solve this? I'm actually not sure this issue is such an edge case.
I'm also encountering this issue. In our use case, we're using a null-resource to run a local-exec provisioner to fetch some SSH credentials and add them to the agent before running a remote-exec provisioner (which needs those credentials) in another resource.
Our workaround is to remove the resource dependency and just chain the provisioners in a single null_resource, but that is less than optimal (especially in cases where we'd want to fetch the credentials only one time for multiple subresources depending on a single null_resource that should be replaced at most 1 time even when multiple subresources are added or updated.)
My sense is that the current state of this issue is in figuring out what is even possible. Without any concrete sense of what exactly it means for "replace_triggered_by to exist in a cycle" there isn't really anything more we can do with this issue at this time.
So far my sense is that this may not actually be possible at all, because changing the action decided for one resource can entirely change the plan for another. The most promising direction discussed so far seems to be to just keep re-running the plan phase until it eventually converges on a stable plan, but it isn't clear to me that this will work in the general case because we cannot guarantee that it will ever converge. For example, a provider might return different values each time it's asked to generate a plan, and so each re-run would generate a new result and it would never land at a consistent final plan to take to the apply phase.
We're open to ideas for how to make this work but personally I'm not currently feeling optimistic that it's technically possible to solve this, even though I'd agree it would be nice in theory.
As I mentioned before, it might be worthwhile to look at this issue a bit more pragmatically. The two examples mentioned here are again cases where some one-off resource is needed to create/manage another; in particular an "ephemeral" resource that you would want to destroy/invalidate after runtime from a security perspective. So: one-time API keys, temporary SSH credentials, etc.
So instead of overhauling the entire dependency graph for a relatively isolated usecase (ephemeral one-off resources), maybe it's an option to make dedicated functionality for handling these cases. Some special kind of resource that does not participate in the normal resource lifecycle, and will only exist for the duration of the Terraform runtime, for example. There are probably plenty other ways to go about this, without having to do a major overhaul of the dependency analysis - which would of course be very tricky and costly.
I'll throw another concrete example of where I'm running into this and where some way of breaking the cycle would be helpful.
I'm using the ACME provider.
The acme_registration resource requires a private key. We can look at the example given, which uses tls_private_key to generate that key:
provider "acme" {
server_url = "https://acme-staging-v02.api.letsencrypt.org/directory"
}
resource "tls_private_key" "private_key" {
algorithm = "RSA"
}
resource "acme_registration" "reg" {
account_key_pem = tls_private_key.private_key.private_key_pem
email_address = "[email protected]"
}
If the acme_registration resource is destroyed, it performs an irreversible deactivation of the account with Lets Encrypt (I'm not certain if that happens with other ACME providers but it doesn't matter for this discussion).
At that point, any attempt to re-register the account, or use it in any way, will fail.
One way this can become an issue is changing the name of a resource, or moving it into or out of a module, or something similar, while the tls_private_key resource does not move.
The result is a replacement of the acme_registration resource using the same private key, which will fail.
What I'd like to do is something like this:
provider "acme" {
server_url = "https://acme-staging-v02.api.letsencrypt.org/directory"
}
resource "tls_private_key" "private_key" {
algorithm = "RSA"
lifecycle {
replace_triggered_by = acme_registration.reg
}
}
resource "acme_registration" "reg" {
account_key_pem = tls_private_key.private_key.private_key_pem
email_address = "[email protected]"
}
The implication being: if the registration changes in any way, w should also replace the private key in order to create a new registration based on that key, because anything else will result in bad failures (often requiring state surgery).
@FWest98
The two examples mentioned here are again cases where some one-off resource is needed to create/manage another; in particular an "ephemeral" resource that you would want to destroy/invalidate after runtime from a security perspective. So: one-time API keys, temporary SSH credentials, etc.
This idea could be promising, but these "one-off" things are usually not exactly ephemeral. The value does need to persistent between runs, until it doesn't (basically, until its dependent resource is replaced).
In essence, replace_triggered_by is exactly what these cases need, except with the dependency direction reversed.
@apparentlymart
any concrete sense of what exactly it means for "replace_triggered_by to exist in a cycle
Perhaps a new lifecycle argument, with simpler semantics could be created, like replaced_with or replaced_together, which doesn't take attributes, only resources, and its meaning is simpler:
when any of the resources in the list will be replaced, this one will too
Maybe implementing this ends up being the same problem in terms of complexity, I'm not sure.
Thanks for sharing that example, @briantist. I don't have a ready answer yet for how to make that work given what I mentioned above about the required order of operations to decide whether there is actually a change, but adding another real example into the mix might help suggest a different way to approach it that would avoid that chicken/egg problem.
As with the other examples we've discussed, the challenge here is that whether there is a change pending for acme_registration.reg depends on the value of tls_private_key.private_key.private_key_pem, but there won't be a change for that value unless either the tls_private_key.private_key configuration has changed or (with the replaced_triggered_by you want to add here) unless there's already a change pending for acme_registration.reg.
Therefore this is undecidable without introducing some additional information from somewhere. Some additional information you mentioned in prose in your comment is that in the problematic situation acme_registration.reg has been destroyed, and therefore Terraform could in principle know it's definitely going to need to take an action against acme_registration.reg (creating it), regardless of what value tls_private_key.private_key.private_key_pem has.
I think "replace this if this other thing is going to be created" could be a more specific statement that would potentially break the cycle. That primitive also seems like it could solve the original problem statement of regenerating a secret whenever a VM is created. I don't really know what is a reasonable name for an argument to specify this rule, but regardless of what it's named I think we'd need to try a prototype implementation of it first and see if it works in practice (as opposed to in thought experiment) and then, if so, we can ponder what the best name for it would be. This seems like a good candidate for an experimental feature so that we can ask folks to try it in some different situations without it immediately falling under the v1.x compatibility promises until it seems clearer what is good syntax and internal implementation.
We do also have an early design sketch internally for something like this "ephemeral resources" thing discussed above, originally aimed at use-cases like those discussed in #8367 and #24886, but I agree with @briantist that such a model doesn't seem sufficient to fully resolve the use-case presented here, since a key part of that (very early) design is that it isn't allowed for a managed resource (a resource block) to refer directly to an ephemeral resource, since the ephemeral resources get cleaned up immediately after terraform apply has finished its other work and so would immediately become invalid. However, I think it is still interesting to consider all of these needs together when evaluating different designs here; if there were some idea of ephemeral resources then we'd need to figure out how/whether that concept would interact with "replace triggered by".
Thanks again!
@apparentlymart first off, thanks so much for your thorough and detailed reply!
I think "replace this if this other thing is going to be created" could be a more specific statement that would potentially break the cycle. That primitive also seems like it could solve the original problem statement of regenerating a secret whenever a VM is created.
That seems reasonable and a better explanation of the logic.
I don't really know what is a reasonable name for an argument to specify this rule, but regardless of what it's named I think we'd need to try a prototype implementation of it first and see if it works in practice (as opposed to in thought experiment) and then, if so, we can ponder what the best name for it would be.
replace_triggered_by_creation_of (offhand idea)
The ephemeral resources thing does sound interesting, but yeah it wouldn't apply to my example.
Hi,
any progress on this issue?
We also stumpled up on, while we want to recreate an gitlab_user_runner when hcloud_server needs to be recreated.
resource "gitlab_user_runner" "gitlab_user_runner" {
lifecycle {
replace_triggered_by = [hcloud_server.gitlab_runner]
}
...
}
resource "hcloud_server" "gitlab_runner" {
user_data = templatefile("${path.module}/templates/cloudinit.yml",
{
token = gitlab_user_runner.gitlab_user_runner.token
...
}
)
}
results in
Error: Cycle: module.hcloud_gitlab_runners.gitlab_user_runner.gitlab_user_runner, module.hcloud_gitlab_runners.hcloud_server.gitlab_runner
