Allow getting Unknown provider configuration during CRUD

[braunsonm]

Module version

0.4.0

Use-cases

If the user has computed configuration provided to a provider:

resource "example_foo" "bar" {
  id = 123
}

provider "example" {
  endpoint = "https://example.test/"
  api_token = example_foo.bar.name
}

I would like to get the api_token during the CRUD operations.

Attempted Solutions

From what I understand these values should be saved in a struct on the provider plugin during the Configure call. However when unknown, you cannot save them to something meaningful in the struct.

Proposal

It would be nice if there was a way Get could be used to get provider configuration, or a type (say a pointer that would be updated) that can be saved in the provider struct which will be updated when the configuration becomes known.

[apparentlymart]

Hi @braunsonm,

I think you're describing the scenario where example_foo.bar.name isn't known yet because example_foo.bar is pending creation. I was initially a bit confused by your example because it seems like the provider configuration is referring to the result of one of its own resource types, but I'm guessing that you just happened to pick example for both and in the real case example_foo would belong to a different provider than provider "example" is configuring.

I think it might be helpful if you could add some more specifics about what you are seeing here, because the sequence of operations is a bit subtle and I'm having trouble translating my understanding from what Terraform Core sends to a provider to what the framework is exposing to the provider in your case.

I have some notes related to this which will hopefully be useful to someone working on this in the future, but I can't be sure yet how relevant they are:

• During the planning phase, it's possible and expected for Terraform Core to call the ConfigureProvider RPC function with a configuration containing unknown values. There is no way to avoid that because Terraform really don't know those values, but a provider configured in that way should only be asked to handle the following other resource-related RPCs:

  • PlanResourceChange
  • ReadResource
  • ReadDataSource
  • UpgradeResourceState

  A provider developer will need to make some design tradeoffs about how to handle such situations today. Although PlanResourceChange is allowed to answer with unknown values when something is unpredictable, that isn't allowed for UpgradeResourceState or ReadResource because the state can never include unknown values.

  I think an important missing piece in Terraform's model today is a way for a provider to respond with, in effect, "I can't answer that because my configuration isn't complete enough", and have Terraform then produce a partial plan. This is essentially the same problem that means that Terraform today can't permit unknown values for count and for_each in resource blocks, and so the best a provider can do today is return a similar error -- either during ConfigureProvider or during one of the downstream RPCs -- saying that it's not valid for one or more of the provider arguments to be unknown at that step.

  To fully resolve this (and the count and for_each limitation) will require some way for Terraform Core itself to produce a "partial plan", which explicit indicates that it has only planned a subset of the work required to converge this configuration, and that the user will need to run terraform apply a second time to make further progress towards convergence.

• During the apply phase, Terraform will defer calling ConfigureProvider until the configuration is fully known, so a provider should never encounter a partially-unknown configuration in that step. This means that the actions taken by the ApplyResourceChange RPC (which is where the create, update, and delete actions will happen) can assume a wholly-known provider configuration.

• Terraform instantiates and then closes the same provider multiple times to reset between phases. Therefore the provider process which handles ApplyResourceChange will be a different process than what previously handled PlanResourceChange, with each one getting its own call to ConfigureProvider. Therefore there is no special need to handle updating an initially-unknown configuration to a known one: Terraform Core will start fresh with a new process when it's ready to deliver a wholly-known provider configuration.

I realize I'm talking in Terraform Core terms rather than framework terms here and so this information is hard to interpret from a framework caller perspective. I'm sharing this primarily as context for framework developers who might consider this in future, rather than trying to answer how to address this with the framework as it exists today, but hopefully at least some of it is useful to understand how Terraform Core and provider plugins interact.

[braunsonm]

but I'm guessing that you just happened to pick example for both and in the real case example_foo would belong to a different provider than provider "example" is configuring.

Exactly that. Sorry that was a poorly described example.

[bflad]

Hi @braunsonm 👋 Thank you for raising this.

As @apparentlymart mentions above, there are certain situations outside the provider's control where the provider configuration values may be unknown (generally due to referencing computed attributes on other resources). What the provider does during these situations will require some intentional design choices, such as choosing between:

• Raise an error during provider configuration to prevent other provider handling. While the least flexible for practitioners, is the easiest for the provider logic since clients can always be created at the provider level and the configured clients passed to all data sources and resources as necessary.
• Save the value with a framework or custom type (e.g. types.String) that supports holding unknown values, raise a warning that other unexpected issues may occur, attempt to continue other provider handling. In practice though, this option has many downsides that would make it generally undesirable, such as potentially raising the diagnostic when the operation would ultimately succeed.
• Save the value with a framework or custom type (e.g. types.String) that supports holding unknown values in the provider configuration logic and delay handling it until the value is really being used for other provider handling. This option requires the most provider logic, but is also the most flexible for practitioners. Essentially, it allows the provider to operate "as much as it can" given the Terraform constraints mentioned above (e.g. on the PlanResourceChange RPC where resource plan modification logic would normally occur in the framework, either returning early with a less-enhanced plan, or returning an error then if not possible)

The latest version of the framework operates in the following ways in terms of passing data from the provider level to the data sources and resources:

• On the ProviderConfigure RPC, the provider Configure method is called. This response can save any Go type for use later by data source and resource logic (refer to the provider.ConfigureResponse type DataSourceData and ResourceData fields), including saving any unknown values.
• On the ApplyResourceChange, PlanResourceChange, ReadDataSource, and ReadResource RPCs, the data source or resource Configure method, if defined, is called before executing other data source or resource logic (such as CRUD methods). This can potentially be used in this situation to delay handling, but the usage is nuanced since it is not guaranteed that unknown values will be fully known.

As a last resort, data source and resource method logic can directly interact with the provider level values to configure clients, etc. when understanding Terraform's current operating model. As an example:

type ExampleProviderModel struct{
  ApiToken types.String `tfsdk:"api_token"`
  Endpoint types.String `tfsdk:"endpoint"`
}

// Other provider.Provider interface methods omitted for brevity.
type ExampleProvider struct{}

func (p ExampleProvider) Configure(ctx context.Context, req provider.ConfigureRequest, resp *provider.ConfigureResponse) {
  var data ExampleProviderModel

  resp.Diagnostics.Append(req.Config.Get(ctx, &data)...)

  if resp.Diagnostics.HasError() {
    return
  }

  // ... potentially other logic, such as errors with null/empty values, etc. ...

  // These response fields accept any Go type.
  // Using ExampleProviderModel for ease of this example.
  resp.DataSourceData = &data
  resp.ResourceData = &data
}

// Other resource.Resource interface methods omitted for brevity.
type FooResource struct{
  ProviderData *ExampleProviderModel
}

func (r *FooResource) Configure(ctx context.Context, req resource.ConfigureRequest, resp *resource.ConfigureResponse) {
  // Panic prevention
  if req.ProviderData == nil {
    return
  }

  // Unchecked type assertion for brevity.
  r.ProviderData = req.ProviderData.(*ExampleProviderModel)
}

func (r FooResource) Create(ctx context.Context, req resource.CreateRequest, resp *resource.CreateResponse) {
  if r.ProviderData == nil {
    resp.Diagnostics.AddError(/*...*/)
    return
  }

  // Example delayed client creation where it assumes values are available.
  client, err := examplesdk.NewClient(r.ProviderData.ApiToken.Value, r.ProviderData.Endpoint.Value)
}

The provider codebase can use any Go coding techniques to refactor this, such as creating a "base" resource type that encapsulates the method logic, shared functions, etc. to simplify the implementation across many methods, resources, etc. If there are further questions about how to setup the provider code in this manner, my best suggestion would be to reach out on HashiCorp Discuss under the Terraform Plugin Development section.

Beyond this sort of provider implementation, I'm not sure there's too much more we could recommend at this time with the framework and Terraform's operating model to make this a better experience. I hope the above recommendation is satisfactory for what is possible today. We would accept any proposals which follow Terraform's operating model, however it is not clear if there is anything else we can do right now without such as a proposal, short of documenting this implementation. I will reference #258 here so this issue will show up as a reminder there, but close this issue until there's a particular actionable proposal we can discuss further.

For further tracking of Terraform core's capabilities with unknown values, my best recommendation would be to check out https://github.com/hashicorp/terraform/issues/30937.

[github-actions[bot]]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues. If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.