Dynamic provider configuration assignment

[WilliamABradley]

Current Terraform Version

Terraform v0.13.0-beta1

Use-cases

It would be nice to be able to create dynamic providers. The main reason for my usage would be for aws assume_role. I have Terraform to create a number of AWS Subaccounts, and then I want to configure those subaccounts in one apply, instead of breaking them up across multiple apply steps.

Currently this is done via modules, but with 0.12, I had to manually define copies of the modules for each subaccount.

As said by the 0.13 modules doc: https://github.com/hashicorp/terraform/blob/master/website/docs/configuration/modules.html.md#limitations-when-using-module-expansion

Modules using count or for_each cannot include configured provider blocks within the module. Only proxy configuration blocks are allowed.

Attempted Solutions

# Organisational Group for Crimson App.
module "org_crimson_app" {
  source          = "./modules/organisation-group"
  organisation_id = local.root_organisation_id
  group_name      = "crimson-app"

  billing_alert_emails = ["<billing account>"]

  accounts = {
    production = {}
    staging = {}
    test    = {}
  }
  zones = {
    public = {
      create  = true
      zone_id = ""
      domain  = "app.crimsoneducation.org"
    }
    internal = {
      create  = true
      zone_id = ""
      domain  = "app.crimsoneducation.io"
    }
  }

  master_role_name       = local.organisation_root_role
  account_owner_username = var.account_owner_username
  account_owner_domain   = var.account_owner_domain

  // Workaround for https://github.com/hashicorp/terraform/issues/17519
  account_configuration = {
    production = module.org_crimson_app_production_config.config
    staging    = module.org_crimson_app_staging_config.config
    test       = module.org_crimson_app_test_config.config
  }
}

// Workaround for https://github.com/hashicorp/terraform/issues/17519
module "org_crimson_app_production_config" {
  source       = "./modules/organisation-account-config"
  account_info = module.org_crimson_app.account_information.production
}

// Workaround for https://github.com/hashicorp/terraform/issues/17519
module "org_crimson_app_staging_config" {
  source       = "./modules/organisation-account-config"
  account_info = module.org_crimson_app.account_information.staging
}

// Workaround for https://github.com/hashicorp/terraform/issues/17519
module "org_crimson_app_test_config" {
  source       = "./modules/organisation-account-config"
  account_info = module.org_crimson_app.account_information.test
}

Source for ./modules/organisation-group:

# The Organisational Unit for this group.
resource "aws_organizations_organizational_unit" "unit" {
  name      = var.group_name
  parent_id = var.organisation_id

  lifecycle {
    prevent_destroy = true
  }
}

# The environment accounts.
resource "aws_organizations_account" "environments" {
  for_each  = var.accounts
  parent_id = aws_organizations_organizational_unit.unit.id

  # Use account_name if provided, otherwise [group]-[account].
  name  = local.account_full_name[each.key]
  email = "${var.account_owner_username}+aws-org-${local.account_full_name[each.key]}@${var.account_owner_domain}"

  role_name = var.master_role_name

  lifecycle {
    # There is no AWS Organizations API for reading role_name
    ignore_changes  = [role_name]
    prevent_destroy = true
  }
}

# Collect all Account Names
locals {
  account_full_name = {
    for account_name, account in var.accounts :
    account_name =>
    lookup(account, "account_name", "${var.group_name}-${account_name}")
  }
}
...

Source for ./modules/organisation-account-config:

// Workaround for https://github.com/hashicorp/terraform/issues/17519
variable "account_info" {
  type = object({
    group_name        = string
    account_name      = string
    account_full_name = string
    alias             = string
    master_role_arn   = string
    zones = map(object({
      zone_name = string
      fqdn      = string
    }))
    add_terraform_bucket   = bool
    billing_alert_amount   = number
    billing_alert_currency = string
    billing_alert_emails   = list(string)
  })
}

# Create Provisioner Assuming Child Account Role.
provider "aws" {
  region = "ap-southeast-2"

  assume_role {
    role_arn = var.account_info.master_role_arn
  }
}

resource "aws_iam_account_alias" "alias" {
  account_alias = var.account_info.alias
}

# Bucket for Terraform State.
resource "aws_s3_bucket" "terraform_state" {
  count = var.account_info.add_terraform_bucket ? 1 : 0

  bucket = "${var.account_info.account_full_name}-terraform"
  versioning {
    enabled = true
  }
}

# Fix for Issue: https://medium.com/faun/aws-eks-the-role-is-not-authorized-to-perform-ec2-describeaccountattributes-error-1c6474781b84
resource "aws_iam_service_linked_role" "elasticloadbalancing" {
  aws_service_name = "elasticloadbalancing.amazonaws.com"
}

...

Proposal

Module for_each:

# Organisational Group for Crimson App.
module "org_crimson_app" {
  source          = "./modules/organisation-group"
  organisation_id = local.root_organisation_id
  group_name      = "crimson-app"

  billing_alert_emails = ["<billing account>"]

  accounts = {
    production = {}
    staging = {}
    test    = {}
  }
  zones = {
    public = {
      create  = true
      zone_id = ""
      domain  = "app.crimsoneducation.org"
    }
    internal = {
      create  = true
      zone_id = ""
      domain  = "app.crimsoneducation.io"
    }
  }

  master_role_name       = local.organisation_root_role
  account_owner_username = var.account_owner_username
  account_owner_domain   = var.account_owner_domain

  account_configuration = module.org_crimson_app_config
}

module "org_crimson_app_config" {
  for_each = module.org_crimson_app.account_information
  source       = "./modules/organisation-account-config"
  account_info = each.value
}

Provider for_each:

This doesn't look as clean, but appeases the docs saying:

In all cases it is recommended to keep explicit provider configurations only in the root module and pass them (whether implicitly or explicitly) down to descendent modules. This avoids the provider configurations from being "lost" when descendent modules are removed from the configuration. It also allows the user of a configuration to determine which providers require credentials by inspecting only the root module.

# Organisational Group for Crimson App.
module "org_crimson_app" {
  source          = "./modules/organisation-group"
  organisation_id = local.root_organisation_id
  group_name      = "crimson-app"

  billing_alert_emails = ["<billing account>"]

  accounts = {
    production = {}
    staging = {}
    test    = {}
  }
  zones = {
    public = {
      create  = true
      zone_id = ""
      domain  = "app.crimsoneducation.org"
    }
    internal = {
      create  = true
      zone_id = ""
      domain  = "app.crimsoneducation.io"
    }
  }

  master_role_name       = local.organisation_root_role
  account_owner_username = var.account_owner_username
  account_owner_domain   = var.account_owner_domain

  account_configuration = module.org_crimson_app_config
}

provider "aws" {
  for_each = module.org_crimson_app.account_information
  alias = each.key

  assume_role {
    role_arn = each.value.master_role_arn
  }
}

module "org_crimson_app_config" {
  for_each = module.org_crimson_app.account_information
  source       = "./modules/organisation-account-config"
  providers = {
    aws = aws[each.key]
  }
  account_info = each.value
}

References

• Another user references a use case: https://discuss.hashicorp.com/t/terraform-v0-13-0-beta-program/9066/9
• https://github.com/hashicorp/terraform/issues/25320

[tenderitaf]

Hi, is this still under the radar ? is there any plan to fix this yet ? thanks

[bflad]

Just to link the issues, see also https://github.com/hashicorp/terraform/issues/19932 for multiple provider instantiation.

[tenderitaf]

can someone from Hashicorp bother himself and answer, Issue is open for a year now!!! Would be good to fix to answer those RFC in terraform before announcing new product capabilities

[extemporalgenome]

Indeed. Without a workaround feature, this is a fundamental weakness of making post-auth information like region conventionally part of the provider rather than a more normal input to the resource, and as such, terraform becomes unreasonably verbose for expressing non-trivial architectures.

[aaomidi]

This comment may help people here as well: https://github.com/hashicorp/terraform/issues/29840#issuecomment-974999903

Again, not tested for more than one, but I think it should just work.

[nfx]

Another use-case many practitioners have is for_each on modules with providers. The most recent use-case:

1. base/databricks-common is initializing databricks provider via host & token combinations, that are outputs from either base/aws or base/azure.
2. aws-classroom includes base/aws & base/databricks-common
3. azure-classroom includes base/azure & base/databricks-common

└── modules
    ├── aws-classroom
    │   └── main.tf
    ├── azure-classroom
    │   └── main.tf
    └── base
        ├── aws
        │   ├── main.tf
        │   └── provider.tf
        ├── aws-shared
        │   ├── main.tf
        │   └── provider.tf
        ├── azure
        │   └── main.tf
        ├── databricks-common
        │   ├── main.tf
        │   ├── policy.tf
        │   ├── provider.tf
        │   └── users.tf
        └── defaults
            └── main.tf

Let's simplify the desired outcome of this feature:

data "http" "classrooms" {
  url = "https://internal-ws-based-on-non-tech-users-input/classrooms.json"
}

module "classrooms" {
  for_each      = data.http.classrooms.value
  source         = "./modules/classroom"
  name           = each.value.name
}

resource "aws_s3_bucket_object" "all_classrooms" {
  bucket = "..."
  key    = "classrooms.json"
  content_base64 = base64(json_encode([for c in module.classrooms: {
    "name" = c.name,
    "url" = c.url
  }]))
}

so that we just leave the auto-approve to terraform and run it as background job in Terraform Cloud or anywhere else every 30 minutes, having no bothering to change the HCL files through GIT every time we need to add a new classroom. But this is not possible because of this bug.

some folks have solved the similar problem with code-generating the classroom equivalent modules via bash script, but this cannot really be done as a single-state apply... theoretically we can hack this around via github_repository_file, and use data.http.classrooms.value to generate & change classrooms.tf file that would have nothing but hardcoded module declarations, but it's still a hack.

[nfx]

@jbardin any updates on the roadmap for this feature?

[trallnag]

@nfx, from what I understand this feature is very difficult to implement, because it would require major changes to the way Terraform works internally. With the basic idea that providers must exist and be setup before "running".

But I agree that something like a dynamic provider would be great.

Another common use case is one Terraform state that manages both a AWS EKS Kubernetes cluster with the AWS provider and Kubernetes resources with the Kubernetes provider.

Currently there is basically no straight-forward way to do both things in the same state. The official recommendation is to split the creation of the cluster and the management of cluster internals into two states.

[nfx]

@trallnag of course the feature might take time, so i'm asking for timelines here. those workarounds are not acceptable in the long term =)

[eferreyramontes]

Any news about it? I'm facing the same issue

[nfx]

could everyone with Hashicorp contact/subscription try reaching up? probably it's the only way to prioritize this.

[bradenwright-opunai]

Yeah def annoying, I'd love to get an update on this too

[spareslant]

It is really good to have this kind of feature in terraform DSL. However this can be achieved (to some extent) if using terraform CDK. But terraform CDK cannot be used in production yet. :(

[trallnag]

However this can be achieved (to some extent) if using terraform CDK

@spareslant, could you expand on this?

[spareslant]

@trallnag, In TerraformCDK you can write terraform code in a proper programming language like python. With this, you can mask some of the code in if-then-else. I used terraform stacks in terraformCDK. First stack runs using usual default provider (credentials), but it generates config that can be used to run other stacks.

So in first run of TerraformCDK it shows and runs only initial one stack. (other stacks will not be visible till now due to conditionals in place. Moment first stack finishes, it has created a new config which can be passed onto other remaining stacks.

I tested this in OCI (Oracle Cloud Infrastructure). I ran first stack with default credentials (default provider), It created new compartment, new user and new api-keys and created policy to allow this new user to deploy rest of the stuff. Remaining stacks then used this new credentials(wrapped in new provider) to deploy the rest of the infrastructure.

It was not in one command run, I had to run terraform two times. It is not what you might be looking for, but achieved some purpose for my testing.

You can check the code here: This is what I wanted to say in code: https://github.com/spareslant/oci_multi_stack_terraform_cdk_python_v2/blob/main/main.py#L21 https://github.com/spareslant/oci_multi_stack_terraform_cdk_python_v2/blob/main/network.py#L40

And README: https://github.com/spareslant/oci_multi_stack_terraform_cdk_python_v2#readme

[trallnag]

@spareslant, ah okay, thanks for the details. So it's using several stacks / layers / states just like one would in vanilla Terraform. Just wrapped in one program / script.

[apparentlymart]

I've done a little more research into what it might take to support this today. There are some significant architectural challenges to overcome before we could support dynamically associating a provider configuration with a resource, rather than treating that as a static concern before expression evaluation.

The key challenges I've identified in my initial (failed) prototyping spike are:

• Our state model tracks provider configuration address as a per-resource value rather than a per-resource-instance value, which means that the assumption that all instances of a particular resource or data block belong to the same provider is baked into the state snapshot format. We'd need to revise the state snapshot format in a forward-incompatible way (older versions of Terraform would not be able to decode the new format) to change that.

  In principle we could make the format change only in the new situation where not all of the instances of a resource have the same provider configuration address, which would at least then mean the compatibility problem wouldn't arise until the first time a configuration creates that situation.

• The provider meta-argument in resource and data blocks currently statically declares which provider configuration to use, and because each provider configuration is statically associated with a particular provider it also implies which provider the resource belongs to. ("Provider" means e.g. hashicorp/aws, while "Provider configuration" represents a particular provider "aws" block.)

  Although we could in principle support the provider configuration varying between instances of a resource, I believe it's still necessary to statically determine the provider itself so that e.g. terraform init can determine which providers need to be installed, various parts of Terraform can know which schemas they are supposed to use for static validation, etc.

  This means that if we change the provider meta-argument to take an arbitrary expression which returns a provider configuration then we will need to find some other way to separately declare the expected provider, so that we can determine the provider statically and then the specific configuration dynamically later.

  One possible strawman language design would be to support a new syntax in provider like provider = aws(var.example), where the aws portion is a static reference to a provider local name (one of the keys in the required_providers block) and the argument is an arbitrary expression that returns an object representing a provider configuration for that provider.

• In my work so far I've been assuming that the goal would be for provider configurations to be a new kind of value in the Terraform language, with a new type kind so that each provider has its own provider configuration data type. That could then for example allow passing provider configurations in as part of a larger data structure, rather than them always having to enter a module through the providers sidecar channel:

  terraform {
    required_providers {
      # References to "aws" below refer to this provider.
      aws = {
        source = "hashicorp/aws"
      }
    }
  }
  
  variable "example" {
    type = map(object({
      instance_type = string
      provider      = providerconfig(aws)
    }))
  }
  
  resource "aws_instance" "example" {
    for_each = var.example
  
    instance_type = each.value.instance_type
  
    provider = aws(each.value.provider)
  }
  

  My investigation so far suggests that it's relatively straightforward to define a new data type representing a provider configuration, but it's unclear how to reconcile that with the existing provider-configuration-specific side-channels of either implicit provider inheritance or explicit provider passing via the providers meta-argument inside a module block. In particular, it isn't clear to me yet how it would look to have a configuration with a mixture of old-style and new-style modules, where some modules are still expecting the old mechanisms for passing providers while others want them to arrive via normal values in input variables.

• Passing providers around as normal values comes with the nice benefit that our usual expression-analysis-based dependency graph building approach can "just work" without any special cases, as long as there's a new expression-level syntax for referring to a provider block defined in the current module.

  However, Terraform's current special cases for dealing with providers statically seem likely to come into conflict with this new generalized form if we try to leave them both implemented together. There's various provider-configuration-specific logic for automatically assuming empty configuration blocks for providers that have no required configuration arguments, for inserting "proxies" to model the implicit or explicit passing of configurations from parent to child module, etc.

  When I prototyped I just deleted all of that stuff because I had the luxury of not having to be backward-compatible, but I doubt that approach will succeed in a real implementation. We'll need to define exactly how the dependency graph ought to be built when a configuration contains a mixture of both traditional static and new-style dynamic provider references.

My focus here was on the problem of dynamically assigning existing provider configurations to individual resource instances. Although it's thematically connected, from a design and implementation standpoint that's actually pretty separate from the other desire to dynamically define provider configurations (e.g. #19932), and so I've not considered that here. That desire has a largely-unrelated set of challenges that involve the same problem that makes providers declared inside child modules not work well today: a provider configuration always needs to outlive all of the resources it's managing so that it can be around to destroy them, whereas declaring them dynamically makes it quite likely that both the provider configuration and the resource would be removed systematically together. If anyone would like to investigate that set of problems, I suggest discussing that over in #19932 instead so that we can keep the two research paths distinct from one another.

[joshueis]

I've done a little more research into what it might take to support this today. There are some significant architectural challenges to overcome before we could support dynamically associating a provider configuration with a resource, rather than treating that as a static concern before expression evaluation.

The key challenges I've identified in my initial (failed) prototyping spike are:

• Our state model tracks provider configuration address as a per-resource value rather than a per-resource-instance value, which means that the assumption that all instances of a particular resource or data block belong to the same provider is baked into the state snapshot format. We'd need to revise the state snapshot format in a forward-incompatible way (older versions of Terraform would not be able to decode the new format) to change that. In principle we could make the format change only in the new situation where not all of the instances of a resource have the same provider configuration address, which would at least then mean the compatibility problem wouldn't arise until the first time a configuration creates that situation.

• The provider meta-argument in resource and data blocks currently statically declares which provider configuration to use, and because each provider configuration is statically associated with a particular provider it also implies which provider the resource belongs to. ("Provider" means e.g. hashicorp/aws, while "Provider configuration" represents a particular provider "aws" block.) Although we could in principle support the provider configuration varying between instances of a resource, I believe it's still necessary to statically determine the provider itself so that e.g. terraform init can determine which providers need to be installed, various parts of Terraform can know which schemas they are supposed to use for static validation, etc. This means that if we change the provider meta-argument to take an arbitrary expression which returns a provider configuration then we will need to find some other way to separately declare the expected provider, so that we can determine the provider statically and then the specific configuration dynamically later. One possible strawman language design would be to support a new syntax in provider like provider = aws(var.example), where the aws portion is a static reference to a provider local name (one of the keys in the required_providers block) and the argument is an arbitrary expression that returns an object representing a provider configuration for that provider.

• In my work so far I've been assuming that the goal would be for provider configurations to be a new kind of value in the Terraform language, with a new type kind so that each provider has its own provider configuration data type. That could then for example allow passing provider configurations in as part of a larger data structure, rather than them always having to enter a module through the providers sidecar channel:

  terraform {
    required_providers {
      # References to "aws" below refer to this provider.
      aws = {
        source = "hashicorp/aws"
      }
    }
  }
  
  variable "example" {
    type = map(object({
      instance_type = string
      provider      = providerconfig(aws)
    }))
  }
  
  resource "aws_instance" "example" {
    for_each = var.example
  
    instance_type = each.value.instance_type
  
    provider = aws(each.value.provider)
  }
  

  My investigation so far suggests that it's relatively straightforward to define a new data type representing a provider configuration, but it's unclear how to reconcile that with the existing provider-configuration-specific side-channels of either implicit provider inheritance or explicit provider passing via the providers meta-argument inside a module block. In particular, it isn't clear to me yet how it would look to have a configuration with a mixture of old-style and new-style modules, where some modules are still expecting the old mechanisms for passing providers while others want them to arrive via normal values in input variables.

• Passing providers around as normal values comes with the nice benefit that our usual expression-analysis-based dependency graph building approach can "just work" without any special cases, as long as there's a new expression-level syntax for referring to a provider block defined in the current module. However, Terraform's current special cases for dealing with providers statically seem likely to come into conflict with this new generalized form if we try to leave them both implemented together. There's various provider-configuration-specific logic for automatically assuming empty configuration blocks for providers that have no required configuration arguments, for inserting "proxies" to model the implicit or explicit passing of configurations from parent to child module, etc. When I prototyped I just deleted all of that stuff because I had the luxury of not having to be backward-compatible, but I doubt that approach will succeed in a real implementation. We'll need to define exactly how the dependency graph ought to be built when a configuration contains a mixture of both traditional static and new-style dynamic provider references.

My focus here was on the problem of dynamically assigning existing provider configurations to individual resource instances. Although it's thematically connected, from a design and implementation standpoint that's actually pretty separate from the other desire to dynamically define provider configurations (e.g. #19932), and so I've not considered that here. That desire has a largely-unrelated set of challenges that involve the same problem that makes providers declared inside child modules not work well today: a provider configuration always needs to outlive all of the resources it's managing so that it can be around to destroy them, whereas declaring them dynamically makes it quite likely that both the provider configuration and the resource would be removed systematically together. If anyone would like to investigate that set of problems, I suggest discussing that over in #19932 instead so that we can keep the two research paths distinct from one another.

Are we saying that in Terraform, the "provider" identified to install when running terraform init is treated as different when having a different configuration? e.i : aws region1 vs aws regions2? What would be the implications of allowing the region to be defined/overridden at the module level? What is the official suggested way for doing multi-region deployments with hashicorp/aws provider? (It's clear the workaround is one module per region, losing all dynamics in the terraform app).

As much as I love terraform, this issue pushes me to leverage CloudFormation.

[rgr-ralf-it]

One use case is Mono terraform deployment:

Multiple Regions AKS per Region Helm Provider per AKS

## ! │ The module at ... is a legacy module which contains its own local
## ! │ provider configurations, and so calls to it may not use the count, for_each, or depends_on arguments.
## ! │ 
## ! │ If you also control the module "...", consider updating this module to
## ! │ instead expect provider configurations to be passed by its caller.
## ! ╵
# provider "helm" {
#   kubernetes {
#     host                   = var.kubernetes_cluster.host
#     client_certificate     = base64decode(var.kubernetes_cluster.client_certificate)
#     client_key             = base64decode(var.kubernetes_cluster.client_key)
#     cluster_ca_certificate = base64decode(var.kubernetes_cluster.cluster_ca_certificate)
#   }
# }

[nfx]

are there any timelines for implementing this, @apparentlymart ?

[lucaspiressimao]

Same here, Dynamic EKS configuration so I need a dynamic provider to apply helm.

One use case is Mono terraform deployment:

Multiple Regions AKS per Region Helm Provider per AKS

## ! │ The module at ... is a legacy module which contains its own local
## ! │ provider configurations, and so calls to it may not use the count, for_each, or depends_on arguments.
## ! │ 
## ! │ If you also control the module "...", consider updating this module to
## ! │ instead expect provider configurations to be passed by its caller.
## ! ╵
# provider "helm" {
#   kubernetes {
#     host                   = var.kubernetes_cluster.host
#     client_certificate     = base64decode(var.kubernetes_cluster.client_certificate)
#     client_key             = base64decode(var.kubernetes_cluster.client_key)
#     cluster_ca_certificate = base64decode(var.kubernetes_cluster.cluster_ca_certificate)
#   }
# }

[adiglp]

Hello @apparentlymart , any update about this issue? we're facing it too, thanks in advance

[piroinno]

Guess this will never happen :-(

[kahawai-sre]

It may not help much, but for Azure specifically, and where the intent is to e.g. for_each through a module and pass in the target deployment scope (Subscription or Subscription and resource group) per instance, rather than duplicating code (which get's messy with hundreds of subscriptions), there is the option of using the Azure AzApi provider as I've provided an example of here: https://github.com/kahawai-sre/azapi-nsgs-demo In case it helps anyone in that context in the meantime ... That said, using AzureRM and having the capability to pass the provider dynamically would of course be a far better solution for many reasons as things stand, so I'm still holding out for that ... please :-)

[archmangler]

Seriously? Two years and a feature which would be native to most DSLs and the most basic programming language is not present? At this point I might as well consider pulumi.

[mahsoud]

we are noticing some resources allow provider configuration to overwrite right on the resource level, which seems like a convenient option:

https://registry.terraform.io/providers/TelkomIndonesia/linux/latest/docs#provider-override

[philippeboyd]

we are noticing some resources allow provider configuration to overwrite right on the resource level, which seems like a convenient option:

https://registry.terraform.io/providers/TelkomIndonesia/linux/latest/docs#provider-override

That would be a nice workaround!

[brandon-fryslie]

My focus here was on the problem of dynamically assigning existing provider configurations to individual resource instances.

This would almost completely solve the issue, at least for many people. A lot of people already generate the providers configuration one way or another and it wouldn't be much of a problem to generate this list. The crux of the problem is that you also need to generate the entire module block using the provider alias which is far less appealing.

I completely understand the concerns around removing the provider before resources are destroyed and understand the design decisions around that. I've run into the issue on older versions of Terraform and it's definitely not fun to debug (if you don't know the root cause). But giving us some way to pass a provider alias without generating a module block and all of it's configuration would at least give us many ways to work around the issue.

I'm going to try overrides.tf files and hope there isn't some restriction that prevents them from operating on provider aliases, that might be workable.

[parthibd]

Any progress?

[nitoxys]

This issue has been open for quite some time.. At least give us an provider override.