terraform
terraform copied to clipboard
hashicorp
Ability to pass providers to modules in for_each
Use-cases
I'd like to be able to provision the same set of resources in multiple regions a for_each on a module. However, looping over providers (which are tied to regions) is currently not supported.
We deploy most of our infra in 2 regions in an active-passive configuration. So being able to instantiate both regions using the same module block would be a huuuge win. It's also our primary use case for for_each on modules being implemented in https://github.com/hashicorp/terraform/issues/17519.
Proposal
Proposed syntax from @jakebiesinger-onduo
provider "google" {
alias = "goog-us-east1"
region = "us-east1"
}
provider "google" {
alias = "goog-us-west1"
region = "us-west1"
}
locals {
regions = toset(['us-east1', 'us-west1'])
providers = {
us-east1 = google.goog-us-east1
us-west1 = google.goog-us-west1
}
}
module "vpc" {
for_each = local.regions
providers = {
google = local.providers[each.key]
}
...
}
Another option would be to de-couple the region from providers, and allow the region to be passed in to individual resources that are region aware. As far as I know, both AWS and GCP credentials at least are global.
References
- count and for_each for modules https://github.com/hashicorp/terraform/issues/17519
- looping over providers https://github.com/terraform-providers/terraform-provider-bigip/issues/247
- Multiple regions with same Provider https://github.com/hashicorp/terraform/issues/451
- Reddit: https://www.reddit.com/r/Terraform/comments/cwp0d4/terraform_multi_region_question_can_i_just_use/
Hello! :robot:
This issue seems to be covering the same problem or request as #9448, so we're going to close it just to consolidate the discussion over there. Thanks!
Yeah, this is not the same as #9448 at all. @pselle @apparentlymart could you help with reopening this issue?
Hey there @mightyguava & @jspiro,
I'm going to re-open the issue as I agree that the concerns are not the same.
I did rename it for clarity; to distinguish this request from instantiating providers with for_each.
@mightyguava
I ran into the same abstraction issue with the azurerm provider. My goal was to automate multiple azure subscriptions and keep the code DRY as possible. Since I have to use Service Principals for auth with the azurerm provider, each subscription requires a separate provider declaration. I have ended up using terragrunt's generate function (https://terragrunt.gruntwork.io/docs/reference/config-blocks-and-attributes/#generate)
.
├── dev
│ └── terragrunt.hcl
├── modules
│ └── my_module
│ └── main.tf
├── prod
│ └── terragrunt.hcl
├── stage
│ └── terragrunt.hcl
└── variables.tf
./dev/terragrunt.hcl:
terraform {
source = "${get_parent_terragrunt_dir()}/../"
}
# will generate content to ./providers.tf
generate "providers" {
path = "providers.tf"
if_exists = "overwrite"
contents = <<EOF
provider "azurerm" {
# my main provider
version = "~> 2.6"
subscription_id = "11111111-2222-3333-4444-555555555555"
client_id = var.client_id
client_secret = var.client_secret
tenant_id = var.tenant_id
features {}
}
provider "azurerm" {
alias = "my_alias_provider"
version = "~> 2.6"
subscription_id = "66666666-7777-8888-9999-000000000000"
client_id = var.client_id
client_secret = var.client_secret
tenant_id = var.tenant_id
features {}
}
EOF
}
# will generate content to ./main.tf
generate "main" {
path = "main.tf"
if_exists = "overwrite"
contents = <<EOF
module "my_main_provider" {
source = "./modules/my_module"
}
module "my_alias_provider" {
source = "./modules/my_module"
providers = {
azurerm = azurerm.my_alias_provider
}
}
EOF
}
./variables.tf:
# passing these from cli or exporting to TF_VAR
# Note that both of my subscriptions use the same SP for auth and
# both in the same tenant, so the difference is only the subscription_id
variable "client_id" {}
variable "client_secret" {}
variable "tenant_id" {}
The ./modules/my_module/main.tf contains the desired code without any provider block declaration (passing down provider declaration from root to child module)
Maybe this is not fully covering your scenario, but it provides some flexibility over different environment settings
First of all, thanks for the great work adding iteration and depends_on for modules - both are going to be really useful and I wished for them so many times back during 0.11 days when we were building the majority of our config.
In addition to each.key, I'd expect to be able to freely use maps with for_each and have each.<property> be a provider. This would require the ability to assign a provider "instance" to a local or list/map members. For example:
locals {
modules_vars = {
instance_1 = {
var1 = ...
var2 = ...
provider = aws.euw1 // ERROR
}
instance_2 = {
var1 = ...
var2 = ...
provider = aws.cnnw1 // ERROR
}
}
}
module "something" {
source = "./module_something"
for_each = local.modules_vars
providers = { aws = each.provider } // ERROR
var1 = each.var1
var2 = each.var2
}
This was the first thing I tried to do when I learned that 0.13 has for_each for modules, which brought me to #17519 and eventually - here. Since we maintain infrastructure in multiple AWS regions and availability zones around the world, most of the modules in our configuration require passing a provider along with at least a few other variables.
While not strictly the same as #9448 I think they might be solved together.
First, like @vivanov-dp said, thanks for adding the for_each support for modules. I had been expecting it for a long time.
However I had not realized that provider configuration in modules was deprecated.
Here is my use case:
- I use terraform to manage the list of AWS accounts I have in my organization
- When I create a new account (from a variable list), I then want to provision it with a few common standard resources.
What I have now is that all those standard resources are in a module. I instanciate the module once per sub account and I pass the IAM role to the module. The module then opens a provider connection to the right account and the right role (different for each module instance).
This still works in 0.13. However, when I tried to migrate to "for_each" to instanciated all the modules for all the sub-account in a single module block, I hit the issue that providers inside modules are not supported anymore.
And since I can't construct a dynamic list of providers, I think I'm stuck.
Am I right to think there is currently no workaround for my use case?
Should I then split account creation and account "basic provisionning" in 2 different terraform projects?
thanks
I personally think that inline provider declaration, which honors the module for_each or count is the cleanest solution:
module "some_module" {
source = "./some-module"
for_each = local.modules_elements
provider "provider1" {
...
}
provider "provider2" {
...
}
var1 = each.var1
var2 = each.var2
}
Ideally, this would support dynamic for providers as well.
Another option is to add for_each for provider as well along these lines:
provider "aws" {
for_each = var.regions
region = each.value
}
resource "some_type" "some_id" {
provider = aws["us-east-1"]
}
@nikolay
That can do the job, but why creating new providers for each module invocation ?
Even if it is "for free" in terms of performance, which I don't really know, there are a bunch of properties to configure the provider and this approach would require to put them all into local.modules_elements and list them all in each provider declaration in each module invocation.
You can't really declare an AWS provider just by setting the region. It requires a profile, or access_key&secret_key too and it is very likely that the assume_role would also be set. It has 15+ properties and many of them become useful as the architecture grows.
@vivanov-dp This was pseudocode just to illustrate my point, which was that the logic of how the provider should be initialized could be encapsulated in the module. I can't think of a situation where the instantiation of a provider would be an expensive operation. Also, providers with assume_role have session information, which may not make sense to be reused across different modules, but will happen due to natural laziness if we have to create too many aliases.
@nikolay What I understand is that you propose to have this:
locals {
modules_vars = {
instance_1 = {
var1 = ...
var2 = ...
region = ...
profile = ...
role_arn = ...
}
instance_2 = {
var1 = ...
var2 = ...
region = ...
profile = ...
role_arn = ...
}
}
}
module "some_module" {
source = "./some-module"
for_each = local.modules_vars
provider "aws" {
region = each.region
profile = each.profile
assume_role {
role_arn = each.role_arn
}
}
var1 = each.var1
var2 = each.var2
}
instead of:
provider "aws" {
alias = "euw1"
region = "eu-west-1"
profile = var.aws_west_profile
assume_role {
role_arn = "arn:${var.aws_partition}:iam::${var.aws_account_id}:role/TerraformRole"
}
}
provider "aws" {
alias = "cnnw1"
region = "cn-northwest-1"
profile = var.aws_cn_profile
assume_role {
role_arn = "arn:${var.aws_cn_partition}:iam::${var.aws_cn_account_id}:role/TerraformRole"
}
}
locals {
modules_vars = {
instance_1 = {
var1 = ...
var2 = ...
provider = aws.euw1
}
instance_2 = {
var1 = ...
var2 = ...
provider = aws.cnnw1
}
}
}
module "something" {
source = "./module_something"
for_each = local.modules_vars
providers = { aws = each.provider }
var1 = each.var1
var2 = each.var2
}
But then I have one set of providers for everything else and one set of the same properties just for the modules. So which one is the source of truth ? Unless I declare my providers by using the same sets of properties - so I have to create a new abstraction - the set of providers properties and use that in all places.
As I said - it can do the job, I think it looks nice and is not a bad idea, but IMO it involves more changes to the existing configuration than if we could just use the already defined providers - which in my case are in an external file, often 1 or 2 directories up the hierarchy and propagated down via a script that generates main.tf & variables.tf.
A use case for me would be to configure a dynamic provider based on output from a module using for_each
such as creating multiple kubernetes clusters (foo) and optionally applying resources (bar)
module "foo" {
source = "./foo"
for_each = var.foo_things
var1 = each.key
var2 = each.values.something
}
module "bar" {
source = "./bar"
for_each = { for k, v in var.bar_things : k => v if v.add_bar_to_foo == true }
provider "some_provider" {
config1 = module.foo[each.values.foo_thing].output1
config2 = module.foo[each.values.foo_thing].output2
config3 = module.foo[each.values.foo_thing].output3
}
var1 = each.key
var2 = each.values.something
}
@vivanov-dp The ideal approach is to have identical code and only data, which varies between environment and clusters within the environment. Right now, almost everything has for_each/count except providers.
@jon-walton Your example is identical to mine, but I illustrated if the module needs more than a single provider.
@jon-walton Your example is identical to mine, but I illustrated if the module needs more than a single provider.
My example illustrates the provider config being supplied to a module is set by the output of another module which also uses for_each
@nikolay Sure, having for_each for providers sounds logical and natural and I fully support it, I believe it deserves its own feature request
@jon-walton Fair enough, we need dynamic providers - one way or another. Right now providers and outputs are the only two static resources in Terraform.
Hi all! Thanks for the interesting discussion here.
It feels to me that both this issue and #9448 are covering the same underlying use-case, which I would describe as: the ability to dynamically declare and use zero or more provider configurations based on data determined at runtime.
These various proposals all have in common a single underlying design constraint: unlike most other concepts in Terraform, provider configurations must be available for operations on resources that belong to them, which includes planning, updating, and eventually destroying. This means that a provider configuration must be available at the same time a new resource is added to the configuration, must have a stable name that can be tracked between runs in the Terraform state, and they must continue to be available until every resource instance belonging to them has been destroyed and/or removed from the state.
It is due to that design constraint that provider configurations remain separated from all other concepts in the restrictions placed on them in the configuration. Design work so far seems to suggest that there are some paths forward to making provider configuration associations (that is, the association of resources to provider configurations) more dynamic, but the requirement that each provider configuration be defined by a static provider block in the root module seems necessary to ensure that the provider block can remain in the configuration long enough to destroy existing resource instances associated with it, which happens after they are removed from the configuration.
One design we've considered (though this is not necessarily the final design we'd move forward with) is to make provider configurations a special kind of value in the language, which can be passed by reference through expressions in a similar sense that other values can. For example:
variable "networks" {
type = map(
object({
cidr_block = string
aws_provider = providerconfig(aws)
})
)
}
resource "aws_vpc" "example" {
for_each = var.networks
provider = each.value.aws_provider
cidr_block = each.value.cidr_block
}
The aws_provider attribute here is showing a hypothetical syntax for declaring that an attribute requires a configuration for the aws provider, with that reference then usable in provider arguments in resource and data blocks where static references would be required today. That syntax is intended to replace the current "proxy provider configuration" special-case syntax, by allowing provider configurations to pass through variables instead. However, this design does have the disadvantage of requiring explicit provider configuration passing, whereas today child modules can potentially inherit non-aliased provider configurations automatically in simple cases.
However, the calling module would still be required to declare the provider configurations statically with provider blocks, perhaps like this:
provider "aws" {
alias = "usw2"
region = "us-west-2"
}
provider "aws" {
alias = "use2"
region = "us-east-2"
}
module "example" {
source = "./modules/example"
networks = {
usw2 = {
cidr_block = "10.1.0.0/16"
aws_provider = provider.aws.usw2
}
use2 = {
cidr_block = "10.2.0.0/16"
aws_provider = provider.aws.use2
}
}
}
Notice that the two provider configurations must still have separate static identities, which can be saved in the state to remember which resource belongs to which. But this new capability of sending dynamic references for provider configurations still allows writing a shared module that can be generic in the number of provider configurations it works with; only the root module is required to retain a static set of provider configurations.
There is also some possibility here of allowing count and for_each in provider blocks to permit provider addresses like provider.aws["use2"] (where use2 is the each.key), but this is more problematic because it creates another opportunity to "trap" yourself in an invalid situation: if you use the same value in for_each for both a resource configuration and its associated provider configuration, removing an item from that map would cause the resource instance and the provider configuration to be removed at the same time, which violates the constraint that the provider configuration must live long enough to destroy the instance recorded in the state. Given how common it is to get into that trap with provider blocks inside child modules today (which is why we've been recommending against that since Terraform 0.11), we're reluctant to introduce another feature that has a similar trap. For that reason, I predict that for_each and count for provider configurations (as proposed in #9448) won't make it through a more detailed design pass for this family of features.
I've shared the above mainly to just show some initial design work that happened for this family of features. However, I do have to be honest and share some unfortunate news: the focus of our work is now shifting towards stabilizing Terraform's current featureset (with minor modifications where necessary) in preparation for a Terraform 1.0, and a mechanism like the one I described above would be too disruptive to Terraform's internal design to arrive before that point.
The practical upshot of this is that further work on this feature couldn't begin until at least after Terraform 1.0 is released. Being realistic about what other work likely confronts us even after the 1.0 release, I'm going to hazard a guess that it will be at least a year before we'd be able to begin detailed design and implementation work for features in this family.
I understand that this is not happy news: I want this feature at least as much as you all do, but with finite resources and conflicting priorities we must unfortunately make some hard tradeoffs. I strongly believe that there is a technical design to address the use-cases discussed here, but I also want to be candid with you all about the timeline so that you can set your expectations accordingly.
@apparentlymart Having providerconfig(aws) is a bit limiting as you can't pass the dynamic index from a TFC variable or terraform.tfvars.json file. The easiest and probably quickest to implement it just to allow something like provider.aws[var.provider_alias] - you still have static providers, just dynamic references to them.
I refer to the blog announcement for TF 0.13 with this block of code:
variable "project_id" {
type = string
}
variable "regions" {
type = map(object({
region = string
network = string
subnetwork = string
ip_range_pods = string
ip_range_services = string
}))
}
module "kubernetes_cluster" {
source = "terraform-google-modules/kubernetes-engine/google"
for_each = var.regions
project_id = var.project_id
name = each.key
region = each.value.region
network = each.value.network
subnetwork = each.value.subnetwork
ip_range_pods = each.value.ip_range_pods
ip_range_services = each.value.ip_range_services
}
This implies we can do for_each over a region...
@cregkly This example is with Google cloud - the provider instance is not constrained within the region with Google, so you don't need multiple provider instances to use different regions - resources have 'region' properties themselves
@cregkly This example is with Google cloud - the provider instance is not constrained within the region with Google, so you don't need multiple provider instances to use different regions - resources have 'region' properties themselves
And I quote the original post:
I'd like to be able to provision the same set of resources in multiple regions a for_each on a module. However, looping over providers (which are tied to regions) is currently not supported.
And then they gave a google cloud example...
@cregkly Yes, but we're talking about providers here, not modules.
Ability to pass providers to modules in for_each
@apparentlymart Can you guys put a better example up on the blog post about TF 13 then? It uses the example of for_each over regions with google cloud. Naturally it is the first thing I wanted to try out with in AWS, then it turns out it can't be done.
At the very least link to the something that explains why this works with Google Cloud and not others like AWS.
I appreciate you insights and transparency on the development to version 1.
I think the person who wrote that blog post was motivated to find an existing registry module with a relatively simple interface so that the module's own complexity wouldn't overwhelm the article with module-specific complexity. The point of it is just to be a generic (but working) example of what the syntax looks like for marketing purposes, not to be documentation. In general I'd suggest thinking of HashiCorp blog posts as being more "notification that the thing exists" than "guide/example on how to use the thing".
The HashiCorp education team wrote a long-form guide on for_each which discusses these things in more detail.
I updated the blog post a while ago, but I am waiting for another team to push the changes live. It looks like our blogging platform was updated between the release of 0.13 and today.
The replaced example is designed to signal the for_each feature without misleading users to believing they can copy paste code and use it as is.
I apologize for the delay in getting this remediated.
Update: I went back to check and the blog post has been updated.
Our use-case is the multi account setup where we deploy stuff like IAM roles for monitoring permission to all accounts and do have a centrally Grafana that does collect these data.
Looks like currently there is no way to handle this without an addon like terragrunt?
The following would be an example on how this could be handled if you require the provider to stay on root level. But this also requires to have the for_each available on providers.
# A list of AWS accounts that also might come from an external source (json / yaml)
locals {
accounts = {
"4711" = { something = "foo" }
"0815" = { something = "bar" }
...
}
}
# Generate a AWS STS token via Vault, each role is mapped to a different AWS account
data "vault_aws_access_credentials" "sts" {
for_each = local.accounts
role = each.key
backend = "aws"
type = "sts"
}
# Create a provider for each account by pasting in the STS tokens
provider "aws" {
for_each = local.accounts # MISING FEATURE
region = "eu-central-1"
alias = each.key
access_key = data.vault_aws_access_credentials.sts[each.key].access_key
secret_key = data.vault_aws_access_credentials.sts[each.key].secret_key
token = data.vault_aws_access_credentials.sts[each.key].security_token
}
# Paste the provider down to the the account module
module "account" {
for_each = local.accounts
something = each.value.something
providers "aws" {
aws = aws[each.key]
}
}
We have the same use case as https://github.com/hashicorp/terraform/issues/24476#issuecomment-709070083 for AWS account bootstrap (has to iterate by each provider)
module "account" {
for_each = local.accounts
something = each.value.something
providers "aws" {
aws = aws[each.key]
}
}
same problem here - it's quite a limitation and it makes for_each next to useless ..
