terraform
terraform copied to clipboard
hashicorp
Instantiating Multiple Providers with a loop
Current Terraform Version
Terraform v0.11.11
Use-cases
In my current situation, I am using the AWS provider so I will scope this feature request to that specific provider, although this may extend to other providers as well.
I am attempting to create resources in multiple AWS accounts. The number of the accounts ranges from 0....x and will be dynamic. I would like to be able to instantiate multiple providers which can assume a role in each account, and in turn, create resources with the associated provider without hard-coding providers for each subsequent account.
For example, something like this:
variable "accounts" {
type = "list"
default = ["123456789012", "210987654321"]
}
variable "names" {
type = "map"
default = {
"123456789012" = "foo"
"210987654321" = "bar"
}
}
provider "aws" {
count = "${length(var.accounts)}"
alias = "${lookup(var.names, element(var.accounts, count.index))}"
assume_role {
role_arn = "arn:aws:iam::${element(var.accounts, count.index)}:role/ASSUMEDROLE"
}
resource "aws_instance" "web" {
count = "${length(var.accounts)}"
provider = "aws.${lookup(var.names, element(var.accounts, count.index))}"
ami = "${data.aws_ami.ubuntu.id}"
instance_type = "t2.micro"
tags = {
Name = "HelloWorld"
}
}
With a more generic formulation, it would be great to have a count attribute in providers, not only to instantiate multiple times the same provider but also to choose between several providers: some having a zero count, some having a non-zero count.
Has anyone ever found any kind of workaround for this kind of scenario? We have multiple AWS accounts and would like to be able to deploy modules across all accounts without having to have repeatable code for each account.
@jaken551 Have you tried this with Terraform 0.12 ? Did you make any progress? I ask because I'm looking for the same functionality.
We are using terraform 0.12 and have the same issue when using the terraform-provider-aws and a few other providers. Basically this is a problem in any provider that wasnt designed to work across multiple organizations/accounts/workspaces/etc. By comparison, terraform-provider-google and terraform-provider-tfe dont really require this feature.
To summarize, the ask here is to allow provider blocks to instantiated dynamically and be dependent on computed values from other resources, such as creating a new AWS account and then using the account ID to assume role to perform actions in the new child account.
provider blocks will also need to support for_each and count to allow loops.
@hhh0505 Our current workaround is for our main terraform pipeline to create:
- 1x AWS master account user with only AssumeRole access to the child account
- 1x GitHub repository with some boilerplate code managed by our custom terraform-provider-git
- 1x Terraform Cloud workspace with the AWS user secrets and the Github repo as the source
At this point, the individual workspaces act as the pipelines to their child accounts. As for the boilerplate code, they can be updated in the main pipeline's repo and will propagate to the downstream repos whenever the main terraform runs.
@jaken551 @kuwas - I'd also like to see provider blocks support for_each. My use case being creating aws_ses_domain_identity resources in multiple regions. These regions being defined by a list variable i.e.
resource "aws_ses_domain_identity" "example" {
for_each = var.ses_regions
provider = each.value
domain = var.ses_domain
}
One workaround is, you need to maintain separate directory for each account and call the same terraform module by passing correct values to the modules / keep some default values. to run plan / apply in all directory use wrapper such as terragrunt. https://blog.gruntwork.io/terragrunt-how-to-keep-your-terraform-code-dry-and-maintainable-f61ae06959d8 https://davidbegin.github.io/terragrunt/use_cases/execute-terraform-commands-on-multiple-modules-at-once.html
I'm an enterprise user and doing a for_each over the vault provider would reduce a lot of code if you could just do:
variable "namespaces" {
type = set(string)
default = []
description = "Names to be created"
}
resource "vault_namespace" "namespace" {
for_each = var.namespaces
path = each.key
}
provider "vault" {
for_each = var.namespaces
alias = each.key
namespace = each.key
}
instead of copy pasting the same piece for each namespace.
Terraform deployment environment protection using separated Cloud Providers accounts is a great method to reduce configuration/deployment error Blast Radius. E.g. different AWS accounts can be used to make deployment environments of any kind (devel, stage, prod, sandbox), with potentially a lot of account with same "common configuration".
Without a solution like for_each in provider block lot of identical code must be replicated, raising a lot the risk of typing errors on addition of each new account/environment. Example:
- using aws profile in provider alias to assume role in accounts
- a master account manage e.g. identity account and any other single deployment/environment account (devel, prod, sandboxN, ecc.)
- creating user and permissions in identity account
- creating base common config for each account (assumable roles with policies, terraform state with S3/DynamoDB, etc.)
Based on account info like:
locals {
accounts = {
"devel" = { ... }
"prod" = { ... }
"sandbox1" = { ... }
"sandbox2" = { ... }
...
}
}
for_each in provider allow to achieve a common configuration using DRY code blocks, looping on accounts as base key for many decoupled resources:
module "example1" {
source = "..."
for_each local.account
provider {
aws = "aws.${each.key}"
}
... create assumable roles or terraform states (S3/dunamoDB) or other common resources in the same way making DRY code ...
}
+1 on this. We would like to use Terraform to deploy/manage IAM Roles in a list of AWS Accounts -- assuming a known role in each account for access. Below is an example implementation that would work for us with Terraform >v0.13 and the existing module.for_each -- just also need an equivalent provider.for_each:
## Just some data... a list(map())
locals {
aws_accounts = [
{ "aws_account_id": "123456789012", "foo_value": "foo", "bar_value": "bar" },
{ "aws_account_id": "987654321098", "foo_value": "foofoo", "bar_value": "barbar" },
]
}
## Here's the proposed magic... `provider.for_each`
provider "aws" {
for_each = local.aws_accounts
alias = each.value.aws_account_id
assume_role {
role_arn = "arn:aws:iam::${each.value.aws_account_id}:role/TerraformAccessRole"
}
}
## Modules reference the provider dynamically using `each.value.aws_account_id`
module "foo" {
source = "./foo"
for_each = local.aws_accounts
providers = {
aws = "aws.${each.value.aws_account_id}"
}
foo = each.value.foo_value
}
module "bar" {
source = "./bar"
for_each = local.aws_accounts
providers = {
aws = "aws.${each.value.aws_account_id}"
}
bar = each.value.bar_value
}
As a workaround for not having provider.for_each or another solution for this, we are considering using a script to compile the Terraform from the dataset which is not ideal.
Allowing alias as a variable in the provider config would also make for cleaner code:
variable awsEnvironments {
type = list
default = [
"paas-luse1",
"paas-lusw2",
"gi-luse1",
"gi-lusw2"
]
}
variable awsRegion {
type = map
default = {
paas-luse1 = "us-east-1"
paas-lusw2 = "us-west-2"
gi-luse1 = "us-east-1"
gi-lusw2 = "us-west-2"
}
}
variable project {
type = map
default = {
paas-lusw2 = {
localEnvironment = "paas-luse1",
localVPCSuffix = "paas",
peerEnvironment = "paas-lusw2",
peerVPCSuffix= "paas"}
}
gi-luse1 = {
localEnvironment = "paas-luse1",
localVPCSuffix = "paas",
peerEnvironment = "gi-lusw2",
peerVPCSuffix= "paas"}
}
gi-lusw2 = {
localEnvironment = "paas-luse1",
localVPCSuffix = "paas",
peerEnvironment = "gi-lusw2",
peerVPCSuffix= "paas"}
}
}
provider "aws" {
count = length(var.awsEnvironments)
profile = var.awsEnvironments[count.index]
alias = var.awsEnvironments[count.index]
region = var.awsRegion[var.awsEnvironments[count.index]]
ignore_tags {
key_prefixes = ["Core:"]
}
}
module "peering" {
source = "../tf-source/vpc/peering"
for_each = var.project
providers = {
aws.local = each.value.localEnvironment
aws.peer = each.value.peerEnvironment
}
localEnvironment = each.value.localEnvironment
peerEnvironment = each.value.peerEnvironment
localSuffix = each.value.localVPCSuffix
peerSuffix = each.value.peerVPCSuffix
}
In this case, the module "../tf-source/vpc/peering" would be expecting two AWS Providers, local and peer.
@DonBower AFAIK, alias is allowed in the providers config for modules
It is. but not as a variable....
Error: Variables not allowed
on provider.tf line 27, in provider "aws":
27: alias = var.awsEnvironments[count.index]
Variables may not be used here.
original comment updated....
@ajbouh @DonBower
Yes, you can pass aliased providers into modules, but the providers and their aliases are still statically defined. Furthermore, when using for_each with modules, you must pass in the same provider instance to every module instance within the loop.
I recall reading a comment from @apparentlymart on the reasoning around this current limitation.
Example 1: this does not work
provider "azurerm" {
for_each = toset(["one", "two"])
alias = each.value
}
module "test" {
for_each = toset(["one", "two"])
providers = { azurerm = azurerm[each.value] }
}
Example 2: this also does not work
provider "azurerm" { alias = "one" }
provider "azurerm" { alias = "two" }
module "test" {
for_each = toset(["one", "two"])
providers = { azurerm = azurerm[each.value] }
}
Example 3: this does work
provider "azurerm" { alias = "one" }
provider "azurerm" { alias = "two" }
module "test-one" {
for_each = toset(["one"])
providers = { azurerm = azurerm.one }
}
module "test-two" {
for_each = toset(["two"])
providers = { azurerm = azurerm.two }
}
I've literally tried to this exact same thing and have been unable.
module "peering" {
for_each = var.vnet_peering_config
source = "./config/network_peering"
providers = {
azurerm.custom = "azurerm.${each.value.provider_alias}"
}
resource_group_name = module.resource_group.rg.name
ne_virtual_network = azurerm_virtual_network.default
we_virtual_network = azurerm_virtual_network.default
remote_virtual_network_name = each.virtual_network_name
remote_resource_group_name = each.resource_group_name
}
I'm actually pretty upset that I cant do it, as with peering each "peer" would need to go into another subscription!
Sigh.
Any updates? Also depends from this ... For AWS provider the only way to use multi-regions for me is dublicate code many times
provider "aws" {
alias = "euwestprovider"
region = "eu-west-1"
...
}
provider "aws" {
alias = "eucentralprovider"
region = "eu-central-1"
...
}
resource "aws_instance" "ec2-eu-west"{
provider = "aws.euwestprovider"
ami = "ami-030dbca661d402413"
instance_type = "t2.nano"
}
resource "aws_instance" "ec2-eu-central" {
provider = "aws.eucentralprovider"
ami = "ami-0ebe657bc328d4e82"
instance_type = "t2.nano"
}
Any updates from the team? This thread has been opened since 2019, please give it some attention.
+1 on the topic. Can you give us some feedback? We are using multiple accounts in my company, and it will be great to be able to loop through organization accounts to create the same resources. for_each in providers block is really needed. Thanks.
In my humble but perhaps irrelevant opinion, Terraform 1.0 shouldn't be released until this feature is implemented.
Hey all 👋 - I found a ~~solution~~ workaround that works well enough to create multiple sets of S3 buckets (or any resource) across multiple regions using multiple providers and modules. It's not as clean as provider interpolation would be, but it does work.
An example using two S3 buckets in different regions:
provider "aws" {
alias = "use1"
region = "us-east-1"
}
provider "aws" {
alias = "use2"
region = "us-east-2"
}
module "s3_use1" {
source = "./s3s"
providers = {
aws.region = aws.use1
}
bucket_name = "my-bucket-in-use1"
}
module "s3_use2" {
source = "./s3s"
providers = {
aws.region = aws.use2
}
bucket_name = "my-bucket-in-use2"
}
in ./s3s:
provider "aws" {
alias = "region"
}
variable "bucket_name" {
}
resource "aws_s3_bucket" "my_s3_bucket" {
bucket = var.bucket_name
provider = aws.region
}
This solution was inspired by this Stack Overflow answer.
Big +1 to get this fixed please. Just dealing with the available regions is hard; I can't imagine having to deal with this for multiple AWS accounts.
With the azurerm module, it is not possible to create a resource group in a specific subscription unless you make the provider explicit. This limitation makes it impossible to iterate over a map or list of configuration data, when the underlying subscription is also parameterized.
provider "azurerm" {
features {}
alias = "specific_subscription"
subscription_id = "00000000-0000-0000-0000-000000000000"
}
resource "azurerm_resource_group" "foo" {
name = "foo-rg"
location = "centralus"
provider = azurerm.specific_subscription
}
Would love suggestions on how to achieve this, or why this approach is flawed according to Terraform design philosophies.
I have the same issue, I can't find a way to checkout workspaces from different resource groups with azurerm provider.
I know this sounds a bit crazy, but I solved this using M4 macros. You can use esyscmd to do whatever you want, and then the output of that will be handled as macros. So, for instance, here's a simplified main.tf.m4 that does something for every AWS region:
/**
* ******** DO NOT EDIT THIS FILE ********
*
* This file is auto-generated. If a new region has launched, please run make.
*/
dnl
dnl You can edit this file, though. The end of the file will shell out to the AWS CLI and
dnl magically build main.tf based on the current regions available.
define(`PROVIDER', `provider "aws" {
version = "= 3.42.0"
# picks up most of its configuration from the environment
alias = "some-special-identifier-prefix-$1"
region = "$1"
}
')dnl
dnl
dnl The macro which drops all of the stuff for each region.
define(`REGION', `
#
# BEGIN AUTO-GENERATED RESOURCES FOR $1
#
PROVIDER($1)
# Put macros here for the resources you want to create
#
# END AUTO-GENERATED RESOURCES FOR $1
#
')dnl
dnl
dnl This is the magical part where we get all of the regions and output macros for each
esyscmd(`aws ec2 describe-regions --region us-east-1 --query "Regions[].RegionName" | jq -r ".[] | \"R\" + \"EGION(\" + . + \")\""')dnl
And a Makefile:
# This Makefile is used to build main.tf, which sets up XXX.
# Rerun make whenever new regions are made available.
# Ensure required binaries are present
REQUIRED_BINS := aws jq m4
$(foreach bin,$(REQUIRED_BINS),\
$(if $(shell command -v $(bin) 2> /dev/null),,$(error Please install `$(bin)`)))
# If you just type make, build this
MAIN = all
# Build main.tf
all: main.tf
.PHONY: all
# Clean the built file
clean:
rm -f main.tf
.PHONY: clean
# The rule that builds the main.tf file
%.tf: %.tf.m4
m4 [email protected] > $@
You could expand on this to have e.g., a HCL or JSON file which lists out info for different accounts and then builds appropriate providers and dependent module/resource instances. The downside is this can't be dynamic, since you have to re-run make to update the Terraform file.
I dealt with it by using a shell script that calls terraform to output a map variable that contains the account structure, it then iterates over each element (account) to build a terraform environment for that environment (copy in the base code, copy in the variable structure, build a backend file so everyone has their own S3 state, etc.) and then launch a new terraform instance for every single one, capture the output and exit codes, and then dump the output from each one back to the caller and determine it's own exit code based on the exit code of all the instances it launched (if anyone exited with an error, exit with an error...). It's a waste - terraform to call terraform (almost - considered literally doing it that way) because terraform can't do a loop in it's provider block.
I'd like to add another use cases to justify this demand.
When you setup an AWS Organization and follow best practices, you often want to create a centralized network account and share the VPCs via AWS RAM. You also might want to place a private route53 zone in each account you share the VPC to (e.g. apps.integration.private, database.integration.private) and associate the VPCs with the zones. Currently you need to configure a provider for each target account and pass it to a module. Neither the account creation nor the „passing to the module“ can be dynamic so you end up with a lot of duplicated code.
When working with AWS organization accounts you might want to bootstrap each account after creating it the same run (e.g. setting IAM account alias, creating some more restrictive IAM provisioner roles than the OrganizationAccountAccessRole) - this is currently not possible because you need to retrieve account IDs and setup a provider for each account with OrganizationAccessRole.
Today I'll give this merged PR a try (https://github.com/hashicorp/terraform-provider-aws/issues/14932) - this could potentially save some of those issues.
What's the point of making a DSL when you can't loop properly? Might as well have stuck with plain YAML.
@apparentlymart @jbardin Can we have an update on this? Other issues have been closed in favor of using this one for tracking, but I cannot find a progress update in the past year on this issue.
Last I heard this was planned for Terraform 1.0 (or maybe delayed until is more accurate), which of course you know is now out. We need it to generically work with an arbitrary set of accounts.
Hi @Nuru,
Sorry, I don't have any update at the moment. While it seems like a simple request on the surface, the underlying architecture of terraform is not suited for using providers in this manner. What this essentially means is that it's going to be a very large project which must complete in resources with many other large projects. While this is definitely a desired feature, and fulfilling this use case has a high priority, I cannot say which release may be targeted for implementation.
