terraform-provider-aws icon indicating copy to clipboard operation
terraform-provider-aws copied to clipboard

Provider produced inconsistent final plan / an invalid new value for .tags_all

Open chgasparoto opened this issue 3 years ago β€’ 57 comments

Community Note

  • Please vote on this issue by adding a πŸ‘ reaction to the original issue to help the community and maintainers prioritize this request
  • Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request
  • If you are interested in working on this issue or have submitted a pull request, please leave a comment

Terraform CLI and Terraform AWS Provider Version

Terraform v0.15.4 on darwin_amd64

  • provider registry.terraform.io/hashicorp/archive v2.2.0
  • provider registry.terraform.io/hashicorp/aws v3.42.0
  • provider registry.terraform.io/hashicorp/null v3.1.0

Affected Resource(s)

  • aws_vpc
  • aws_subnet

Terraform Configuration Files

Please include all Terraform configurations required to reproduce the bug. Bug reports without a functional reproduction may be closed without investigation.

# Copy-paste your Terraform configurations here - for large Terraform configs,
# please use a service like Dropbox and share a link to the ZIP file. For
# security, you can also encrypt the files using our GPG public key: https://keybase.io/hashicorp

# https://github.com/chgasparoto/youtube-cleber-gasparoto/tree/main/0002-terraform-lambda-layer

terraform {
  required_version = "0.15.4"
}

provider "aws" {
  region  = var.aws_region
  profile = var.aws_profile

  default_tags {
    tags = {
      Project   = "Lambda Layers with Terraform"
      CreatedAt = formatdate("YYYY-MM-DD", timestamp())
      ManagedBy = "Terraform"
      Owner     = "Cleber Gasparoto"
    }
  }
}
resource "aws_vpc" "example" {
  cidr_block = "10.1.0.0/16"
  tags = {
    Name = "my-vpc-resource"
  }
}
resource "aws_subnet" "example" {
  cidr_block = "10.1.1.0/24"
  vpc_id     = aws_vpc.example.id
  tags = {
    Name = "my-subnet-resource"
  }
}

Debug Output

aws_iam_role.cat_api_lambda: Modifications complete after 1s [id=cat-api-lambda-role]
aws_lambda_function.cat_api: Modifying... [id=cat_api]
aws_lambda_function.cat_api: Modifications complete after 0s [id=cat_api]
β•·
β”‚ Error: Provider produced inconsistent final plan
β”‚
β”‚ When expanding the plan for aws_vpc.example to include new values learned so far during apply, provider
β”‚ "registry.terraform.io/hashicorp/aws" produced an invalid new value for .tags_all: new element "CreatedAt" has
β”‚ appeared.
β”‚
β”‚ This is a bug in the provider, which should be reported in the provider's own issue tracker.
β•΅
β•·
β”‚ Error: Provider produced inconsistent final plan
β”‚
β”‚ When expanding the plan for aws_vpc.example to include new values learned so far during apply, provider
β”‚ "registry.terraform.io/hashicorp/aws" produced an invalid new value for .tags_all: new element "ManagedBy" has
β”‚ appeared.
β”‚
β”‚ This is a bug in the provider, which should be reported in the provider's own issue tracker.
β•΅
β•·
β”‚ Error: Provider produced inconsistent final plan
β”‚
β”‚ When expanding the plan for aws_vpc.example to include new values learned so far during apply, provider
β”‚ "registry.terraform.io/hashicorp/aws" produced an invalid new value for .tags_all: new element "Owner" has
β”‚ appeared.
β”‚
β”‚ This is a bug in the provider, which should be reported in the provider's own issue tracker.
β•΅
β•·
β”‚ Error: Provider produced inconsistent final plan
β”‚
β”‚ When expanding the plan for aws_vpc.example to include new values learned so far during apply, provider
β”‚ "registry.terraform.io/hashicorp/aws" produced an invalid new value for .tags_all: new element "Project" has
β”‚ appeared.
β”‚
β”‚ This is a bug in the provider, which should be reported in the provider's own issue tracker.

Panic Output

Expected Behavior

VPC and Network created with default tags successfully

Actual Behavior

VPC and Network are not created

Steps to Reproduce

  1. terraform apply

Important Factoids

References

  • #0000

chgasparoto avatar May 30 '21 07:05 chgasparoto

Thank you for reporting this issue @chgasparoto ! This seems related to the the apply-time computed tag, CreatedAt , unfortunately. I'm seeing the terraform plan isn't correctly calculating the tags_all attribute as it technically should either include all the possible tags (default_tags + resource tags) or suggest the map is known after apply:

Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
  + create

Terraform will perform the following actions:

  # aws_subnet.example will be created
  + resource "aws_subnet" "example" {
      + arn                             = (known after apply)
      + assign_ipv6_address_on_creation = false
      + availability_zone               = (known after apply)
      + availability_zone_id            = (known after apply)
      + cidr_block                      = "10.1.1.0/24"
      + id                              = (known after apply)
      + ipv6_cidr_block_association_id  = (known after apply)
      + map_public_ip_on_launch         = false
      + owner_id                        = (known after apply)
      + tags                            = {
          + "Name" = "my-subnet-resource"
        }
      + tags_all                        = {
          + "Name" = "my-subnet-resource"
        }
      + vpc_id                          = (known after apply)
    }

  # aws_vpc.example will be created
  + resource "aws_vpc" "example" {
      + arn                              = (known after apply)
      + assign_generated_ipv6_cidr_block = false
      + cidr_block                       = "10.1.0.0/16"
      + default_network_acl_id           = (known after apply)
      + default_route_table_id           = (known after apply)
      + default_security_group_id        = (known after apply)
      + dhcp_options_id                  = (known after apply)
      + enable_classiclink               = (known after apply)
      + enable_classiclink_dns_support   = (known after apply)
      + enable_dns_hostnames             = (known after apply)
      + enable_dns_support               = true
      + id                               = (known after apply)
      + instance_tenancy                 = "default"
      + ipv6_association_id              = (known after apply)
      + ipv6_cidr_block                  = (known after apply)
      + main_route_table_id              = (known after apply)
      + owner_id                         = (known after apply)
      + tags                             = {
          + "Name" = "my-vpc-resource"
        }
      + tags_all                         = {
          + "Name" = "my-vpc-resource"
        }
    }

anGie44 avatar Jun 01 '21 18:06 anGie44

If anyone figures out a work around in the meantime, please paste it here.

matiu2 avatar Jun 02 '21 22:06 matiu2

My work around is:

  1. Comment out the default_tags block
  2. Build
  3. Uncomment the default_tags block
  4. re-apply

matiu2 avatar Jun 02 '21 22:06 matiu2

Hi @matiu2 , thank you for following up on this issue! I don't believe there is a current workaround when using an apply-time Computed key/value pair like CreatedAt = timestamp() . I'm curious, in the workaround you've provided, were you also working with dynamic tag values in the default_tags.tags block?

Following the steps you've provided, if I'm using a timestamp value in the default_tags block, the final apply will not produce any modifications to the resource unfortunately as terraform won't detect the changes in the default_tags block even though it has been uncommented. The reason being that internally, the default_tags block is read as an empty map during run time (as it's marked as Computed just by having that one timestamp() tag for example), and merging the resource tags (in the example, Name="my-subnet-resource" etc.) on top of that empty map leaves us with no diff as it represents the original map.

anGie44 avatar Jun 03 '21 15:06 anGie44

Hello, Same problem for me while addind a tag "Name" which is not a part of default tags :

RESOURCE :

resource "aws_ami_copy" "server_encrypted" {
  name              = format("encrypted-server-ami.%s", local.timestamp_sanitized)
  description       = format("server (encrypted) created on : %s", local.timestamp_sanitized)
  source_ami_id     = data.aws_ami.latest_server.id
  source_ami_region = var.region
  encrypted         = true
  kms_key_id        = var.kms_key_id

  tags = {
    Name      = format("encrypted-server-ami.%s", local.timestamp_sanitized)
  }
}

PROVIDER :

provider "aws" {
  region = var.region
 default_tags {
  tags = {
   Owner     = var.Owner
   Stage     = var.Stage
   Project   = var.Project
   ManagedBy = var.ManagedBy
   Region    = var.region
  }
}
}

ERROR

Error: Provider produced inconsistent final plan
When expanding the plan for aws_ami_copy.server_encrypted to include
new values learned so far during apply, provider
"registry.terraform.io/hashicorp/aws" produced an invalid new value for
.tags_all: new element "Name" has appeared.
This is a bug in the provider, which should be reported in the provider's own
issue tracker.

Regards

ashraf133 avatar Jun 07 '21 09:06 ashraf133

Hi, same error here when adding an extra "Name" tag to a S3 bucket.

Used versions

  • terraform_version: 0.14.9
  • provider: registry.terraform.io/hashicorp/aws v3.39.0

Expected Behavior

S3 bucket created with default tags + customized tags successfully

Actual Behavior

S3 bucket is not created

Code snippet

provider "aws" {
  profile = var.aws_profile
  region  = var.aws_region

  default_tags {
    tags = {
      Created_by = "Terraform"
      Project    = "AWS_demo_fullstack_devops"
    }
  }
}

resource "aws_s3_bucket" "s3_bucket" {
  bucket        = var.bucket_name
  acl           = "private"
  force_destroy = true
  tags = {
    Name = var.bucket_name
  }
}

Plan output

  # module.s3_assets.aws_s3_bucket.s3_bucket will be created
  + resource "aws_s3_bucket" "s3_bucket" {
      + acceleration_status         = (known after apply)
      + acl                         = "private"
      + arn                         = (known after apply)
      + bucket                      = (known after apply)
      + bucket_domain_name          = (known after apply)
      + bucket_regional_domain_name = (known after apply)
      + force_destroy               = true
      + hosted_zone_id              = (known after apply)
      + id                          = (known after apply)
      + region                      = (known after apply)
      + request_payer               = (known after apply)
      + tags                        = (known after apply)
      + tags_all                    = {
          + "Created_by" = "Terraform"
          + "Project"    = "AWS_demo_fullstack_devops"
        }
      + website_domain              = (known after apply)
      + website_endpoint            = (known after apply)

      + versioning {
          + enabled    = (known after apply)
          + mfa_delete = (known after apply)
        }
    }

Error output (during apply)

Error: Provider produced inconsistent final plan

When expanding the plan for module.s3_assets.aws_s3_bucket.s3_bucket to
include new values learned so far during apply, provider
"registry.terraform.io/hashicorp/aws" produced an invalid new value for
.tags_all: new element "Name" has appeared.

This is a bug in the provider, which should be reported in the provider's own
issue tracker.

Regards, Mari

Marinaburkhardt avatar Jun 09 '21 12:06 Marinaburkhardt

Thank you for providing additional context @ashraf133 and @Marinaburkhardt ! In both cases seems having a resource tags set with a var is not not being picked (by tags_all, though presumably it's in the similar situation where the tag value is known_after_apply) 😞. Will add this to the investigation πŸ‘

anGie44 avatar Jun 09 '21 13:06 anGie44

I just ran into the same issue here with config from this pull request running in this workspace. Terraform v0.14.9 hashicorp/aws v3.45.0

Update: I received the same error when I removed the computed Environment value and made it hardcoded. This error is complaining about adding additional tags on individual resources like this, but yes, it's also using the random id, like Environment was in the last commit. When I removed the tags that were computed with random_id in favor of hardcoded values (i.e. this commit), my module succeed here.

anniehedgpeth avatar Jun 11 '21 20:06 anniehedgpeth

Error seems to be fixed, I retried the deployment and now it worked. But the output doesn't show up as expected:

Apply output:

# module.s3_assets.aws_s3_bucket.s3_bucket will be created
  + resource "aws_s3_bucket" "s3_bucket" {
      + acceleration_status         = (known after apply)
      + acl                         = "private"
      + arn                         = (known after apply)
      + bucket                      = "assets-eu-central-1-4dc5"
      + bucket_domain_name          = (known after apply)
      + bucket_regional_domain_name = (known after apply)
      + force_destroy               = true
      + hosted_zone_id              = (known after apply)
      + id                          = (known after apply)
      + region                      = (known after apply)
      + request_payer               = (known after apply)
      + tags                        = {
          + "Name" = "assets-eu-central-1-4dc5"
        }
      + tags_all                    = {
          + "Created_by" = "Terraform"
          + "Name"       = "assets-eu-central-1-4dc5"
          + "Project"    = "AWS_demo_fullstack_devops"
        }
      + website_domain              = (known after apply)
      + website_endpoint            = (known after apply)

      + versioning {
          + enabled    = (known after apply)
          + mfa_delete = (known after apply)
        }
    }

I think the "Name" tag shouldn't be listed under tags_all und just under tags instead, since it's just a resource tag (see the code snippet used above in my previous comment).

Hope that helps πŸ˜„

Marinaburkhardt avatar Jun 14 '21 09:06 Marinaburkhardt

@Marinaburkhardt the one resource type I don't have this problem with is S3 buckets, lots of other resource have this problem for me (RDS, Route53, Elasticsearch, cognito, and others)

sorengraungaard avatar Jun 14 '21 12:06 sorengraungaard

Same for me, how to upvote for this issue?

How to Fix: just launch again (but this solution doesn't fit for a Jenkins/Gitlab chain)

module.mq04.aws_mq_broker.mq: Creation complete after 9m54s [id=b-581xxx7d-xxxx-xxxx-xxxx-14547xx12345]

Error: Provider produced inconsistent final plan

When expanding the plan for module.rds01.aws_db_subnet_group.postgresql to
include new values learned so far during apply, provider
"registry.terraform.io/hashicorp/aws" produced an invalid new value for
.tags_all: new element "Name" has appeared.

This is a bug in the provider, which should be reported in the provider's own
issue tracker.
$ terraform version
Terraform v0.13.7
+ provider registry.terraform.io/hashicorp/aws v3.46.0

share-me avatar Jun 22 '21 09:06 share-me

Another case where this bug hurts, using the ALB module https://registry.terraform.io/modules/terraform-aws-modules/alb/aws/latest

Error: Provider produced inconsistent final plan

When expanding the plan for
module.xxxxxxxx[0].module.alb.aws_lb_listener.frontend_https[0]
to include new values learned so far during apply, provider
"registry.terraform.io/hashicorp/aws" produced an invalid new value for
.tags_all: new element "Name" has appeared.

This is a bug in the provider, which should be reported in the provider's own
issue tracker.

From plan. Note the code attempts to add a Name tag to tags causing the

  # module.xxxxxxx[0].module.alb.aws_lb_listener.frontend_https[0] will be created
  + resource "aws_lb_listener" "frontend_https" {
      + alpn_policy       = (known after apply)
      + arn               = (known after apply)
      + certificate_arn   = (known after apply)
      + id                = (known after apply)
      + load_balancer_arn = (known after apply)
      + port              = 443
      + protocol          = (known after apply)
      + ssl_policy        = (known after apply)
      + tags              = (known after apply)
      + tags_all          = {
                       (all provider level tags here)
        }

      + default_action {
          + order            = (known after apply)
          + target_group_arn = (known after apply)
          + type             = (known after apply)
        }
      + default_action {
          + order            = (known after apply)
          + target_group_arn = (known after apply)
          + type             = (known after apply)
        }
    }

My versions:

$ tf version
Terraform v0.13.5
+ provider registry.terraform.io/hashicorp/aws v3.40.0

rtoma avatar Jun 28 '21 09:06 rtoma

I ran into this with the AWS EKS module as well.

Error: Provider produced inconsistent final plan
β”‚ 
β”‚ When expanding the plan for module.eks.aws_security_group.workers[0] to include new values learned so far during apply, provider "registry.terraform.io/hashicorp/aws" produced an
β”‚ invalid new value for .tags_all["Name"]: was cty.StringVal("Shawn EKS Testing"), but now cty.StringVal("Shawn EKS Testing-GEyDNwpt-eks_worker_sg").
β”‚ 
β”‚ This is a bug in the provider, which should be reported in the provider's own issue tracker.
β•΅
β•·
β”‚ Error: Provider produced inconsistent final plan
β”‚ 
β”‚ When expanding the plan for module.eks.aws_security_group.workers[0] to include new values learned so far during apply, provider "registry.terraform.io/hashicorp/aws" produced an
β”‚ invalid new value for .tags_all: new element "kubernetes.io/cluster/Shawn EKS Testing-GEyDNwpt" has appeared.
β”‚ 
β”‚ This is a bug in the provider, which should be reported in the provider's own issue tracker.
β•΅
β•·
β”‚ Error: Provider produced inconsistent final plan
β”‚ 
β”‚ When expanding the plan for module.eks.aws_security_group.cluster[0] to include new values learned so far during apply, provider "registry.terraform.io/hashicorp/aws" produced an
β”‚ invalid new value for .tags_all["Name"]: was cty.StringVal("Shawn EKS Testing"), but now cty.StringVal("Shawn EKS Testing-GEyDNwpt-eks_cluster_sg").
β”‚ 
β”‚ This is a bug in the provider, which should be reported in the provider's own issue tracker.
β•΅
β•·
β”‚ Error: invalid value for name_prefix (must match [\w+=,.@-])
β”‚ 
β”‚   with module.eks.aws_iam_role.cluster[0],
β”‚   on .terraform/modules/eks/cluster.tf line 121, in resource "aws_iam_role" "cluster":
β”‚  121:   name_prefix           = var.cluster_name
β”‚ 
β•΅
β•·
β”‚ Error: invalid value for name_prefix (must match [\w+=,.@-])
β”‚ 
β”‚   with module.eks.aws_iam_policy.cluster_elb_sl_role_creation[0],
β”‚   on .terraform/modules/eks/cluster.tf line 167, in resource "aws_iam_policy" "cluster_elb_sl_role_creation":
β”‚  167:   name_prefix = "${var.cluster_name}-elb-sl-role-creation"
β”‚ 

For my own instance of this, I was able to resolve it by removing the space in the tag.Name value I was using and replace with - instead. Ideally, it would have been good to see this warning in the plan. While resolved, I am sharing case this applies to any other use cases.

My Version:

Terraform v1.0.1

tuxtek avatar Jul 06 '21 18:07 tuxtek

declare the default tags as local variable locals { common_tags = { createdby = "me" createdon = formatdate("MM/DD/YYYY", timestamp()) purpose = "test" environment = "dev" buildby = "automation" } } and in each resource where u must be adding the name append this also tags = { for k, v in merge({ "Name" = "test-vpc-1"},local.common_tags) : k => v }

sneha43sweety avatar Jul 09 '21 11:07 sneha43sweety

I ran into this with the rds cluster resource. Terraform v1.0.2 and aws provider v3.50.0:

provider "aws" {
  region              = var.aws_region
  allowed_account_ids = [var.aws_account_id]

  default_tags {
    tags = {
      Terraform = "true"
    }
  }
}

resource "aws_rds_cluster" "clone" {
  ...

  tags = {
    Name    = "clone"
    Created = timestamp()
  }
}
Error: Provider produced inconsistent final plan

When expanding the plan for
module.db_clone.aws_rds_cluster.clone to include new 
values learned so far during apply, provider
"registry.terraform.io/hashicorp/aws" produced an invalid new value for
.tags_all: new element "Name" has appeared.

This is a bug in the provider, which should be reported in the provider's
own issue tracker.

jgeurts avatar Jul 21 '21 02:07 jgeurts

I ran into this with the rds cluster resource. Terraform v1.0.2 and aws provider v3.50.0:

provider "aws" {
  region              = var.aws_region
  allowed_account_ids = [var.aws_account_id]

  default_tags {
    tags = {
      Terraform = "true"
    }
  }
}

resource "aws_rds_cluster" "clone" {
  ...

  tags = {
    Name    = "clone"
    Created = timestamp()
  }
}
Error: Provider produced inconsistent final plan

When expanding the plan for
module.db_clone.aws_rds_cluster.clone to include new 
values learned so far during apply, provider
"registry.terraform.io/hashicorp/aws" produced an invalid new value for
.tags_all: new element "Name" has appeared.

This is a bug in the provider, which should be reported in the provider's
own issue tracker.

Please remove the default tags and put that tag in local variable. With each resource merge the local variable along with resource tag. Its one of the workaround

sneha43sweety avatar Jul 21 '21 04:07 sneha43sweety

We are upgrading to Terraform v0.12.31 and AWS provider v3.38.0

Our current terraform configuration looks like this -

provider aws { version = β€œ3.38.0” }

locals { common_tags = { Source = var.source_tag Team = var.owner_tag Lifecycle = var.lifecycle_tag Project = var.project_tag EnvironmentType = var.environment_type_tag Environment = var.environment_id Contact = var.contact_tag } }

resource aws_dynamodb_table xyz { tags = local.common_tags }

Code output -

aws_dynamodb_table.xyz will be updated in-place resource "aws_dynamodb_table" "xyz" { tags= {} (known after apply) tags_all = {}

When we run the terraform files, getting below error -

Provider produced inconsistent final plan When expanding the plan for aws_dynamodb_table.xyz to include new values learned so far during apply, provider β€œregistry.terraform.io/-/aws” produced an invalid new value for .tags_all: new element β€œContact” has appeared. This is a bug in the provider, which should be reported in the provider’s own issue tracker.

The same configuration works when we manually delete the existing Dynamo DB table and re-run the files. But we want to avoid getting this error in first place. What changes do we need to make in our files to avoid this error ?

Code output (when tables are deleted manually) -

aws_dynamodb_table.xyz will be created resource "aws_dynamodb_table" "xyz" { tags = (known after apply) tags_all = (known after apply)

sarangk39 avatar Jul 21 '21 06:07 sarangk39

I ran into the same error with the aws_security_group resource in the terraform-aws-modules/security-group/aws module: https://github.com/terraform-aws-modules/terraform-aws-security-group/blob/master/main.tf#L11-L25

β”‚ Error: Provider produced inconsistent final plan
β”‚ 
β”‚ When expanding the plan for module.development_public_alb_sg.aws_security_group.this_name_prefix[0] 
to include new values learned so far during apply, provider "registry.terraform.io/hashicorp/aws" produced an
β”‚ invalid new value for .tags_all: new element "Name" has appeared.
β”‚ 
β”‚ This is a bug in the provider, which should be reported in the provider's own issue tracker.
❯ terraform --version
Terraform v1.0.1
on darwin_amd64
+ provider registry.terraform.io/hashicorp/aws v3.44.0
+ provider registry.terraform.io/hashicorp/random v3.1.0

I was able to work around the error by re-running terraform apply.

inhumantsar avatar Aug 05 '21 21:08 inhumantsar

Sam issue for me

cmd-werner-diers avatar Aug 10 '21 07:08 cmd-werner-diers

For those who face this issue with a default tag value set to timestamp().

To avoid this problem I use a variable that contains the timestamp which is set just prior to the apply.

export TF_VAR_TIMESTAMP="$( date --rfc-3339=seconds )"

variable "TIMESTAMP" { default = "1900-01-01 00:00:00+02:00" }

provider "aws" {
  region = var.AWS_REGION
  default_tags {
    tags = {
      "tf:apply" = var.TIMESTAMP  
    }
  }
}

This works with Terraform v0.13.7

kihom-77 avatar Aug 11 '21 15:08 kihom-77

Having this same issue with TF 1.0,9 when applying tags to aws_organizations_account -- only way around it is to comment out the default tags, apply, uncomment default tags, apply.

StackRef avatar Oct 18 '21 15:10 StackRef

We have also hit this issue in a slightly different form whereas a depends_on on the resource will cause a diff in tags vs tags_all at plan time even though all of the tags are statically defined. Works completely fine when dependencies are automatically determined; kinda weird.

dmccaffery avatar Oct 21 '21 11:10 dmccaffery

still confirmed with Terraform 1.0.10 and aws provider 3.64.x..

if I have a dynamic (in my case a random id) as part of default_tags set in the provider configs, my apply run is very inconsistent. its the same 'new element has appeared" style messages as above.

JamesMcOrmond avatar Nov 05 '21 20:11 JamesMcOrmond

Hi @matiu2 , thank you for following up on this issue! I don't believe there is a current workaround when using an apply-time Computed key/value pair like CreatedAt = timestamp() . I'm curious, in the workaround you've provided, were you also working with dynamic tag values in the default_tags.tags block?

Following the steps you've provided, if I'm using a timestamp value in the default_tags block, the final apply will not produce any modifications to the resource unfortunately as terraform won't detect the changes in the default_tags block even though it has been uncommented. The reason being that internally, the default_tags block is read as an empty map during run time (as it's marked as Computed just by having that one timestamp() tag for example), and merging the resource tags (in the example, Name="my-subnet-resource" etc.) on top of that empty map leaves us with no diff as it represents the original map.

My case is similar to this, but instead of a computed value using a function like timestamp(), I'm passing a resource ID from another resource in one of the tags. Originally did this as a method of ordering an implicit dependency, but then I kinda liked having the information about the relationship in the tag. But looking like I'll need to just use depends_on instead.

lorengordon avatar Nov 05 '21 21:11 lorengordon

TF: v1.0.11 AWS: 3.66.0

I am having this issue with EBS volumes. Default_tags set at the parent TF provider which is picked up by the TF module and the ebs_vol resource in the module adds the missing tags:

provider "aws" {
  region = "eu-west-1"
  default_tags {
    tags = module.tags.tags
  }
}

in elasticsearch TF module:

resource "aws_ebs_volume" "data-nodes-vols" {
  tags = merge(var.default_tags,
    tomap({
      "Name"       = "${var.cluster_name}_es_data_vol_${count.index % local.data_node_counts}",
      "NodeId"     = count.index % local.data_node_counts,
      "AsgEbsUuid" = random_uuid.asg_ebs_instance_uuids[count.index % local.data_node_counts].result
      "Device"     = lookup(element(var.es_data_block_device, floor(count.index / local.data_node_counts)), "device_name")
    })
  )

Results in:

Error: Provider produced inconsistent final plan

When expanding the plan for module.elasticsearch.aws_ebs_volume.data-nodes-vols[0] to include new values learned so far during apply, provider "registry.terraform.io/hashicorp/aws" produced an invalid new value for .tags_all: new element "AsgEbsUuid" has appeared.

This is a bug in the provider, which should be reported in the provider's own issue tracker.

For every additional tag added (Name, NodeId, AsgEbsUuid, Device)

Weirdly, this is not happening for every parent terraform that is using this elasticsearch TF module even though i cannot find any difference in the tags passed.

Edit: This is what i get on the plan:

      + tags              = {
          + "AsgEbsUuid" = "9a644e01-ecb1-6de7-d657-cb44dc735c1f"
          + "Device"     = "/dev/sdf"
          + "Name"       = "intelligence_es_data_vol_0"
          + "NodeId"     = "0"
        }
      + tags_all          = {
          + "AsgEbsUuid"        = "9a644e01-ecb1-6de7-d657-cb44dc735c1f"
          + "Client"            = "Intelligence"
          + "ClusterName"       = "Intelligence"
          + "DeploymentState"   = "opendistro"
          + "DeploymentVersion" = "21.11.2"
          + "Device"            = "/dev/sdf"
          + "Environment"       = "Development"
          + "NodeId"            = "0"
          + "OffHour"           = "N/A"
          + "OnHour"            = "N/A"
          + "Role"              = "Intelligence ES v7"
          + "ServiceName"       = "Elasticsearch"
        }

Extra edit: I seem to have this working. I inherited this branch from a colleague and i run pre-commit terraform-fmt (running tf v1.0.11 locally) which updated a few things, namely map -> map(any) etc in variables, among other changes, but i also removed an erroneous space from one of the tags:

 -   role_tag         = " Intelligence ES v7"
 +   role_tag         = "Intelligence ES v7"

The apply seemed to work after this with no problems but not sure which fixed the problem.

thenom avatar Nov 20 '21 17:11 thenom

provider "aws" { region = "eu-west-1" }

declare the default tags as local variable locals { common_tags = { tags = module.tags.tags } } resource "aws_ebs_volume" "data-nodes-vols" { tags = merge(local.common_tags, tomap({ "Name" = "${var.cluster_name}es_data_vol${count.index % local.data_node_counts}", "NodeId" = count.index % local.data_node_counts, "AsgEbsUuid" = random_uuid.asg_ebs_instance_uuids[count.index % local.data_node_counts].result "Device" = lookup(element(var.es_data_block_device, floor(count.index / local.data_node_counts)), "device_name") }) )

try this way

ssukumar-conga avatar Nov 22 '21 04:11 ssukumar-conga

Thanks @ssukumar-conga but this is the way we had it originally so the move to defaut_tags is what wanted so we didn't have to tag every resource individually. Mainly supplying the info to help and get help with this reported issue.

thenom avatar Nov 22 '21 09:11 thenom

Just a note on my extra edit on my last post. Its seems this was a red herring as the first deploy has this issue but the 2nd attempt works fine. This is replicable. related link: https://discuss.hashicorp.com/t/using-default-tags/24376

thenom avatar Nov 23 '21 10:11 thenom

This is not a red herring. Hitting the same issue when using

provider "aws" {
  alias  = "us-east-1"
  region = "us-east-1"
  default_tags {
    tags = local.default_tags_us_east_1
  }
}

and

resource "aws_ebs_volume" "vol_ebs_useast1_az" {
  provider = aws.us-east-1
  availability_zone = "us-east-1a"
  size              = 10
  tags = {
    somekey = "somevalue"
  }
}

Once the following is commented out:

  tags = {
    somekey = "somevalue"
  }

I stop getting:

β•·
β”‚ Error: Provider produced inconsistent final plan
β”‚
β”‚ When expanding the plan for aws_ebs_volume.vol_ebs_useast1_az to include new values learned so far during apply, provider "registry.terraform.io/hashicorp/aws" produced an
β”‚ invalid new value for .tags_all: new element "app" has appeared.
β”‚
β”‚ This is a bug in the provider, which should be reported in the provider's own issue tracker.

lkolchin avatar Nov 29 '21 03:11 lkolchin

Found the problem. It bombs out when one of your tags is equal to timestamp() (or formatdate("....", timestamp()). But it works fine with other dynamic values (like the ones derived from data "aws_caller_identity" "current" {} for example). Seems to be a bug :(

lkolchin avatar Nov 29 '21 05:11 lkolchin