[Bug]: "Provider produced inconsistent final plan" when updating launch_template used in aws_batch_compute_environment

[LucaIcaro]

Terraform Core Version

1.6.6

AWS Provider Version

5.68

Affected Resource(s)

aws_batch_compute_environment, aws_launch_template

Expected Behavior

terraform apply should not return errors

Actual Behavior

terraform apply updates the LT but returns errors like this:

╷
│ Error: Provider produced inconsistent final plan
│ 
│ When expanding the plan for aws_batch_compute_environment.this to include new values learned so far during
│ apply, provider "registry.terraform.io/hashicorp/aws" changed the planned action from Update to
│ DeleteThenCreate.
│ 
│ This is a bug in the provider, which should be reported in the provider's own issue tracker.

Relevant Error/Panic Output Snippet

╷
│ Error: Provider produced inconsistent final plan
│ 
│ When expanding the plan for aws_batch_compute_environment.this to include new values learned so far during
│ apply, provider "registry.terraform.io/hashicorp/aws" changed the planned action from Update to
│ DeleteThenCreate.
│ 
│ This is a bug in the provider, which should be reported in the provider's own issue tracker.
╵
╷
│ Error: Provider produced inconsistent final plan
│ 
│ When expanding the plan for aws_batch_compute_environment.this to include new values learned so far during
│ apply, provider "registry.terraform.io/hashicorp/aws" produced an invalid new value for .tags_all: was known,
│ but now unknown.
│ 
│ This is a bug in the provider, which should be reported in the provider's own issue tracker.
╵
╷
│ Error: Provider produced inconsistent final plan
│ 
│ When expanding the plan for aws_batch_compute_environment.this to include new values learned so far during
│ apply, provider "registry.terraform.io/hashicorp/aws" produced an invalid new value for .ecs_cluster_arn: was
│ known, but now unknown.
│ 
│ This is a bug in the provider, which should be reported in the provider's own issue tracker.
╵
╷
│ Error: Provider produced inconsistent final plan
│ 
│ When expanding the plan for aws_batch_compute_environment.this to include new values learned so far during
│ apply, provider "registry.terraform.io/hashicorp/aws" produced an invalid new value for .id: was known, but
│ now unknown.
│ 
│ This is a bug in the provider, which should be reported in the provider's own issue tracker.
╵
╷
│ Error: Provider produced inconsistent final plan
│ 
│ When expanding the plan for aws_batch_compute_environment.this to include new values learned so far during
│ apply, provider "registry.terraform.io/hashicorp/aws" produced an invalid new value for .status_reason: was
│ known, but now unknown.
│ 
│ This is a bug in the provider, which should be reported in the provider's own issue tracker.
╵
╷
│ Error: Provider produced inconsistent final plan
│ 
│ When expanding the plan for aws_batch_compute_environment.this to include new values learned so far during
│ apply, provider "registry.terraform.io/hashicorp/aws" produced an invalid new value for .service_role: was
│ known, but now unknown.
│ 
│ This is a bug in the provider, which should be reported in the provider's own issue tracker.
╵
╷
│ Error: Provider produced inconsistent final plan
│ 
│ When expanding the plan for aws_batch_compute_environment.this to include new values learned so far during
│ apply, provider "registry.terraform.io/hashicorp/aws" produced an invalid new value for .status: was known,
│ but now unknown.
│ 
│ This is a bug in the provider, which should be reported in the provider's own issue tracker.
╵
╷
│ Error: Provider produced inconsistent final plan
│ 
│ When expanding the plan for aws_batch_compute_environment.this to include new values learned so far during
│ apply, provider "registry.terraform.io/hashicorp/aws" produced an invalid new value for .tags: was
│ cty.MapValEmpty(cty.String), but now null.
│ 
│ This is a bug in the provider, which should be reported in the provider's own issue tracker.
╵
╷
│ Error: Provider produced inconsistent final plan
│ 
│ When expanding the plan for aws_batch_compute_environment.this to include new values learned so far during
│ apply, provider "registry.terraform.io/hashicorp/aws" produced an invalid new value for .arn: was known, but
│ now unknown.
│ 
│ This is a bug in the provider, which should be reported in the provider's own issue tracker.
╵
╷
│ Error: Provider produced inconsistent final plan
│ 
│ When expanding the plan for aws_batch_compute_environment.this to include new values learned so far during
│ apply, provider "registry.terraform.io/hashicorp/aws" produced an invalid new value for
│ .compute_environment_name_prefix: was known, but now unknown.
│ 
│ This is a bug in the provider, which should be reported in the provider's own issue tracker.
╵
╷
│ Error: Provider produced inconsistent final plan
│ 
│ When expanding the plan for aws_batch_compute_environment.this to include new values learned so far during
│ apply, provider "registry.terraform.io/hashicorp/aws" produced an invalid new value for
│ .compute_resources[0].allocation_strategy: was cty.StringVal(""), but now null.
│ 
│ This is a bug in the provider, which should be reported in the provider's own issue tracker.
╵
╷
│ Error: Provider produced inconsistent final plan
│ 
│ When expanding the plan for aws_batch_compute_environment.this to include new values learned so far during
│ apply, provider "registry.terraform.io/hashicorp/aws" produced an invalid new value for
│ .compute_resources[0].tags: was cty.MapValEmpty(cty.String), but now null.
│ 
│ This is a bug in the provider, which should be reported in the provider's own issue tracker.
╵
╷
│ Error: Provider produced inconsistent final plan
│ 
│ When expanding the plan for aws_batch_compute_environment.this to include new values learned so far during
│ apply, provider "registry.terraform.io/hashicorp/aws" produced an invalid new value for
│ .compute_resources[0].desired_vcpus: was known, but now unknown.
│ 
│ This is a bug in the provider, which should be reported in the provider's own issue tracker.
╵
╷
│ Error: Provider produced inconsistent final plan
│ 
│ When expanding the plan for aws_batch_compute_environment.this to include new values learned so far during
│ apply, provider "registry.terraform.io/hashicorp/aws" produced an invalid new value for
│ .compute_resources[0].bid_percentage: was cty.NumberIntVal(0), but now null.
│ 
│ This is a bug in the provider, which should be reported in the provider's own issue tracker.
╵
╷
│ Error: Provider produced inconsistent final plan
│ 
│ When expanding the plan for aws_batch_compute_environment.this to include new values learned so far during
│ apply, provider "registry.terraform.io/hashicorp/aws" produced an invalid new value for
│ .compute_resources[0].ec2_key_pair: was cty.StringVal(""), but now null.
│ 
│ This is a bug in the provider, which should be reported in the provider's own issue tracker.
╵
╷
│ Error: Provider produced inconsistent final plan
│ 
│ When expanding the plan for aws_batch_compute_environment.this to include new values learned so far during
│ apply, provider "registry.terraform.io/hashicorp/aws" produced an invalid new value for
│ .compute_resources[0].placement_group: was cty.StringVal(""), but now null.
│ 
│ This is a bug in the provider, which should be reported in the provider's own issue tracker.
╵
╷
│ Error: Provider produced inconsistent final plan
│ 
│ When expanding the plan for aws_batch_compute_environment.this to include new values learned so far during
│ apply, provider "registry.terraform.io/hashicorp/aws" produced an invalid new value for
│ .compute_resources[0].spot_iam_fleet_role: was cty.StringVal(""), but now null.
│ 
│ This is a bug in the provider, which should be reported in the provider's own issue tracker.
╵
╷
│ Error: Provider produced inconsistent final plan
│ 
│ When expanding the plan for aws_batch_compute_environment.this to include new values learned so far during
│ apply, provider "registry.terraform.io/hashicorp/aws" produced an invalid new value for
│ .compute_resources[0].launch_template[0].launch_template_name: was cty.StringVal(""), but now null.
│ 
│ This is a bug in the provider, which should be reported in the provider's own issue tracker.

Terraform Configuration Files

resource "aws_batch_compute_environment" "this" {

  compute_environment_name = "test"
  type                            = "MANAGED"

  compute_resources {
    instance_role = "local.istance_role"
    min_vcpus           = 1
    max_vcpus           = 10
    instance_type       = [local.instance_list]
    image_id            =  "local.ami"

    launch_template {
      launch_template_id = aws_launch_template.this.id
      version = aws_launch_template.this.latest_version
    }

    subnets            = [local.subnet_ids]
    security_group_ids = [local.sg_ids]
    type               = "EC2"
  }

  lifecycle {
    create_before_destroy = true
  }
}

resource "aws_launch_template" "this" {
  update_default_version = true
  user_data = base64encode(local.user_data)
}

locals {

  user_data = <<-EOF
        Content-Type: multipart/mixed; boundary="//"
        MIME-Version: 1.0

        --//
        Content-Type: text/x-shellscript; charset="us-ascii"
        MIME-Version: 1.0
        Content-Transfer-Encoding: 7bit
        Content-Disposition: attachment; filename="userdata.txt"

        #!/bin/bash
        echo hello
        echo world
        --//--
        EOF
}

Steps to Reproduce

• terraform init
• terraform apply
• change the userdata
• terraform apply

Debug Output

No response

Panic Output

No response

Important Factoids

No response

References

No response

Would you like to implement a fix?

None

[github-actions[bot]]

Community Note

Voting for Prioritization

• Please vote on this issue by adding a 👍 reaction to the original post to help the community and maintainers prioritize this request.
• Please see our prioritization guide for information on how we prioritize.
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request.

Volunteering to Work on This Issue

• If you are interested in working on this issue, please leave a comment.
• If this would be your first contribution, please review the contribution guide.

[frosforever]

~Is this addressed by https://github.com/hashicorp/terraform-provider-aws/pull/38628?~ Nope as it seems like the changes in that PR are already on main

[inez]

I wonder if there is a known workaround for this issue?

[frosforever]

@inez yell if you find one. I've taken to never updating any compute environment and instead creating new ones and swapping them into the queue. This is manual and tedious but the alternative is that TF blows away the CE when the apply gets an inconsistent plan which is very disruptive.

[simonblake-mp]

just an extra datapoint, this is still happening with provider v5.99.1 on tf v1.9.5

[frosforever]

I haven't had time to dig into this further but wrote up OPs example as an acceptance test which currently fails on main.

Test assumed to be in https://github.com/hashicorp/terraform-provider-aws/blob/bc2e2bb74e4b38a5308b1090d835dd1e36bcda92/internal/service/batch/compute_environment_test.go

func TestAccBatchComputeEnvironment_updateLaunchTemplateId(t *testing.T) {
	ctx := acctest.Context(t)
	var ce awstypes.ComputeEnvironmentDetail
	rName := sdkacctest.RandomWithPrefix(acctest.ResourcePrefix)
	resourceName := "aws_batch_compute_environment.test"
	launchTemplateResourceName := "aws_launch_template.test"

	user_data1 := "foo"
	user_data2 := "bar"

	initialTemplateContent := fmt.Sprintf(`
			locals {
			  user_data_foo = <<-EOF
					Content-Type: multipart/mixed; boundary="//"
					MIME-Version: 1.0

					--//
					Content-Type: text/x-shellscript; charset="us-ascii"
					MIME-Version: 1.0
					Content-Transfer-Encoding: 7bit
					Content-Disposition: attachment; filename="userdata.txt"

					#!/bin/bash
					echo hello
					echo %[2]q
					--//--
					EOF
			}

		   resource "aws_launch_template" "test" {
			 name = %[1]q
			 user_data = base64encode(local.user_data_foo)
		   }
   `, rName, user_data1)

	updatedTemplateContent := fmt.Sprintf(`
			locals {
			  user_data_foo = <<-EOF
					Content-Type: multipart/mixed; boundary="//"
					MIME-Version: 1.0

					--//
					Content-Type: text/x-shellscript; charset="us-ascii"
					MIME-Version: 1.0
					Content-Transfer-Encoding: 7bit
					Content-Disposition: attachment; filename="userdata.txt"

					#!/bin/bash
					echo hello
					echo %[2]q
					--//--
					EOF
			}

		   resource "aws_launch_template" "test" {
			 name = %[1]q
			 user_data = base64encode(local.user_data_foo)
		   }
   `, rName, user_data2)

	resource.ParallelTest(t, resource.TestCase{
		PreCheck:                 func() { acctest.PreCheck(ctx, t); testAccPreCheck(ctx, t) },
		ErrorCheck:               acctest.ErrorCheck(t, names.BatchServiceID),
		ProtoV5ProviderFactories: acctest.ProtoV5ProviderFactories,
		CheckDestroy:             testAccCheckComputeEnvironmentDestroy(ctx),
		Steps: []resource.TestStep{
			{
				Config: acctest.ConfigCompose(
					initialTemplateContent,
					testAccComputeEnvironmentConfig_launchTemplateWithVersion(rName, `aws_launch_template.test.latest_version`)),
				Check: resource.ComposeAggregateTestCheckFunc(
					testAccCheckComputeEnvironmentExists(ctx, resourceName, &ce),
					acctest.CheckResourceAttrRegionalARN(ctx, resourceName, names.AttrARN, "batch", fmt.Sprintf("compute-environment/%s", rName)),
					resource.TestCheckResourceAttrPair(resourceName, "compute_resources.0.launch_template.0.launch_template_id", launchTemplateResourceName, names.AttrID),
					resource.TestCheckResourceAttr(resourceName, "compute_resources.0.launch_template.0.launch_template_name", ""),
				),
			},
			// Swap to version 2 of the launch template
			{
				Config: acctest.ConfigCompose(
					updatedTemplateContent,
					testAccComputeEnvironmentConfig_launchTemplateWithVersion(rName, `aws_launch_template.test.latest_version`)),
				Check: resource.ComposeAggregateTestCheckFunc(
					testAccCheckComputeEnvironmentExists(ctx, resourceName, &ce),
					acctest.CheckResourceAttrRegionalARN(ctx, resourceName, names.AttrARN, "batch", fmt.Sprintf("compute-environment/%s", rName)),
					resource.TestCheckResourceAttrPair(resourceName, "compute_resources.0.launch_template.0.launch_template_id", launchTemplateResourceName, names.AttrID),
					resource.TestCheckResourceAttr(resourceName, "compute_resources.0.launch_template.0.launch_template_name", ""),
				),
			},
			{
				ResourceName:      resourceName,
				ImportState:       true,
				ImportStateVerify: true,
			},
		},
	})
}



func testAccComputeEnvironmentConfig_launchTemplateWithVersion(rName string, version string) string {
	return acctest.ConfigCompose(testAccComputeEnvironmentConfig_base(rName), fmt.Sprintf(`
resource "aws_batch_compute_environment" "test" {
  compute_environment_name = %[1]q

  compute_resources {
	allocation_strategy = "SPOT_PRICE_CAPACITY_OPTIMIZED"
    instance_role = aws_iam_instance_profile.ecs_instance.arn
    instance_type = [
      "c4.large",
    ]

    launch_template {
      launch_template_id = aws_launch_template.test.id
      version = %[2]s
    }

    max_vcpus = 16
    min_vcpus = 0
    security_group_ids = [
      aws_security_group.test.id
    ]
    spot_iam_fleet_role = aws_iam_role.ec2_spot_fleet.arn
    subnets = [
      aws_subnet.test.id
    ]
    type = "SPOT"
  }

  service_role = aws_iam_role.batch_service.arn
  type         = "MANAGED"
  depends_on   = [aws_iam_role_policy_attachment.batch_service]
}
	
`, rName, version))
}

Highly likely this can be reduced to a more minimal example but at least it demonstrates the issue. This test fails for me with

2025/06/12 09:49:33 Initializing Terraform AWS Provider...
=== RUN   TestAccBatchComputeEnvironment_updateLaunchTemplateId
=== PAUSE TestAccBatchComputeEnvironment_updateLaunchTemplateId
=== CONT  TestAccBatchComputeEnvironment_updateLaunchTemplateId
    compute_environment_test.go:1507: Step 2/3 error: Error running apply: exit status 1
        
        Error: Provider produced inconsistent final plan
        
        When expanding the plan for aws_batch_compute_environment.test to include new
        values learned so far during apply, provider
        "registry.terraform.io/hashicorp/aws" changed the planned action from Update
        to DeleteThenCreate.
        
        This is a bug in the provider, which should be reported in the provider's own
        issue tracker.
        
        Error: Provider produced inconsistent final plan
        
        When expanding the plan for aws_batch_compute_environment.test to include new
        values learned so far during apply, provider
        "registry.terraform.io/hashicorp/aws" produced an invalid new value for
        .status_reason: was known, but now unknown.
...

[github-actions[bot]]

[!WARNING] This Issue has been closed, meaning that any additional comments are much easier for the maintainers to miss. Please assume that the maintainers will not see them.

Ongoing conversations amongst community members are welcome, however, the issue will be locked after 30 days. Moving conversations to another venue, such as the AWS Provider forum, is recommended. If you have additional concerns, please open a new issue, referencing this one where needed.

[github-actions[bot]]

This functionality has been released in v6.8.0 of the Terraform AWS Provider. Please see the Terraform documentation on provider versioning or reach out if you need any assistance upgrading.

For further feature requests or bug reports with this functionality, please create a new GitHub issue following the template. Thank you!

[github-actions[bot]]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues. If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.