[Bug]: Error when update Data Lifecycle Policy

[fdmsantos]

Terraform Core Version

1.7.2

AWS Provider Version

5.42.0

Affected Resource(s)

aws_opensearchserverless_lifecycle_policy

Expected Behavior

Terraform update data lifecycle policy

Actual Behavior

Terraform gives error when try update data lifecycle policy

Relevant Error/Panic Output Snippet

│ Error: Provider produced inconsistent result after apply
│ 
│ When applying changes to module.opensearch_serverless.aws_opensearchserverless_lifecycle_policy.this[0], provider "provider[\"registry.terraform.io/hashicorp/aws\"]" produced an unexpected new value: .policy_version:
│ was cty.StringVal("MTcxMTM1OTI3OTI2MV8x"), but now cty.StringVal("MTcxMTM2MTU2MDA4MV8y").
│ 
│ This is a bug in the provider, which should be reported in the provider's own issue tracker.
╵

Terraform Configuration Files

resource "aws_opensearchserverless_security_policy" "example" { name = "example" type = "encryption" description = "encryption security policy for example-collection" policy = jsonencode({ Rules = [ { Resource = [ "collection/example-collection" ], ResourceType = "collection" } ], AWSOwnedKey = true }) }

Steps to Reproduce

terraform apply change policy terraform apply

Debug Output

No response

Panic Output

No response

Important Factoids

No response

References

No response

Would you like to implement a fix?

None

[github-actions[bot]]

Community Note

Voting for Prioritization

• Please vote on this issue by adding a 👍 reaction to the original post to help the community and maintainers prioritize this request.
• Please see our prioritization guide for information on how we prioritize.
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request.

Volunteering to Work on This Issue

• If you are interested in working on this issue, please leave a comment.
• If this would be your first contribution, please review the contribution guide.

[acwwat]

@fdmsantos I am unable to reproduce the problem with the latest version of the provider. I've tried changing the policy in different ways, and policy_version updated properly as I expect. The resource code also seems to be correct with handling the computed value. What I suspect is that the migration to autoflex via #37085 might have changed the logic and addressed the issue as a result. Please try the Terraform AWS Provider v5.47.0 or later and see if it fixes your issue.

[thda]

I am able to reproduce with a recent version (5.59)

[mstrisoline]

- Installing hashicorp/aws v5.61.0...
- Installed hashicorp/aws v5.61.0 (signed by HashiCorp)

Re-running seems to work, but all changes introduce an initial failure on apply.

[workmanw]

We get this problem intermittently and I'm not completely sure why.

Error

module.common.aws_opensearchserverless_access_policy.data_access_policy: Modifying... [id=dev-core-search]
╷
│ Error: Provider produced inconsistent result after apply
│ 
│ When applying changes to
│ module.common.aws_opensearchserverless_access_policy.data_access_policy,
│ provider "provider[\"registry.terraform.io/hashicorp/aws\"]" produced an
│ unexpected new value: .policy_version: was
│ cty.StringVal("MTcyMTMxODM4NjY3NV[82](https://..../service-core-search/-/jobs/41819046#L82)"), but now
│ cty.StringVal("MTcyMzc1MjM2MTk2MF[83](https://...../service-core-search/-/jobs/41819046#L83)").
│ 
│ This is a bug in the provider, which should be reported in the provider's
│ own issue tracker.

hashicorp/aws: v5.61.0 terraform: 1.9.3

TF definition

locals {
  collection_name = "dev-core-search"
}

resource "aws_opensearchserverless_collection" "collection" {
  name = local.collection_name
  type = "SEARCH"

  depends_on = [aws_opensearchserverless_security_policy.encryption_policy]
}

resource "aws_opensearchserverless_security_policy" "encryption_policy" {
  name        = local.collection_name
  type        = "encryption"
  description = "encryption policy for ${local.collection_name}"
  policy      = jsonencode({
    Rules = [
      {
        Resource     = ["collection/${local.collection_name}"],
        ResourceType = "collection"
      }
    ],
    AWSOwnedKey = true
  })
}

resource "aws_opensearchserverless_security_policy" "network_policy" {
  name        = local.collection_name
  type        = "network"
  description = "public access for dashboard, VPC access for collection endpoint"
  policy      = jsonencode([
    {
      Description = "VPC access for collection endpoint",
      Rules       = [
        {
          ResourceType = "collection",
          Resource     = [
            "collection/${local.collection_name}"
          ]
        }
      ],
      AllowFromPublic = true
    },
    {
      Description = "Public access for dashboards",
      Rules       = [
        {
          ResourceType = "dashboard"
          Resource     = [
            "collection/${local.collection_name}"
          ]
        }
      ],
      AllowFromPublic = true
    }
  ])
}

resource "aws_opensearchserverless_access_policy" "data_access_policy" {
  name        = local.collection_name
  type        = "data"
  description = "allow index and collection access"
  policy      = jsonencode([
    {
      Rules = [
        {
          ResourceType = "index",
          Resource     = [
            "index/${local.collection_name}/*"
          ],
          Permission = [
            "aoss:CreateIndex",
            "aoss:DeleteIndex",
            "aoss:UpdateIndex",
            "aoss:DescribeIndex",
            "aoss:ReadDocument",
            "aoss:WriteDocument"
          ]
        },
        {
          ResourceType = "collection",
          Resource     = [
            "collection/${local.collection_name}"
          ],
          Permission = [
            "aoss:CreateCollectionItems",
            "aoss:DeleteCollectionItems",
            "aoss:UpdateCollectionItems",
            "aoss:DescribeCollectionItems"
          ]
        }
      ],
      Principal = [
        module.k8s_service_account_role.role.arn
      ]
    },
    {
      Description = "Developer read-only access",
      Rules = [
        {
          ResourceType = "index",
          Resource     = [
            "index/${local.collection_name}/*"
          ],
          Permission = [
            "aoss:DescribeIndex",
            "aoss:ReadDocument"
          ]
        },
        {
          ResourceType = "collection",
          Resource     = [
            "collection/${local.collection_name}"
          ],
          Permission = [
            "aoss:DescribeCollectionItems"
          ]
        }
      ],
      Principal = [
        module.globals.developer_role_arn
      ]
    }
  ])
}

[johankarlssonpaf]

We have the same issue, we can create a policy but not change it.

resource "aws_opensearchserverless_security_policy" "example" {
  name        = "network-example"
  type        = "network"
  description = "Public access"
  policy = jsonencode([
    {
      Description = "Public access to collection and Dashboards endpoint for example collection",
      Rules = [
        {
          ResourceType = "collection",
          Resource = [
            "collection/example-collection"
          ]
        },
        {
          ResourceType = "dashboard"
          Resource = [
            "collection/example-collection"
          ]
        }
      ],
      AllowFromPublic = true
    }
  ])
}

Run terraform init, terraform apply, change the description in the json code of the policy and run terraform apply.

terraform version
Terraform v1.9.5
on darwin_arm64
+ provider registry.terraform.io/hashicorp/aws v5.64.0

terraform apply    
aws_opensearchserverless_security_policy.example: Refreshing state... [id=network-example]

Terraform used the selected providers to generate the following execution plan. Resource actions are indicated
with the following symbols:
  ~ update in-place

Terraform will perform the following actions:

  # aws_opensearchserverless_security_policy.example will be updated in-place
  ~ resource "aws_opensearchserverless_security_policy" "example" {
        id             = "network-example"
        name           = "network-example"
      ~ policy         = jsonencode(
          ~ [
              ~ {
                  ~ Description     = "Public access to collection and Dashboards endpoint for example collection" -> "Public access to collection and Dashboards endpoint for example collection1"
                    # (2 unchanged attributes hidden)
                },
            ]
        )
        # (3 unchanged attributes hidden)
    }

Plan: 0 to add, 1 to change, 0 to destroy.

Do you want to perform these actions?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.

  Enter a value: yes

aws_opensearchserverless_security_policy.example: Modifying... [id=network-example]
╷
│ Error: Provider produced inconsistent result after apply
│ 
│ When applying changes to aws_opensearchserverless_security_policy.example, provider
│ "provider[\"registry.terraform.io/hashicorp/aws\"]" produced an unexpected new value: .policy: was
│ cty.StringVal("[{\"AllowFromPublic\":true,\"Description\":\"Public access to collection and Dashboards endpoint
│ for example
│ collection1\",\"Rules\":[{\"Resource\":[\"collection/example-collection\"],\"ResourceType\":\"collection\"},{\"Resource\":[\"collection/example-collection\"],\"ResourceType\":\"dashboard\"}]}]"),
│ but now cty.StringVal("[{\"AllowFromPublic\":true,\"Description\":\"Public access to collection and Dashboards
│ endpoint for example
│ collection\",\"Rules\":[{\"Resource\":[\"collection/example-collection\"],\"ResourceType\":\"collection\"},{\"Resource\":[\"collection/example-collection\"],\"ResourceType\":\"dashboard\"}]}]").
│ 
│ This is a bug in the provider, which should be reported in the provider's own issue tracker.
╵
╷
│ Error: Provider produced inconsistent result after apply
│ 
│ When applying changes to aws_opensearchserverless_security_policy.example, provider
│ "provider[\"registry.terraform.io/hashicorp/aws\"]" produced an unexpected new value: .policy_version: was
│ cty.StringVal("MTcyNDg0MjUwOTQyMV8x"), but now cty.StringVal("MTcyNDg0MjUzMTk3OV8y").
│ 
│ This is a bug in the provider, which should be reported in the provider's own issue tracker.

Tested both on ARM and X86 and same error.

[joshtrutwin]

Another report - attempted to update the data access policy with some new values:

Terraform v1.9.2
on linux_amd64
Initializing plugins and modules...
module.test_osis_pipeline.aws_opensearchserverless_access_policy.aoss_osis_data_access_policy: Modifying... [id=test-dynamo-pipeline-osis]
╷
│ Error: Provider produced inconsistent result after apply
│ 
│ When applying changes to
│ module.test_osis_pipeline.aws_opensearchserverless_access_policy.aoss_osis_data_access_policy,
│ provider "provider[\"registry.terraform.io/hashicorp/aws\"]" produced an
│ unexpected new value: .policy: was
│ cty.StringVal("[{\"Description\":\"Pipeline Write Access Data
│ Policy\",\"Principal\":[\"arn:aws:iam::<snipped>:role/osis/test-dynamo-pipeline-pipelines-access-role\"],\"Rules\":[{\"Permission\":[\"aoss:CreateIndex\",\"aoss:DescribeIndex\",\"aoss:ReadDocument\",\"aoss:UpdateIndex\",\"aoss:WriteDocument\"],\"Resource\":[\"index/osis-test-collection/*\"],\"ResourceType\":\"index\"}]}]"),
│ but now cty.StringVal("[{\"Description\":\"Pipeline Write Access Data
│ Policy\",\"Principal\":[\"arn:aws:iam::<snipped>:role/osis/test-dynamo-pipeline-pipelines-access-role\"],\"Rules\":[{\"Permission\":[\"aoss:DescribeIndex\",\"aoss:CreateIndex\",\"aoss:UpdateIndex\",\"aoss:WriteDocument\"],\"Resource\":[\"index/osis-test-collection/*\"],\"ResourceType\":\"index\"}]}]").
│ 
│ This is a bug in the provider, which should be reported in the provider's
│ own issue tracker.
╵
╷
│ Error: Provider produced inconsistent result after apply
│ 
│ When applying changes to
│ module.test_osis_pipeline.aws_opensearchserverless_access_policy.aoss_osis_data_access_policy,
│ provider "provider[\"registry.terraform.io/hashicorp/aws\"]" produced an
│ unexpected new value: .policy_version: was
│ cty.StringVal("MTcyNTY1NTc0MjU0NV8x"), but now
│ cty.StringVal("MTcyNTY1ODcwMzA4MV8y").
│ 
│ This is a bug in the provider, which should be reported in the provider's
│ own issue tracker.
╵
Operation failed: failed running terraform apply (exit 1)

[joshtrutwin]

Plan for the post above:

Terraform will perform the following actions:

  # module.test_osis_pipeline.aws_opensearchserverless_access_policy.aoss_osis_data_access_policy will be updated in-place
  ~ resource "aws_opensearchserverless_access_policy" "aoss_osis_data_access_policy" {
        id             = "test-dynamo-pipeline-osis"
        name           = "test-dynamo-pipeline-osis"
      ~ policy         = jsonencode(
          ~ [
              ~ {
                  ~ Rules       = [
                      ~ {
                          ~ Permission   = [
                              - "aoss:DescribeIndex",
                                "aoss:CreateIndex",
                              + "aoss:DescribeIndex",
                              + "aoss:ReadDocument",
                                "aoss:UpdateIndex",
                                # (1 unchanged element hidden)
                            ]
                            # (2 unchanged attributes hidden)
                        },
                    ]
                    # (2 unchanged attributes hidden)
                },
            ]
        )
        # (3 unchanged attributes hidden)
    }

Plan: 0 to add, 1 to change, 0 to destroy.

[joshtrutwin]

and when attempting a second apply:

Terraform v1.9.2
on linux_amd64
Initializing plugins and modules...
module.test_osis_pipeline.aws_opensearchserverless_access_policy.aoss_osis_data_access_policy: Modifying... [id=test-dynamo-pipeline-osis]
╷
│ Error: updating Security Policy (test-dynamo-pipeline-osis)
│ 
│   with module.test_osis_pipeline.aws_opensearchserverless_access_policy.aoss_osis_data_access_policy,
│   on .terraform/modules/test_osis_pipeline/opensearch.tf line 14, in resource "aws_opensearchserverless_access_policy" "aoss_osis_data_access_policy":
│   14: resource "aws_opensearchserverless_access_policy" "aoss_osis_data_access_policy" {
│ 
│ operation error OpenSearchServerless: UpdateAccessPolicy, https response
│ error StatusCode: 400, RequestID: 586d4ae2-5e37-471f-af7a-361f644d744d,
│ ValidationException: No changes detected in policy or policy description
╵
Operation failed: failed running terraform apply (exit 1)

[imaginaryCorn]

We were urgently blocked by this issue, so I created a rough odd workaround that may be useful or a path to something better if I am going off the rails here (main assumption I am aware of is this replacement of a policy being able to prevent interruptions on AOSS, I have no idea if two policies can briefly overlap without issues):

• The policy resource has a problem updating in-place, so instead re-create the policy resource on any content changes changes.
• Give the policy name a random string so that its replacement can be made first before the deletion, just in case live connections get briefly blocked by a lack of permissions.
• Re-generate that name only when the policy string changes.
• Avoid using jsonencode, as mentioned from https://github.com/hashicorp/terraform-provider-awscc/issues/825. If needing dynamic outputs we use templatestring or templatefile
• Finally, ignore changes to the policy field itself. Policy updates get triggered the long way round, but still consistently at least on my own runs and re-runs.

locals {
  data_policy_readwrite = <<-AOSSPOLICY
  [
    {
      "Principal": [$${role_arns}],
      "Rules": [
        {
          "Permission": [
            "aoss:CreateCollectionItems",
            "aoss:DeleteCollectionItems",
            "aoss:UpdateCollectionItems",
            "aoss:DescribeCollectionItems"
          ],
          "Resource": [
            "collection/example"
          ],
          "ResourceType": "collection"
        },
        {
          "Permission": [
            "aoss:CreateIndex",
            "aoss:DeleteIndex",
            "aoss:UpdateIndex",
            "aoss:DescribeIndex",
            "aoss:ReadDocument",
            "aoss:WriteDocument"
          ],
          "Resource": [
            "index/example/*"
          ],
          "ResourceType": "index"
        }
      ]
    }
  ]
  AOSSPOLICY
}


resource "random_string" "readwrite_regenerate" {
  length  = 8
  lower   = true
  special = false
  upper   = false
  numeric = false

  # Tie changes to the policy content to the re-creation of the random string:
  keepers = {
    policy = local.data_policy_readwrite
  }
}

resource "aws_opensearchserverless_access_policy" "read_write_data_policy" {
  name        = "example-read-write-${random_string.readwrite_regenerate.result}"
  type        = "data"
  description = "Gives read and write permissions to the example collection"

  # may be overkill atop overkill, but reduce the whitespace in the policy string before applying it:
  policy = replace(local.data_policy_readwrite, "/\n */", " ")

  lifecycle {
    create_before_destroy = true
    ignore_changes        = [policy]
  }
}

Maybe we could have name_prefix available to these policy resources to maybe make forced recreation/create-before-delete easier?

[johankarlssonpaf]

https://github.com/hashicorp/terraform-provider-aws/releases/tag/v5.68.0 solved our issues with aws_opensearchserverless_security_policy updates.

[fdmsantos]

Solved on 5.68

[github-actions[bot]]

[!WARNING] This issue has been closed, meaning that any additional comments are hard for our team to see. Please assume that the maintainers will not see them.

Ongoing conversations amongst community members are welcome, however, the issue will be locked after 30 days. Moving conversations to another venue, such as the AWS Provider forum, is recommended. If you have additional concerns, please open a new issue, referencing this one where needed.

[github-actions[bot]]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues. If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.