Idempotency error: json objects read as strings cause resource Update calls when whitespace is detected

[drewmullen]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment
• The resources and data sources in this provider are generated from the CloudFormation schema, so they can only support the actions that the underlying schema supports. For this reason submitted bugs should be limited to defects in the generation and runtime code of the provider. Customizing behavior of the resource, or noting a gap in behavior are not valid bugs and should be submitted as enhancements to AWS via the CloudFormation Open Coverage Roadmap.

Terraform CLI and Terraform AWS Cloud Control Provider Version

Affected Resource(s)

• awscc_networkmanager_core_network

Terraform Configuration Files

Please include all Terraform configurations required to reproduce the bug. Bug reports without a functional reproduction may be closed without investigation.

resource "awscc_networkmanager_global_network" "main" {
  description = "My Global Network"
}

resource "awscc_networkmanager_core_network" "main" {
  description       = "My Core Network"
  global_network_id = awscc_networkmanager_global_network.main.id
  policy_document   = <<EOF
{
  "version": "2021.12",
  "core-network-configuration": {
    "vpn-ecmp-support": false,
    "asn-ranges": [
      "64512-64555"
    ],
    "edge-locations": [
      {
        "location": "us-east-1",
        "asn": 64512
      }
    ]
  },
  "segments": [
    {
      "name": "shared",
      "require-attachment-acceptance": true
    },
    {
      "name": "prod",
      "require-attachment-acceptance": true
    },
    {
      "name": "finance",
      "require-attachment-acceptance": true
    },
    {
      "name": "hr",
      "require-attachment-acceptance": true
    },
    {
      "name": "vpn",
      "require-attachment-acceptance": true
    }
  ],
  "segment-actions": [
    {
      "action": "share",
      "mode": "attachment-route",
      "segment": "shared",
      "share-with": "*"
    }
  ],
  "attachment-policies": [
    {
      "rule-number": 100,
      "condition-logic": "or",
      "conditions": [
        {
          "type": "tag-value",
          "operator": "equals",
          "key": "segment",
          "value": "shared"
        }
      ],
      "action": {
        "association-method": "constant",
        "segment": "shared"
      }
    },
    {
      "rule-number": 200,
      "condition-logic": "or",
      "conditions": [
        {
          "type": "tag-value",
          "operator": "equals",
          "key": "segment",
          "value": "prod"
        }
      ],
      "action": {
        "association-method": "constant",
        "segment": "prod"
      }
    },
    {
      "rule-number": 300,
      "condition-logic": "or",
      "conditions": [
        {
          "type": "tag-value",
          "operator": "equals",
          "key": "segment",
          "value": "finance"
        }
      ],
      "action": {
        "association-method": "constant",
        "segment": "finance"
      }
    },
    {
      "rule-number": 400,
      "condition-logic": "or",
      "conditions": [
        {
          "type": "tag-value",
          "operator": "equals",
          "key": "segment",
          "value": "hr"
        }
      ],
      "action": {
        "association-method": "constant",
        "segment": "hr"
      }
    },
    {
      "rule-number": 500,
      "condition-logic": "or",
      "conditions": [
        {
          "type": "tag-value",
          "operator": "equals",
          "key": "segment",
          "value": "vpn"
        }
      ],
      "action": {
        "association-method": "constant",
        "segment": "vpn"
      }
    }
  ]
}
EOF
}

Expected Behavior

1. terraform apply (create)
2. terraform apply (no change)

Actual Behavior

1. terraform apply (create)
2. terraform apply (white space change detected)

  # awscc_networkmanager_core_network.main will be updated in-place
  ~ resource "awscc_networkmanager_core_network" "main" {
        id                = "core-network-0ecdf681d32cba7b3"
      + owner_account     = (known after apply)
      ~ policy_document   = jsonencode( # whitespace changes

3. error

╷
│ Error: AWS SDK Go Service Operation Incomplete
│ 
│   with awscc_networkmanager_core_network.main,
│   on main.tf line 25, in resource "awscc_networkmanager_core_network" "main":
│   25: resource "awscc_networkmanager_core_network" "main" {
│ 
│ Waiting for Cloud Control API service UpdateResource operation completion returned: waiter state transitioned to FAILED. StatusMessage: Idempotency error.
│ Identical request made with different client token. (Service: NetworkManager, Status Code: 409, Request ID: <>). ErrorCode:
│ ResourceConflict

Steps to Reproduce

1. terraform apply

Important Factoids

References

• #0000

[ewbankkit]

After discussion with the relevant AWS service team it seems that the error is caused by attempting an update operation when there are no material changes to the resource. We need a way of suppressing the difference for JSON documents as we do in terraform-provider-aws. The immediate challenge is how to infer that such functionality is required given that the PolicyDocument property of the AWS::NetworkManager::CoreNetwork resource is of plain string type: https://github.com/hashicorp/terraform-provider-awscc/blob/39c2da41134bf303b046042544bce1690cfa7cc1/internal/service/cloudformation/schemas/AWS_NetworkManager_CoreNetwork.json#L18-L21

[ewbankkit]

A solution may be to mark that property as being of type object and define no nested properties for it. We would then need to change schema code generation to:

1. Generate a Terraform string type for the attribute, NOT a map type
2. Add a PlanModifier to suppress differences for equivalent JSON documents

[ewbankkit]

There are a number of such schema-less object properties, all of which seem like they would benefit from handling as JSON strings. For example:

generating Terraform resource code for "awscc_codeartifact_domain" from "../service/cloudformation/schemas/AWS_CodeArtifact_Domain.json" into "../aws/codeartifact/domain_resource_gen.go" and "../aws/codeartifact/domain_resource_gen_test.go"
PermissionsPolicyDocument is of type object but has no schema
generating Terraform resource code for "awscc_codeartifact_repository" from "../service/cloudformation/schemas/AWS_CodeArtifact_Repository.json" into "../aws/codeartifact/repository_resource_gen.go" and "../aws/codeartifact/repository_resource_gen_test.go"
PermissionsPolicyDocument is of type object but has no schema
generating Terraform resource code for "awscc_ec2_vpc_endpoint" from "../service/cloudformation/schemas/AWS_EC2_VPCEndpoint.json" into "../aws/ec2/vpc_endpoint_resource_gen.go" and "../aws/ec2/vpc_endpoint_resource_gen_test.go"
PolicyDocument is of type object but has no schema
generating Terraform resource code for "awscc_efs_file_system" from "../service/cloudformation/schemas/AWS_EFS_FileSystem.json" into "../aws/efs/file_system_resource_gen.go" and "../aws/efs/file_system_resource_gen_test.go"
FileSystemPolicy is of type object but has no schema
generating Terraform resource code for "awscc_stepfunctions_state_machine" from "../service/cloudformation/schemas/AWS_StepFunctions_StateMachine.json" into "../aws/stepfunctions/state_machine_resource_gen.go" and "../aws/stepfunctions/state_machine_resource_gen_test.go"
Definition is of type object but has no schema

[ewbankkit]

Related:

• https://github.com/hashicorp/terraform-provider-awscc/issues/402
• https://github.com/hashicorp/terraform-plugin-framework/issues/70

[ewbankkit]

Even a suitable AttributePlanModifier is only half the story until #70 is addressed. The ReadResource path isn't affected by the PlanModifier so Terraform will still show an Objects have changed outside of Terraform message:

Note: Objects have changed outside of Terraform

Terraform detected the following changes made outside of Terraform since the last "terraform apply":

  # awscc_networkmanager_core_network.main has been changed
  ~ resource "awscc_networkmanager_core_network" "main" {
        id                = "core-network-0c63c87f4f8751278"
      ~ policy_document   = jsonencode( # whitespace changes
            {
                attachment-policies        = [
                    {
                        action          = {
                            association-method = "constant"
                            segment            = "shared"
                        }
                        condition-logic = "or"
                        conditions      = [
                            {
                                key      = "segment"
                                operator = "equals"
                                type     = "tag-value"
                                value    = "shared"
                            },
                        ]
                        rule-number     = 1
                    },
                ]
                core-network-configuration = {
                    asn-ranges       = [
                        "64512-64555",
                    ]
                    edge-locations   = [
                        {
                            asn      = 64512
                            location = "us-east-1"
                        },
                    ]
                    vpn-ecmp-support = false
                }
                segment-actions            = [
                    {
                        action     = "share"
                        mode       = "attachment-route"
                        segment    = "shared"
                        share-with = "*"
                    },
                ]
                segments                   = [
                    {
                        description                   = "segment-for-shared-services"
                        name                          = "shared"
                        require-attachment-acceptance = true
                    },
                ]
                version                    = "2021.12"
            }
        )
        tags              = [
            {          },
          # (1 unchanged element hidden)
        ]
        # (8 unchanged attributes hidden)
    }

Unless you have made equivalent changes to your configuration, or ignored the relevant attributes using ignore_changes, the following plan
may include actions to undo or respond to these changes.

[ewbankkit]

A workaround for this is to compose the Terraform jsondecode and jsonencode functions to produce a normalized JSON string:

resource "awscc_networkmanager_core_network" "main" {
  description       = "My Core Network"
  global_network_id = awscc_networkmanager_global_network.main.id
  policy_document   = jsonencode(jsondecode(data.aws_networkmanager_core_network_policy_document.main.json))
  tags              = local.terraform_tag
}

Now no diffs are shown:

% terraform plan
awscc_networkmanager_global_network.main: Refreshing state... [id=global-network-07142e587830550d4]
awscc_networkmanager_core_network.main: Refreshing state... [id=core-network-07036ef298285217f]

No changes. Your infrastructure matches the configuration.

Terraform has compared your real infrastructure against your configuration and found no differences, so no changes are needed.

[ewbankkit]

If one of the policy values contains spaces I have noticed that the Cloud Control handler for the CoreNetwork resource is removing them:

% terraform plan
awscc_networkmanager_global_network.main: Refreshing state... [id=global-network-0560af1a3976af6d0]
awscc_networkmanager_core_network.main: Refreshing state... [id=core-network-0682841d123a739d2]

Note: Objects have changed outside of Terraform

Terraform detected the following changes made outside of Terraform since the last "terraform apply":

  # awscc_networkmanager_core_network.main has been changed
  ~ resource "awscc_networkmanager_core_network" "main" {
        id                = "core-network-0682841d123a739d2"
      ~ policy_document   = jsonencode(
          ~ {
              ~ segments                   = [
                  ~ {
                      ~ description                   = "Segment for shared services" -> "Segmentforsharedservices"
                        # (2 unchanged elements hidden)
                    },
                ]
                # (4 unchanged elements hidden)
            }
        )
        tags              = [
            {          },
          # (1 unchanged element hidden)
        ]
        # (8 unchanged attributes hidden)
    }

Unless you have made equivalent changes to your configuration, or ignored the relevant attributes using ignore_changes, the following plan
may include actions to undo or respond to these changes.

─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────

Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
  ~ update in-place

Terraform will perform the following actions:

  # awscc_networkmanager_core_network.main will be updated in-place
  ~ resource "awscc_networkmanager_core_network" "main" {
        id                = "core-network-0682841d123a739d2"
      + owner_account     = (known after apply)
      ~ policy_document   = jsonencode(
          ~ {
              ~ segments                   = [
                  ~ {
                      ~ description                   = "Segmentforsharedservices" -> "Segment for shared services"
                        # (2 unchanged elements hidden)
                    },
                ]
                # (4 unchanged elements hidden)
            }
        )
        tags              = [
            {          },
          # (1 unchanged element hidden)
        ]
        # (8 unchanged attributes hidden)
    }

Plan: 0 to add, 1 to change, 0 to destroy.

─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────

Note: You didn't use the -out option to save this plan, so Terraform can't guarantee to take exactly these actions if you run "terraform
apply" now.

[tfhartmann]

A workaround for this is to compose the Terraform jsondecode and jsonencode functions to produce a normalized JSON string:

resource "awscc_networkmanager_core_network" "main" {
  description       = "My Core Network"
  global_network_id = awscc_networkmanager_global_network.main.id
  policy_document   = jsonencode(jsondecode(data.aws_networkmanager_core_network_policy_document.main.json))
  tags              = local.terraform_tag
}

Now no diffs are shown:

% terraform plan
awscc_networkmanager_global_network.main: Refreshing state... [id=global-network-07142e587830550d4]
awscc_networkmanager_core_network.main: Refreshing state... [id=core-network-07036ef298285217f]

No changes. Your infrastructure matches the configuration.

Terraform has compared your real infrastructure against your configuration and found no differences, so no changes are needed.

Hi! I think I'm seeing this problem, even with the encode(decode()) work around. When I try to follow the example in this doc https://registry.terraform.io/providers/hashicorp%20%20/aws/latest/docs/guides/using-aws-with-awscc-provider

I get the following error :

│ Calling Cloud Control API service CreateResource operation returned: operation error CloudControl: CreateResource, https response error StatusCode: 400, RequestID:
│ bebae537-d031-4316-9194-6a2c0b3cf1de, api error ValidationException: Model validation failed (#/PolicyDocument: expected type: JSONObject, found: String)

This is on initial attempt to create the resources.

[ewbankkit]

@tfhartmann A one-off fix for awscc_networkmanager_core_network.policy_document was done in #537 which is planned to released in v0.25.0 of this provider, likely tomorrow.

[tfhartmann]

@tfhartmann A one-off fix for awscc_networkmanager_core_network.policy_document was done in #537 which is planned to released in v0.25.0 of this provider, likely tomorrow.

Awesome! Thanks!

[joshtrutwin]

Having similar issues with jsonencode on policies for

https://registry.terraform.io/providers/hashicorp/awscc/latest/docs/resources/opensearchserverless_access_policy https://registry.terraform.io/providers/hashicorp/awscc/latest/docs/resources/opensearchserverless_security_policy

resource "awscc_opensearchserverless_security_policy" "test_aoss_encryption_policy" {
  name        = "ce-test-aoss-encryption-policy"
  description = "Encryption Policy for CE Test Collection"
  type        = "encryption"

  policy = jsonencode({
    "Rules" : [
      {
        "Resource" : [
          "collection/ce-test-collection"
        ],
        "ResourceType" : "collection"
      }
    ],
    "AWSOwnedKey" : true
  })
}

perpetual drift followed by apply error:

awscc_opensearchserverless_security_policy.test_aoss_encryption_policy: Modifying... [id=encryption|ce-test-aoss-encryption-policy]
╷
│ Error: AWS SDK Go Service Operation Incomplete
│ 
│   with awscc_opensearchserverless_security_policy.test_aoss_encryption_policy,
│   on opensearch.tf line 31, in resource "awscc_opensearchserverless_security_policy" "test_aoss_encryption_policy":
│   31: resource "awscc_opensearchserverless_security_policy" "test_aoss_encryption_policy" {
│ 
│ Waiting for Cloud Control API service UpdateResource operation completion
│ returned: waiter state transitioned to FAILED. StatusMessage: Invalid
│ request provided: UpdateSecurityPolicyRequest(Description=Encryption Policy
│ for CE Test Collection, Name=ce-test-aoss-encryption-policy,
│ Policy={"AWSOwnedKey":true,"Rules":[{"Resource":["collection/ce-test-collection"],"ResourceType":"collection"}]},
│ PolicyVersion=MTY3NzYyMjUyMTgyNl8x, Type=encryption). ErrorCode:
│ InvalidRequest
╵
Operation failed: failed running terraform apply (exit 1)

Thanks

[joshtrutwin]

Cloudtrail shows:

    "eventSource": "aoss.amazonaws.com",
    "eventName": "UpdateSecurityPolicy",
    "sourceIPAddress": "cloudformation.amazonaws.com",
    "userAgent": "cloudformation.amazonaws.com",
    "errorCode": "ValidationException",
    "errorMessage": "No changes detected in policy or policy description",
    "requestParameters": {
        "type": "encryption",
        "name": "ce-test-aoss-encryption-policy",
        "policyVersion": "MTY3NzYyMjUyMTgyNl8x",
        "description": "Encryption Policy for CE Test Collection",
        "policy": "{\"AWSOwnedKey\":true,\"Rules\":[{\"Resource\":[\"collection/ce-test-collection\"],\"ResourceType\":\"collection\"}]}",
        "clientToken": "<snip>"
    },