[Bug]: Lake Formation permissions with multiple LF Tag Key/Value pairs saves values in wrong Key in State

[joferrao]

Terraform Core Version

0.14.11

AWS Provider Version

4.47.8

Affected Resource(s)

The apply of the following resource is successful but impossible to destroy because firstValue gets saved in the secondKey in state (same for secondValue in the firstKey).

resource "aws_lakeformation_permissions" "this" {
  principal                        = "XXXXXXXXX"
  permissions                   = ["DESCRIBE"]
  permissions_with_grant_option = ["DESCRIBE"]
  catalog_id                    = data.aws_caller_identity.current.account_id

  lf_tag_policy {
    resource_type = "DATABASE"

    expression {
      key    = "secondKey"
      values = ["secondValue"]
    }
    expression {
      key    = "firstKey"
      values = ["firstValue"]
    }
  }
}

Statefile

 {
      "mode": "managed",
      "type": "aws_lakeformation_permissions",
      "name": "this",
      "provider": "provider[\"registry.terraform.io/hashicorp/aws\"]",
      "instances": [
        {
          "schema_version": 0,
          "attributes": {
            "catalog_id": "YYYYYYYYYY",
            "catalog_resource": false,
            "data_location": [],
            "database": [],
            "id": "1886857363",
            "lf_tag": [],
            "lf_tag_policy": [
              {
                "catalog_id": "YYYYYYYYYY",
                "expression": [
                  {
                    "key": "secondKey",
                    "values": [
                      "firstValue"
                    ]
                  },
                  {
                    "key": "firstKey",
                    "values": [
                      "secondValue"
                    ]
                  }
                ],
                "resource_type": "DATABASE"
              }
            ],
            "permissions": [
              "DESCRIBE"
            ],
            "permissions_with_grant_option": [
              "DESCRIBE"
            ],
            "principal": "XXXXXXXXX",
            "table": [],
            "table_with_columns": []
          },
          "sensitive_attributes": [],
          "private": "bnVsbA=="
        }
      ]
    },

And while using TF_LOG=DEBUG, we found the following:

2022/12/19 12:09:05 [WARN] Provider "registry.terraform.io/hashicorp/aws" produced an unexpected new value for aws_lakeformation_permissions.share_data_with_external_roles, but we are tolerating it because it is using the legacy plugin SDK.
    The following problems may be the cause of any confusing errors from downstream operations:
      - .lf_tag_policy[0].expression[0].key: was cty.StringVal("firstKey"), but now cty.StringVal("secondKey")
      - .lf_tag_policy[0].expression[1].key: was cty.StringVal("secondKey"), but now cty.StringVal("firstKey")
aws_lakeformation_permissions.share_data_with_external_roles: Creation complete after 1s [id=1886857363]

Expected Behavior

LF Key Values are saved in the LF Keys that they were originally created in.

Actual Behavior

LF Key Values are switched to the wrong LF Key (in statefile).

Relevant Error/Panic Output Snippet

2022/12/19 12:09:05 [WARN] Provider "registry.terraform.io/hashicorp/aws" produced an unexpected new value for aws_lakeformation_permissions.share_data_with_external_roles, but we are tolerating it because it is using the legacy plugin SDK.
    The following problems may be the cause of any confusing errors from downstream operations:
      - .lf_tag_policy[0].expression[0].key: was cty.StringVal("firstKey"), but now cty.StringVal("secondKey")
      - .lf_tag_policy[0].expression[1].key: was cty.StringVal("secondKey"), but now cty.StringVal("firstKey")
aws_lakeformation_permissions.share_data_with_external_roles: Creation complete after 1s [id=1886857363]

Terraform Configuration Files

resource "aws_lakeformation_permissions" "this" {
  principal                        = "XXXXXXXXX"
  permissions                   = ["DESCRIBE"]
  permissions_with_grant_option = ["DESCRIBE"]
  catalog_id                    = data.aws_caller_identity.current.account_id

  lf_tag_policy {
    resource_type = "DATABASE"

    expression {
      key    = "secondKey"
      values = ["secondValue"]
    }
    expression {
      key    = "firstKey"
      values = ["firstValue"]
    }
  }
}

Steps to Reproduce

1. terraform apply
2. aws lakeformation list-permissions # yields the permissions were correctly created
3. terraform state pull # shows the lf values are saved in the wrong lf keys

Debug Output

2022/12/19 12:09:05 [WARN] Provider "registry.terraform.io/hashicorp/aws" produced an unexpected new value for aws_lakeformation_permissions.share_data_with_external_roles, but we are tolerating it because it is using the legacy plugin SDK. The following problems may be the cause of any confusing errors from downstream operations: - .lf_tag_policy[0].expression[0].key: was cty.StringVal("secondKey"), but now cty.StringVal("firstKey") - .lf_tag_policy[0].expression[1].key: was cty.StringVal("firstKey"), but now cty.StringVal("secondKey") aws_lakeformation_permissions.share_data_with_external_roles: Creation complete after 1s [id=1886857363]

Panic Output

No response

Important Factoids

No response

References

No response

Would you like to implement a fix?

None

[github-actions[bot]]

Community Note

Voting for Prioritization

• Please vote on this issue by adding a 👍 reaction to the original post to help the community and maintainers prioritize this request.
• Please see our prioritization guide for information on how we prioritize.
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request.

Volunteering to Work on This Issue

• If you are interested in working on this issue, please leave a comment.
• If this would be your first contribution, please review the contribution guide.

[joferrao]

I'm wondering if anyone is using LF at all with Terraform as this just makes LF-Tags unusable. We now ran different tests and even when we declare the LF-Tag permissions in alphabetical order, their values still end up mixed up in the state leading to: impossible to update or to destroy.

I tried to follow the instructions to figure out if I can propose a fix myself, but my work laptop isn't able to. I will try at home at some point. If someone with experience in this provide has a clue of where to start looking for the issue, would be appreciate.

[joferrao]

As an update to the ticket, it seems that Terraform isn't specifically storing the keys in alphabetical order as I just encountered yet another situation where this issue happened (value of the keys got mixed up) even though they were ordered.

[gatomei3]

After further investigation of this bug we discovered that, when grating permissions using a composite LF-tag expression (having multiple key-value pairs), the AWS provider sorts the list of pairs in alphabetical order by keys. However, on the AWS side, this order is not (always) maintained, and the expression will have the key-value pairs ordered by other rules (not known by us). This difference in the key-value pairs ordering is what confuses the AWS provider, and when a read of the newly created permissions is attempted, the provider will expect the pairs to be in alphabetical order, but it will receive them from AWS in a different order. As such, trying to update its state, the provider will replace the keys, so that they respect the order from AWS, but it won't update their corresponding values too. So the resulting mapping of keys to values will be invalid. Example:

• When the AWS provider will send the request to AWS to Grant Permissions, the LF-tag expression will look like the one following: Expression: [{“TagKey”:”key_a”, “TagValues”: [”value_a”]}, { “TagKey”:”key_b”, “TagValues”: [”value_b”] }]
• AWS will grant the permissions, but it will save the expression as: Expression: [ {“TagKey”:”key_b”, “TagValues”: [”value_b”] }, { “TagKey”:”key_a”, “TagValues”: [”value_a”] }]
• The AWS provider will send a request to AWS to list the permissions newly created. It will expect the tags in alphabetical order, but it will receive them as they are stored in AWS. It will try to update the order on its side too, but it will only change the keys order and their corresponding values not. So the resulting expression, that will also be stored in the TF state file, will be: Expression: [{“TagKey”:”key_b”, “TagValues”: [”value_a”] }, { “TagKey”:”key_a”, “TagValues”: [”value_b”] }]

[maiconbaum]

I'm facing the exact same issue on AWS Provider 4.55.0 using Terraform 1.3.9. The stored state follows exactly the same logic as @gatomei3 described.

[hermes-pimentel]

Same issue here.

[balluashok]

Same issue here.

[ewbankkit]

It looks like we need to change lf_tag_policy.expression to TypeSet as order is not important when calculating diffs.

[balluashok]

Thanks @ewbankkit - how long it may take to implement the change , we created ticket to Hashi corp on this. Can you please share your inputs.

[ewbankkit]

@balluashok Working on it now -- I have added an acceptance test case that reproduces the problem. We should have a fix for this week's Terraform AWS Provider v4.64.0 release.

[balluashok]

Thanks @ewbankkit for the update.

[github-actions[bot]]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues. If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.