3 issues when changing globalSecondaryIndexes with DynamoDb after creation

[Jimmy89]

What happened?

As the issues are related, I put them into one issue post.

Starting with number 1: after creating a dynamodb with two global secondary indexes with projectionType set to INCLUDE I wanted to update the nonKeyAttributes with some new fields. This requires recreating the secondary index, as an update is not allowed. However, pulumi does not seem to recognize that this action is required and therefore tries to keep 'updating' the non-updatable secondary index. Resulting into issue 1, where the update should have been a replace statement (of the secondary index).

I manually removed the first global secondary index and manually recreated it, with the exact same naming. I then run pulumi refresh. Issue 2: pulumi does not seem to recognize keys based on their index name, but on the order within the array returned from AWS. Therefore, it thinks the second secondary index is actually the first 'old' one. I expect pulumi to match on the indexName instead.

Then frustrated as I become I removed both global secondary keys manually within Dynamodb, run pulumi refresh and pulumi up and here issue 3 starts: aws-native does not allow more than one global secondary key creation in one update statement (so each creation must be a separate update call, according to the error).

Expected Behavior

Issue 1: pulumi detecting a replace statement of the global secondary index.

Issue 2: pulumi detecting that one global secondary index was not adjusted as the index name and fields were equal.

Issue 3: pulumi being able to update more than 1 global secondary index in one run.

Steps to reproduce

new aws.dynamodb.Table(`dynamodb-TESTME`, {
  attributeDefinitions: [
    {
      attributeName: "PK",
      attributeType: "S",
    },
    {
      attributeName: "SK",
      attributeType: "S",
    },
    {
      attributeName: "G1PK",
      attributeType: "S",
    },
    {
      attributeName: "G1SK",
      attributeType: "S",
    },
    {
      attributeName: "G2PK",
      attributeType: "S",
    },
    {
      attributeName: "G2SK",
      attributeType: "S",
    },
  ],
  keySchema: [
    {
      attributeName: "PK",
      keyType: "HASH",
    },
    {
      attributeName: "SK",
      keyType: "RANGE",
    },
  ],
  globalSecondaryIndexes: [
    {
      indexName: "GSI1",
      keySchema: [
        {
          attributeName: "G1PK",
          keyType: "HASH",
        },
        {
          attributeName: "G1SK",
          keyType: "RANGE",
        },
      ],
      projection: {
        projectionType: "INCLUDE",
        nonKeyAttributes: ["field1"],
      },
    },
    {
      indexName: "GSI2",
      keySchema: [
        {
          attributeName: "G2PK",
          keyType: "HASH",
        },
        {
          attributeName: "G2SK",
          keyType: "RANGE",
        },
      ],
      projection: {
        projectionType: "INCLUDE",
        nonKeyAttributes: ["field1"],
      },
    },
  ],
  tableName: "TEST_TABLE",
  billingMode: "PAY_PER_REQUEST",
  timeToLiveSpecification: {
    attributeName: "TTL",
    enabled: false,
  },
});

1. Create the dynamodb table with above code
2. Change of indexName GSI1 the nonKeyAttributes by adding field2 to the array.
3. Run pulumi up, get error

error: operation UPDATE failed with "InvalidRequest": Cannot update GSI's properties other than Provisioned Throughput and Contributor Insights Specification. You can create a new GSI with a different name.

4. Manually remove GSI1 global secondary key from the created Dynamodb table through AWS console.
5. Run pulumi refresh and note that pulumi confuses GSI2 with GSI1.
6. Manually remove GSI2 global secondary key as well through AWS console. No global secondary index keys should exist anymore.
7. Run pulumi refresh -y and pulumi up -y and get error

  error: operation UPDATE failed with "InvalidRequest": Cannot perform more than one GSI creation or deletion in a single update

Output of pulumi about

Dependencies:
NAME                VERSION
@pulumi/aws         5.42.0
@pulumi/aws-native  0.69.0
@pulumi/pulumi      3.76.0
@pulumi/random      4.13.2

Additional context

No response

Contributing

Vote on this issue by adding a 👍 reaction. To contribute a fix for this issue, leave a comment (and link to your pull request, if you've opened one already).

[mjeffryes]

Thanks for the detailed repro steps. Just a quick double-check: Is aws an alias for @pulumi/aws-native in your repro example? or is it an alias for @pulumi/aws?

I think issue 1 will be resolved when we complete #736. I think this might be our first report of the other two issues though, so thank you for walking through those!

For a workaround while we work on a fix; have you tried removing the index from your program, running pulumi up and then adding the index back to the program (with your updates) and running pulumi up again?

[Jimmy89]

@mjeffryes aws is an alias for @pulumi/aws-native.

As workaround I have manually removed all indexes and then added/uncommented one index and running pulumi up each time.

[uqgwil10]

I believe we have also just ran into issue #2 on "@pulumi/aws": "5.42.0" Where it is trying to recreate some indexes because the array indexes don't match.

This could maybe be an issue because our indexes were originally created on "@pulumi/aws": "^4.14.0" and we just upgraded to "5.42.0" and now this is only becoming an issue. So maybe older versions didn't care about order in the array where as recently it seems like it trying to force the order that you declare them.

[mjeffryes]

I believe we have also just ran into issue https://github.com/pulumi/pulumi-aws-native/pull/2 on "@pulumi/aws": "5.42.0" Where it is trying to recreate some indexes because the array indexes don't match.

@uqgwil10 would you mind filing a ticket on the pulumi/pulumi-aws repo so we can track it there. (A fix in this repo won't help you if you're using "@pulumi/aws")

[chrishoward]

@Jimmy89 As per @uqgwil10 's comment above, we are still experiencing the Issue #2 you refer to. Did you find a good solution for it? Or did it stop happening for you after following some workaround?

[Jimmy89]

@chrishoward I have no workaround, only manually deleting all GSI keys manually and one-by-one adding them back through pulumi up. Very annoying issue and it is doable with a very small database (with specific timezone usage), but I really hope this issue will be fixed one day.