Elasticache resource continuously 'modifying' when securityGroup is specified

[robertgates55]

When specifying a security group on a ReplicationGroup (either via spec.forProvider.securityGroupIds or spec.forProvider.securityGroupIdRefs - referencing a crossplane-created sg or a manually created one) the created replicationgroup in AWS is constantly reverting to modifying.

Example:

apiVersion: cache.aws.crossplane.io/v1beta1
kind: ReplicationGroup
metadata:
  name: rg-test-without-sg
  namespace: dev
spec:
  deletionPolicy: Delete
  forProvider:
    applyModificationsImmediately: true
    atRestEncryptionEnabled: true
    authEnabled: false
    automaticFailoverEnabled: true
    cacheNodeType: cache.t3.micro
    cacheParameterGroupName: default.redis5.0
    cacheSubnetGroupName: dev-dev1-cache-subnet-group
    engine: redis
    engineVersion: 5.0.6
    numCacheClusters: 3
    port: 6379
    preferredMaintenanceWindow: tue:02:00-tue:03:00
    region: eu-west-1
    replicationGroupDescription: dev/robgates-redis-no-sg
    snapshotWindow: 03:00-04:00
    tags:
    - key: Owner
      value: crossplane
    transitEncryptionEnabled: false
---
apiVersion: cache.aws.crossplane.io/v1beta1
kind: ReplicationGroup
metadata:
  name: rg-test-with-sg
  namespace: dev
spec:
  deletionPolicy: Delete
  forProvider:
    applyModificationsImmediately: true
    atRestEncryptionEnabled: true
    authEnabled: false
    automaticFailoverEnabled: true
    cacheNodeType: cache.t3.micro
    cacheParameterGroupName: default.redis5.0
    cacheSubnetGroupName: dev-dev1-cache-subnet-group
    engine: redis
    engineVersion: 5.0.6
    numCacheClusters: 3
    port: 6379
    preferredMaintenanceWindow: tue:02:00-tue:03:00
    region: eu-west-1
    replicationGroupDescription: dev/robgates-redis-with-sg
    securityGroupIds:
    - sg-0d3a26d0467c76eed
    snapshotWindow: 03:00-04:00
    tags:
    - key: Owner
      value: crossplane
    transitEncryptionEnabled: false

Looking in CloudTrail, I can see constant (every 30s) calls to the ModifyNetworkInterfaceAttribute event, constantly updating each of the replicationgroup's enis with the reference to the securitygroup specified.

{
    "eventVersion": "1.08",
    "userIdentity": {
        "type": "AssumedRole",
...
        "sessionContext": {
            "sessionIssuer": {
                "type": "Role",
...
                "userName": "AWSServiceRoleForElastiCache"
            },
            "webIdFederationData": {},
            "attributes": {
                "creationDate": "2021-12-18T11:28:17Z",
                "mfaAuthenticated": "false"
            }
        },
        "invokedBy": "ec.amazonaws.com"
    },
    "eventTime": "2021-12-18T11:28:17Z",
    "eventSource": "ec2.amazonaws.com",
    "eventName": "ModifyNetworkInterfaceAttribute",
    "awsRegion": "eu-west-1",
    "sourceIPAddress": "ec.amazonaws.com",
    "userAgent": "ec.amazonaws.com",
    "requestParameters": {
        "networkInterfaceId": "eni-01f3ba00b4b809dae",
        "groupSet": {
            "items": [
                {
                    "groupId": "sg-0d3a26d0467c76eed"
                }
            ]
        }
    },
    "responseElements": {
        "requestId": "61bbee3c-abad-4d1b-b215-dd0a5bbf1f6f",
        "_return": true
    },
    "requestID": "61bbee3c-abad-4d1b-b215-dd0a5bbf1f6f",
    "eventID": "a0a1ff88-ae10-4131-a3e9-103d80ad15de",
    "readOnly": false,
    "eventType": "AwsApiCall",
    "managementEvent": true,
...
    "eventCategory": "Management"
}

Is this known, or have similar things been observed before? It obviously means that the connection secret is never created as the elasticache replicationgroup never settles.

Crossplane: v1.4.1
AWS Provider: v0.20.0

[nicolasbelanger]

Any updates on this one? We just deployed our first ReplicationGroup and got a similar symptom, constantly modifying after a few seconds... In the status I can see that it might have to do with resharding?

    pendingModifiedValues:
      resharding:
        slotMigration:
          progressPercentage: 0

[nicolasbelanger]

Any updates on this one? We just deployed our first ReplicationGroup and got a similar symptom, constantly modifying after a few seconds... In the status I can see that it might have to do with resharding?

    pendingModifiedValues:
      resharding:
        slotMigration:
          progressPercentage: 0

Looks resolved by setting engineVersion: 6.x (previously engineVersion: 6.2)...

[chlunde]

Do you still see this @robertgates55 ? I have not seen this issue.

[JMoserCricut]

We're still seeing this issue @chlunde using provider-aws v0.33.0 and crossplane v1.8.1

Same errors, same cloudtrail logs of ModifyNetworkInterfaceAttribute. We've tried applying the engineVersion as 6.x as @nicolasbelanger mentioned but it didn't resolve our issue

[chlunde]

We don't see this. Here's an example. Can you set that securitygroup successfully using the console?

  forProvider:
    applyModificationsImmediately: false
    atRestEncryptionEnabled: true
    authEnabled: true
    automaticFailoverEnabled: true
    cacheNodeType: cache.t3.micro
    cacheParameterGroupName: redis6-parameters
    cacheSubnetGroupName: cache-subnet-group
    cacheSubnetGroupNameRefs:
      name: cache-subnet-group
    engine: redis
    engineVersion: 6.x
    numNodeGroups: 1
    port: 6379
    preferredMaintenanceWindow: sat:01:00-sat:02:00
    region: eu-north-1
    replicasPerNodeGroup: 1
    replicationGroupDescription: Elasticache redis
    securityGroupIdRefs:
    - name: sg-name
    securityGroupIdSelector:
      matchControllerRef: true
    securityGroupIds:
    - sg-123 # a real one..
    snapshotRetentionLimit: 0
    snapshotWindow: 03:30-04:30
# tags skipped
    transitEncryptionEnabled: true

[haooliveira84]

The same problem happens in the upgrade between 6.x to 7.x

[haooliveira84]

I've updated the provider for the v0.38.0 and it has been solved! 🚀

[github-actions[bot]]

Crossplane does not currently have enough maintainers to address every issue and pull request. This issue has been automatically marked as stale because it has had no activity in the last 90 days. It will be closed in 14 days if no further activity occurs. Leaving a comment starting with /fresh will mark this issue as not stale.