ParameterStore should reuse AWS Sessions for multiple requests

[nazarewk]

Describe the solution you'd like I would like the Parameter Store provider to reuse a single AWS Session (in a worst case, probably easiest todo) on all requests done for a single ExternalSecret resource.

Best case scenario the controller could just keep refreshing a single authenticated session for the lifetime of the controller.

What is the added value? Looks like ParameterStore (and most likely Secrets Manager) provider issues a fresh session for each request made to the api effectively doubling the number of requests done to external API (and reducing performance).

Combined with using GetParameter API instead of batch GetParameters it results in a lot of waste and possible performance issues for large scale deployments.

I would expect session reuse to significantly improve the reconciliation times.

Give us examples of the outcome

I am not exactly sure what is wrong with the AWS SDK usage or how could it be improved at this point.

Observations (Constraints, Context, etc):

For the last few days i'm looking into performance issues of ESO on our staging cluster (~30 environments deployed at once, ~30 ExternalSecret objects in each, ~5 Parameters in each secret). I am posting my findings as issues.

After observing "principalid" changing between each request to Parameter Store in AWS CloudTrail Lake, Into verified the provider issues a fresh session for every single GetParameter request.

Notice the AssumeRoleWithWebIdentity.requestParameters.roleSessionName matches the following GetParameter.principalid and each has only a single pair of requests:

CloudTrail entries

[
  {
    "eventTime": "2022-05-27 07:00:00.000",
    "operation": "AssumeRoleWithWebIdentity",
    "eventSource": "sts.amazonaws.com",
    "principalid": "arn:aws:iam::5<REDACTED>8:oidc-provider/oidc.eks.us-east-1.amazonaws.com/id/8C<REDACTED>8B:sts.amazonaws.com:system:serviceaccount:external-secrets:external-secrets",
    "session_username": "",
    "username": "system:serviceaccount:external-secrets:external-secrets",
    "requestParameters": "{roleArn=arn:aws:iam::5<REDACTED>8:role/staging-01-external-secrets, roleSessionName=1653634800072212330}",
    "userIdentity": "{type=WebIdentityUser, principalid=arn:aws:iam::5<REDACTED>8:oidc-provider/oidc.eks.us-east-1.amazonaws.com/id/8C<REDACTED>8B:sts.amazonaws.com:system:serviceaccount:external-secrets:external-secrets, arn=null, accountid=null, accesskeyid=null, username=system:serviceaccount:external-secrets:external-secrets, sessioncontext=null, invokedby=null, identityprovider=arn:aws:iam::5<REDACTED>8:oidc-provider/oidc.eks.us-east-1.amazonaws.com/id/8C<REDACTED>8B}"
  },
  {
    "eventTime": "2022-05-27 07:00:00.000",
    "operation": "GetParameter",
    "eventSource": "ssm.amazonaws.com",
    "principalid": "A<REDACTED>7:1653634800072212330",
    "session_username": "staging-01-external-secrets",
    "username": "",
    "requestParameters": "{withDecryption=true, name=/staging/kubernetes/staging/defaults/partners-reporting/secret.snowflake_password}",
    "userIdentity": "{type=AssumedRole, principalid=A<REDACTED>7:1653634800072212330, arn=arn:aws:sts::5<REDACTED>8:assumed-role/staging-01-external-secrets/1653634800072212330, accountid=5<REDACTED>8, accesskeyid=ASIAXPRZ43VPEWA5YJHK, username=null, sessioncontext={attributes={creationdate=2022-05-27 07:00:00.000, mfaauthenticated=false}, sessionissuer={type=Role, principalid=A<REDACTED>7, arn=arn:aws:iam::5<REDACTED>8:role/staging-01-external-secrets, accountid=5<REDACTED>8, username=staging-01-external-secrets}, webidfederationdata={federatedprovider=arn:aws:iam::5<REDACTED>8:oidc-provider/oidc.eks.us-east-1.amazonaws.com/id/8C<REDACTED>8B, attributes=null}, sourceidentity=null, ec2roledelivery=null, ec2issuedinvpc=null}, invokedby=null, identityprovider=null}"
  },
  {
    "eventTime": "2022-05-27 07:00:01.000",
    "operation": "GetParameter",
    "eventSource": "ssm.amazonaws.com",
    "principalid": "A<REDACTED>7:1653634801170591852",
    "session_username": "staging-01-external-secrets",
    "username": "",
    "requestParameters": "{withDecryption=true, name=/staging/kubernetes/staging/defaults/partners-reporting/secret.snowflake_password}",
    "userIdentity": "{type=AssumedRole, principalid=A<REDACTED>7:1653634801170591852, arn=arn:aws:sts::5<REDACTED>8:assumed-role/staging-01-external-secrets/1653634801170591852, accountid=5<REDACTED>8, accesskeyid=ASIAXPRZ43VPAH2HJEON, username=null, sessioncontext={attributes={creationdate=2022-05-27 07:00:01.000, mfaauthenticated=false}, sessionissuer={type=Role, principalid=A<REDACTED>7, arn=arn:aws:iam::5<REDACTED>8:role/staging-01-external-secrets, accountid=5<REDACTED>8, username=staging-01-external-secrets}, webidfederationdata={federatedprovider=arn:aws:iam::5<REDACTED>8:oidc-provider/oidc.eks.us-east-1.amazonaws.com/id/8C<REDACTED>8B, attributes=null}, sourceidentity=null, ec2roledelivery=null, ec2issuedinvpc=null}, invokedby=null, identityprovider=null}"
  },
  {
    "eventTime": "2022-05-27 07:00:01.000",
    "operation": "AssumeRoleWithWebIdentity",
    "eventSource": "sts.amazonaws.com",
    "principalid": "arn:aws:iam::5<REDACTED>8:oidc-provider/oidc.eks.us-east-1.amazonaws.com/id/8C<REDACTED>8B:sts.amazonaws.com:system:serviceaccount:external-secrets:external-secrets",
    "session_username": "",
    "username": "system:serviceaccount:external-secrets:external-secrets",
    "requestParameters": "{roleArn=arn:aws:iam::5<REDACTED>8:role/staging-01-external-secrets, roleSessionName=1653634801170591852}",
    "userIdentity": "{type=WebIdentityUser, principalid=arn:aws:iam::5<REDACTED>8:oidc-provider/oidc.eks.us-east-1.amazonaws.com/id/8C<REDACTED>8B:sts.amazonaws.com:system:serviceaccount:external-secrets:external-secrets, arn=null, accountid=null, accesskeyid=null, username=system:serviceaccount:external-secrets:external-secrets, sessioncontext=null, invokedby=null, identityprovider=arn:aws:iam::5<REDACTED>8:oidc-provider/oidc.eks.us-east-1.amazonaws.com/id/8C<REDACTED>8B}"
  },
  {
    "eventTime": "2022-05-27 07:00:03.000",
    "operation": "AssumeRoleWithWebIdentity",
    "eventSource": "sts.amazonaws.com",
    "principalid": "arn:aws:iam::5<REDACTED>8:oidc-provider/oidc.eks.us-east-1.amazonaws.com/id/8C<REDACTED>8B:sts.amazonaws.com:system:serviceaccount:external-secrets:external-secrets",
    "session_username": "",
    "username": "system:serviceaccount:external-secrets:external-secrets",
    "requestParameters": "{roleArn=arn:aws:iam::5<REDACTED>8:role/staging-01-external-secrets, roleSessionName=1653634803899903136}",
    "userIdentity": "{type=WebIdentityUser, principalid=arn:aws:iam::5<REDACTED>8:oidc-provider/oidc.eks.us-east-1.amazonaws.com/id/8C<REDACTED>8B:sts.amazonaws.com:system:serviceaccount:external-secrets:external-secrets, arn=null, accountid=null, accesskeyid=null, username=system:serviceaccount:external-secrets:external-secrets, sessioncontext=null, invokedby=null, identityprovider=arn:aws:iam::5<REDACTED>8:oidc-provider/oidc.eks.us-east-1.amazonaws.com/id/8C<REDACTED>8B}"
  },
  {
    "eventTime": "2022-05-27 07:00:03.000",
    "operation": "GetParameter",
    "eventSource": "ssm.amazonaws.com",
    "principalid": "A<REDACTED>7:1653634803899903136",
    "session_username": "staging-01-external-secrets",
    "username": "",
    "requestParameters": "{withDecryption=true, name=/staging/kubernetes/staging/defaults/partners-reporting/secret.snowflake_password}",
    "userIdentity": "{type=AssumedRole, principalid=A<REDACTED>7:1653634803899903136, arn=arn:aws:sts::5<REDACTED>8:assumed-role/staging-01-external-secrets/1653634803899903136, accountid=5<REDACTED>8, accesskeyid=ASIAXPRZ43VPLFMKFSO7, username=null, sessioncontext={attributes={creationdate=2022-05-27 07:00:03.000, mfaauthenticated=false}, sessionissuer={type=Role, principalid=A<REDACTED>7, arn=arn:aws:iam::5<REDACTED>8:role/staging-01-external-secrets, accountid=5<REDACTED>8, username=staging-01-external-secrets}, webidfederationdata={federatedprovider=arn:aws:iam::5<REDACTED>8:oidc-provider/oidc.eks.us-east-1.amazonaws.com/id/8C<REDACTED>8B, attributes=null}, sourceidentity=null, ec2roledelivery=null, ec2issuedinvpc=null}, invokedby=null, identityprovider=null}"
  }
]

[moolen]

Hey @nazarewk thank you for the research, i think we can re-use sessions in pkg/provider/aws/auth by storing them in a map[string]*session.Session. The lookup key can be the store's {namespace}/{name}. That way we even can re-use sessions across multiple Kind=ExternalSecret.

[albertollamaso]

@moolen Do we ever need to map the session?. I think a simple var type *session.Session where we can store the first session is created first time would do it. Then we can reuse it all the time.

My concern is, what if the session expired. I did a quick check and didn't see the session package has a way to check expiration.

Happy to contribute with a PR after we align.

[moolen]

I believe this can be closed; feature has been implemented.