Required AWS endpoint configuration does not work with STS/IRSA

[derSascha]

Bug description

The AWS secrets-manager keystore requires an endpoint configuration, and fails if not provided:

keystore:
     aws:
       secretsmanager:
         endpoint: secretsmanager.us-east-2.amazonaws.com  # Use the SecretsManager in your region.
         region:   us-east-2                               # Use your region
         kmskey:   ""                                      # Your AWS-KMS master key (CMK) - optional.
         credentials:
           accesskey: "" # Your AWS Access Key
           secretkey: "" # Your AWS Secret Key

The example use secretsmanager.us-east-2.amazonaws.com as endpoint. The implementation use this endpoint to configuration a session client, that overwrites the endpoints of all services in the SDK:

https://github.com/minio/kes/blob/2e4e7be2b9da884cdc98d762849c2566f7b9018e/internal/keystore/aws/secrets-manager.go#L77-L84

This works if e.g. the access and secret key was provided, and we only need to contact the secrets-manager.

In case the config does not contain the secrets, the SDK use the environment, and might need to contact the STS service to fetch a token. This does not work anymore with the secrets-manager endpoint configured in the shared session.

Discovered this bug while setting up IRSA on EKS (see here). The client makes a sts/AssumeRoleWithWebIdentity request, but this fails with 404 on the secrets-manager endpoint. After removing the endpoint option, the SDK makes a request to sts.us-east-1.amazonaws.com and everything works.

Expected behavior

The endpoint config option should be optional, or set on the secrets-manager client instead:

https://github.com/minio/kes/blob/2e4e7be2b9da884cdc98d762849c2566f7b9018e/internal/keystore/aws/secrets-manager.go#L91

Additional context

Maybe the region option should be optional too. A container running on EKS with IRSA has the following environment variables:

AWS_STS_REGIONAL_ENDPOINTS:    regional
AWS_DEFAULT_REGION:            us-east-1
AWS_REGION:                    us-east-1
AWS_ROLE_ARN:                  arn:aws:iam::...                                  
AWS_WEB_IDENTITY_TOKEN_FILE:   /var/run/secrets/eks.amazonaws.com/serviceaccount/token

[derSascha]

@aead any chance to look into this? The fix in the pull-request is rather small. It's more or less impossible to run KES on AWS EKS securely without the changes.