Feature request: Support for retrieving batch of secrets

[heitorlessa]

Use case

Have the same experience as SSM but Secrets Manager now that it's possible.

It was quietly launched recently: https://aws.amazon.com/about-aws/whats-new/2023/11/aws-secrets-manager-batch-retrieval-secrets/

Solution/User Experience

Same experience as get_parameters_by_name, as each secret might have a different TTL associated with it: https://docs.powertools.aws.dev/lambda/python/latest/utilities/parameters/#getting_started_parameter_by_namepy

Alternative solutions

No response

Acknowledgment

• [X] This feature request meets Powertools for AWS Lambda (Python) Tenets
• [X] Should this be considered in other Powertools for AWS Lambda languages? i.e. Java, TypeScript, and .NET

[heitorlessa]

FYI @aws-powertools/lambda-dotnet-core @aws-powertools/lambda-typescript-core @aws-powertools/lambda-java-core

[leandrodamascena]

Thanks for opening this @heitorlessa. We will try to add this in v3.

[leandrodamascena]

Hey @heitorlessa, @hjgraca, @am29d and @dreamorosi, I have a few questions here and would like to discuss the best customer experience with you. This will make it easier to implement in other runtimes.

Function parameters - Name or ARN?

After examining the implementations in Python, JavaScript, and .NET AWS SDK, I found that they all implement two parameters in batch_get_secret_value method: SecretIdList, which is a list of ARNs, and Filters, where you can search for secrets based on various attributes, including the name. However, in my experience, customers rarely use ARNs; instead, they tend to utilize the name parameter to retrieve the secret value. That said, I thought in an experience like this:

def _get_multiple(self, names: List[str], **sdk_options) -> Dict[str, str]:
    if not names:
        raise GetSecretError("You must inform at least one name.")

    sdk_options["Filters"] = [{"Key": "name", "Values": names}]

    secrets_result = boto3.client("secrets").batch_get_secret_value(**sdk_options)

Do we iterate and return all values, or do customers handle pagination on their end?

By default, this operation does not automatically allow to paginate results using SDK. This means that when you call this operation for the first time, if the number of items in AWS Secrets exceeds the limit defined by MaxResults, the response will include a NextToken. To retrieve the remaining results, you need to make subsequent calls and pass the NextToken returned from the previous response. My question is: Do we provide the NextToken for customers to handle pagination themselves, or do we automatically iterate and return the complete set of values in a single response? In my opinion, we should iterate and return everything in a single response, this way we are facilitating the customer experience. We already do the same with the get_parameters (SSM).

For now, these are the two questions that are blocking the start of work here.

Thanks

[dreamorosi]

Hey thanks for the thoughtful proposal and for tagging us.

Regarding the first point, I agree with you, most people won't use the ARN so I agree with your idea of using filters.

I would however propose to merge the Filters field by overwriting the Names key instead of replacing it entirely. In JS it would look like this:

const filters = [...sdk_options.Filters, {"Key": "name", "Values": names}]

// use `filters` in the next line

This way customers can have an escape hatch to pass additional filters if the name is too ambiguous, for example:

secrets.get_multiple(names=["foo/bar", "foo/baz"], sdk_options=[{"Key": "primary-region", "Values": "eu-west-1"}])

Which would then get merged to:

const filters = [{"Key": "primary-region", "Values": "eu-west-1"}, {"Key": "name", "Values": names}]

--

For the second point, I agree with you and we should align with what we do already for the SSMProvider and fetch everything before returning.

[leandrodamascena]

I would however propose to merge the Filters field by overwriting the Names key instead of replacing it entirely. In JS it would look like this:

Great point @dreamorosi! This way the customer can search using the name as a filter and any other field they want.

Thanks for the contribution.. I'll start working on this today.

[leandrodamascena]

Do not forget when creating documentation:

When using this approach of concatenating filters (names + sdk_options), it's important to make clear that the search is performed using the AND operator, not the OR operator. This means that the search results will only include items that match both the name and the sdk_option, not items that match either the name or the sdk_option.

[leandrodamascena]

Removing this from the initial V3 release, but planned to add after the official release.