Add AWS SettingsManager support

[rishacha]

Initial Checks

• [x] I have searched Google & GitHub for similar requests and couldn't find anything
• [x] I have read and followed the docs and still think this feature is missing

Description

Can we add support for AWS SecretManager as a new pydantic source? I have an idea of how to implement a custom source by looking at sources.py

Affected Components

• [ ] Compatibility between releases
• [x] Data validation/parsing
• [ ] Data serialization - .model_dump() and .model_dump_json()
• [ ] JSON Schema
• [ ] Dataclasses
• [ ] Model Config
• [ ] Field Types - adding or changing a particular data type
• [ ] Function validation decorator
• [ ] Generic Models
• [ ] Other Model behaviour - model_construct(), pickling, private attributes, ORM mode
• [ ] Plugins and integration with other tools - mypy, FastAPI, python-devtools, Hypothesis, VS Code, PyCharm, etc.

[samuelcolvin]

Sounds good to me 👍 PR welcome.

It should be fairly easy to copy other sources.

Is there good documentation on the AWS side?

[rishacha]

There are 2 ways to retrieve secrets for python.

1. Using the AWS SDK - i.e. boto3. So I will refer to the simpler boto3 client and this for the implementation.

2. Using an LRU Cache based package - aws-secretsmanager-caching From their documentation -

When you retrieve a secret, you can use the Secrets Manager Python-based caching component to cache it for future use. Retrieving a cached secret is faster than retrieving it from Secrets Manager. Because there is a cost for calling Secrets Manager APIs, using a cache can reduce your costs. For all of the ways you can retrieve secrets, see Retrieve secrets.

The cache policy is Least Recently Used (LRU), so when the cache must discard a secret, it discards the least recently used secret. By default, the cache refreshes secrets every hour. You can configure how often the secret is refreshed in the cache, and you can hook into the secret retrieval to add more functionality.

The cache does not force garbage collection once cache references are freed. The cache implementation does not include cache invalidation. The cache implementation is focused around the cache itself, and is not security hardened or focused. If you require additional security such as encrypting items in the cache, use the interfaces and abstract methods provided.

Here, pydantic-settings would store it in memory instead of cache using the settings object which is an instantiation of the Settings() class however, I had a doubt regarding this

So should I use the AWS SDK to connect with the secret manager client or assume this cache-based method?

@samuelcolvin - Need some clarity on the approach. Thanks.