pydantic
Add AWS SSM Parameter Store support
In #385, I added a code change to natively support retrieving configuration from the AWS Systems Manager Parameter Store. However, that was somewhat premature as support for such a feature had not yet been agreed upon.
This issue is to track support for that feature.
Background
From the product page:
AWS Systems Manager provides a centralized store to manage your configuration data, whether plain-text data such as database strings or secrets such as passwords. This allows you to separate your secrets and configuration data from your code. Parameters can be tagged and organized into hierarchies, helping you manage parameters more easily. For example, you can use the same parameter name, "db-string", with a different hierarchical path, "dev/db-string” or “prod/db-string", to store different values. Systems Manager is integrated with AWS Key Management Service (KMS), allowing you to automatically encrypt the data you store. You can also control user and resource access to parameters using AWS Identity and Access Management (IAM). Parameters can be referenced through other AWS services, such as Amazon ECS, AWS Lambda, and AWS CloudFormation.
It is similar to AWS Secrets Manager, with a few differences:
- support for non-secret data (i.e. can make public parameters)
- free storage & retrieval of parameters (for standard parameters)
- makes greater use of a hierarchy of parameters which fits in well with nested parameters within Pydantic Settings (e.g.
/app/prod/db/usernameand/app/prod/db/passwordcould mape well todb: DbSettings)
Fit within Pydantic Settings
Given that we currently support Azure Key Vault settings source (https://github.com/pydantic/pydantic-settings/pull/272) and have green-lighted support for AWS Secrets Manager (https://github.com/pydantic/pydantic-settings/issues/175#issuecomment-1750167497), I think rounding out support with AWS SSM Parameter Store seems like a reasonable addition.
Will people use it?
I maintain a project to support for AWS SSM Parameter Store for Pydantic v1 (github, pypi). At time of writing, it has 19 stars on Github and PyPi stats reports the usage as such:
Downloads last day: 56 Downloads last week: 178 Downloads last month: 653
Not tremendously popular, but it is used by projects in the wild.
seems @ceb10n has also worked on an extension for this recently, supporting pydantic>=2.0.1: pydantic-settings-aws
blog post (with some use cases) docs github pypi
pydantic-settings-aws shows similar stats to @alukach's project:
Downloads last day: 44 Downloads last week: 496 Downloads last month: 1,168
Parameter Store and Secrets Manager are great, and we use then a lot at work. For projects that use AWS, they are a great fit.
But although I developed the pydantic-settings-aws sometime ago, i don't see the necessity to add an optional dependecy of boto3 in the pydantic-settings project.
If someday people start using these features a lot, maybe it will make more sense to add it, but before that, I agree that it would bring some unnecessary complexity to the code base.
But if the project decides to add it the future, I'll be glad to help ;)
We added AWS secret manager support. The PR is merged but still hasn't been released.
We added AWS secret manager support. The PR is merged but still hasn't been released.
@hramezani What does that mean for this issue? #385 could be reopened if we want to additionally support AWS Parameter Store (I'm curious how AWS Secret Manager got buy-in).
We added AWS secret manager support. The PR is merged but still hasn't been released.
@hramezani What does that mean for this issue? #385 could be reopened if we want to additionally support AWS Parameter Store (I'm curious how AWS Secret Manager got buy-in).
@alukach there were some requests for AWS secret manager and it seems people are using it a lot. we haven't received a request for AWS Parameter Store after your issue. If you are still interested, please rebase your PR so we can consider it again.
@alukach If you are still interested, please reopen and refresh your PR
I've moved on from this need. My implementation is available, I'll leave it in your very capable hands to take it from here. :rocket:
@alukach If you are still interested, please reopen and refresh your PR
I've moved on from this need. My implementation is available, I'll leave it in your very capable hands to take it from here. 🚀
Thanks for letting me know. Unfortunately, I am busy at this moment. Let's keep the issue open. Maybe someone else will take care of it at some point.