airflow
airflow copied to clipboard
apache
The AWS Secrets Backends (SM and SSM) do not allow configuration of Assume Role Methods via Backend Kwargs
Description
My current understanding, is that the methods documented here outline the ways we can authorize AWS Secrets Manager or AWS Systems Manager as a Secrets Backend. This requires managing authorization via Environment Variables or Profiles defined in an .aws/config file.
This does not include the Assume Role methods and configuration framework offered by the AWSBaseHook. I am proposing we extend the Secrets Backend integrations, so that they can be configured in a similar manner.
Use case/motivation
This would enable users to have a consistent authorization and configuration framework (with the Hooks/Operators) when connecting to AWS Secrets Backends. This could also reduce the number of cases where users need to deploy Access & Secret Keys to their Airflow environment.
For example, users could pass a Role in the BACKEND_KWARGS for the Worker or Scheduler (once authorized) to assume to connect to the service powering their Secrets Backend.
Related issues
No response
Are you willing to submit a PR?
- [X] Yes I am willing to submit a PR!
Code of Conduct
- [X] I agree to follow this project's Code of Conduct
Feel free . Also @o-nikolas and @ferruzzi and @vincbeck might want to chime-in
This does not include the Assume Role methods and configuration framework offered by the AWSBaseHook. I am proposing we extend the Secrets Backend integrations, so that they can be configured in a similar manner.
Right now almost everything related to boto3.session.Session in Hooks managed by airflow.providers.amazon.aws.hooks.base_aws.BaseSessionFactory or Custom Session Factory. The downside of this - Factory expect Airflow Connection, and as workaround might be possible to create Fake Airflow Connection and use it for obtain credentials.
@potiuk - I appreciate the ping. I'm not familiar enough with the Secrets Backend yet to have any useful opinion on this one. At face value, it sounds like a useful addition, but I can't predict any unintended side-effects at this time. :+1:
