apache
[SIP-164] Support fetching configuration from External Source
[SIP] Proposal for fetching configuration from external source
Motivation
We would like to run Apache Superset on ECS Fargate (for example) and not needing to customize the container itself, just run the Docker Hub official Image.
This also applies to other Cloud Providers and Parameter-Store like providers (i.e.: Google, Vault, Nomad, Consul, etc...)
Proposed Change
The proposed change is to have a ConfigurationFetcher Contract set via Environment Variable, i.e.: CONFIGURATION_FROM=awsparameterstore where it defaults to the current behaviour (Check superset_config.yml on FileSystem) and CONFIGURATION_LOCATION=<aws:arn>
New or Changed Public Interfaces
No changes to existing endpoints, just the CLI where we can specify where we want to fetch the configuration from.
New dependencies
- boto3
- botocore
Migration Plan and Compatibility
No need for a migration plan, given that the default behavior will remain the same
Rejected Alternatives
N/A
Thank you for the proposal! It's been numbered - let me/us know if you have questions about moving this through the SIP process.
I don't see a DISCUSS thread on the dev@ mailing list. You'll need to do that, or this may eventually be closed as abandoned.
To open a [DISCUSS] thread on the Apache Superset Dev mailing list, you first have to be subscribed to the Dev mailing list. You can do that by sending a blank email to [email protected].
Once you've received confirmation that you've subscribed, you can email the listserv. Send an email to [email protected] with a title like "[DISCUSS][SIP-XXX] My Sip Title" and include a brief synopsis about your SIP and a link to this GitHub thread.
Once discussion has settled out, one of the Committers/PMC members may decide to open up a [VOTE] thread as the next step.
Curious if @mistercrunch or @dpgaspar have alternate ideas here, this seems like something they may have dealt with in the past. If people are into this, I can help nudge it along with a DISCUSS thread.
Oh! this is super-related to this -> https://github.com/apache/superset/pull/34210
Out of the box my plan is to have much better support for env vars (for everything that's static - ie not a python method or object, but supporting inline typed json inline in env vars), and to allow for managing some configs (you define which ones by configuration) in the metadata database itself.
As I work on this, I'll think about how we could have ways to fetch configs from other places.
Anyone still planning to take this on? If so, please start a DISCUSS thread, otherwise it may soon be closed as abandoned.