Allow IAM Role-based authentication for ALL AWS API connectors

[superwesman]

• All Redash AWS Connectors should allow user to provide a Role which Redash should assume.
• Redash AWS Connectors should all be consistent with respect to credentials.

Currently the Redash AWS connectors are difficult to use because, in many cases, we're forced to use IAM User key's. This is difficult to manage due to our company's policies regarding IAM User key rotation.

Furthermore, the available authentication options are inconsistent across the connectors.

• Amazon Athena connector requires "IAM Role to assume". This is perfect. Good job, whoever did that. ✊🏼
• Amazon Cloudwatch connector requires permanent IAM User keys.
• Amazon CloudwatchLogs connector requires permanent IAM User keys.
• Amazon Elasticsearch connector has an option for permanent IAM User keys, but can also use an AWS client Profile.
• DynamoDB connector requires permanent IAM User keys.

Assumed Roles should be the preferred mechanism. You should add "IAM Role to assume" like you have in the Athena connector and deprecate the Profile- and key-based options.

The Redshift connectors should probably be considered out of scope for this request, as they are "db" connectors and not "api" connectors.

And, before you ask, no I won't be contributing a pull request. 😜 Other corporate policy makes that needlessly difficult and this is a simple enough change.

[susodapop]

Thanks for calling attention to this. The diversity of authentication patterns is a consequence of having different authors at different times: these connectors were all community contributions.

We'll certainly accept a pull request that unifies the approach. However given that we on the core team use none of these services it's not something we can test on our own.

I'm happy to make a pass at it. But to prioritise it, I need to hear from community members who can at least test an experimental change to each of these query runners before we merge it. Worst case is we break existing connections without notice during an upgrade.

[superwesman]

thanks for the prompt reply, Jesse! We've got no corporate policy preventing us from testing any experimental changes. 😎 We'd be glad to help, just let us know what you need.

Normalizing all of these will probably be helpful for future contributors/reviews, so that's another perk 🤙🏼

[susodapop]

Normalizing all of these will probably be helpful for future contributors/reviews

Strong agree. And that is a generous offer on your part.

I'm self-assigning this issue and will aim to begin research / implementation this month. I'm drawn in a thousand directions at once these days, so don't hesitate to ping me to keep it on my radar.

[pygillier]

hey @susodapop do you already have something ?

I'm working on the cloudwatch runners to improve them and wouldn't reinvent the wheel if you already have something :)

[susodapop]

Thanks for the ping. I won't have bandwidth for this until September. Happy to consult if you'd like some input on your implementation, though!

[zachliu]

since the older versions of botocore (>=1.13,<1.14.0) in redash do not support AWS SSO, i couldn't test the Athena connector's IAM Role to assume locally with botocore being held at 1.13.50

the earliest botocore version that supports this is 1.17.0 released on Jun 9, 2020

anyone knows how to upgrade botocore in redash without breaking other libs/functions?

---------- update --------- i changed the botocore version constraint from >=1.13,<1.14.0 to >=1.13,<1.18.0 in both requirements_all_ds.txt and requirements_dev.txt. nothing seems to break so far :crossed_fingers: