redash
redash copied to clipboard
getredash
Athena: Make s3_staging_dir an option
What type of PR is this? (check all applicable)
- [x] Feature
Description
If you use a workgroup with output_location set, you can omit the s3_staging_dir setting.
The following is a quote from the boto3 document.
To run the query, you must specify the query results location using one of the ways: either for individual queries using either this setting (client-side), or in the workgroup, using WorkGroupConfiguration .
Related Tickets & Documents
Mobile & Desktop Screenshots/Recordings (if there are UI changes)
We tested that a test connection succeeds even when s3_staging_dir is empty.
Hey @kanga333 thanks for this effort. Sorry for missing your pings on this. Feel free to reopen if you still believe this would be valuable for the Athena query runner. We can get it reviewed and merged pretty quick using our new process. The only question I have on a quick look here is whether this will break any existing Athena connections (it doesn't seem like it would).
@susodapop Thank you for your reply. This change will not affect existing Athena connections. If s3_staging_dir exists, Athena Runner will continue to use it, and PyAthena will also accept it. This change only makes s3_staging_dir optional.
@kanga333 , thanks for the PR! We've updated a lot of things now that we're Community-driven so - if you're still interested in getting this merged - would you mind rebasing off master to re-run the CI, as well as updating merge conflicts?
We're trying to clean up our PR todo list, so if you're not interested, that's fine - we'll close the PR in about a week if we don't hear back. If you're interested in reopening the PR afterwards, we would also very much welcome that.
This looks like a really simple change, which should be pretty straight forward to fix the merge conflict for.
@kanga333 Any idea if this would still be useful to merge? :smile:
--
On a separate note, it looks like pyathena has several major releases in the meantime. We should probably look at the newer versions once we have things stabilised. :smile:
https://pypi.org/project/pyathena/#history
