Redshift IAM support

[hsheth2]

We should be able take the user's IAM credentials and call redshift:GetClusterCredentials to get the credentials dynamically.

As per https://stackoverflow.com/questions/44243169/connect-to-redshift-using-python-using-iam-role, the AWS-native redshift_connector driver has support for this built in, but it's possible to do ourselves as well.

We're currently using the Redshift sqlalchemy dialect with the psycopg2 driver, so if we wanted to use the AWS-provided IAM stuff then we'd need to switch to the redshift_connector (which probably would be worthwhile anyways).

• Slack ref: https://datahubspace.slack.com/archives/CUMUWQU66/p1664523700921499?thread_ts=1664450288.929819&cid=CUMUWQU66
• https://github.com/sqlalchemy-redshift/sqlalchemy-redshift/blob/main/setup.py#L47

[github-actions[bot]]

This issue is stale because it has been open for 30 days with no activity. If you believe this is still an issue on the latest DataHub release please leave a comment with the version that you tested it with. If this is a question/discussion please head to https://slack.datahubproject.io. For feature requests please use https://feature-requests.datahubproject.io

[treff7es]

Our new Redshift connector fully relies on Redshift Client and if I'm not mistaken if you should be able to use IAM credential (except for profiling which still relies on SqlAlchemy) This is from the above StackOverflow post:

conn = redshift_connector.connect(
    iam=True,
    database='dev',
    db_user='<username>', # the database user in call to get-cluster-credentials
    cluster_identifier='my-redshift-cluster', # identifier of your cluster
    profile='redshift_profile' # profile in ~./aws/config with correct permissions
 )

and this should be set by:

  type: redshift
  config:
       extra_client_options:
          iam: true
          db_user: <username>
          cluster_identifier: my-redshift-cluster
          profile: redshift_profile

[hsheth2]

@treff7es that's pretty nifty - can you update the docs to show an example of that, and then we can close out this issue