Support 'external_id' for temporary credentials

[ad-m-ss]

trafficstars

Driver version

v2.0.907 Latest

Redshift version

N/A

Client Operating System

N/A

Python version

N/A

Table schema

N/A

Problem description

1. Expected behaviour: Library is able to use EKS IRSA / AWS SSO to discovery credential, then use AssumeRole with role_arn and external_id to receive current one.
2. Actual behaviour: No support for external_id.
3. Error message/stack trace: N/A
4. Any other details that can be helpful:

Partner Hosted Foundational Technical Review requires "CAA-002 - Use external ID with cross-account roles to access customer accounts."

Credential management is already supported. In my opinion, AWS SDK (in this scenario boto3) defines the credentials standard for IAM that is adopted for language in AWS. In my opinion, this library should only receive instance boto3.Session to build proper IAM credentials on its own. It allows use aws-assume-role-lib ( https://github.com/benkehoe/aws-assume-role-lib ) to automatically refresh credentials when reconnection happens (The fact that it is an external library is a separate problem in boto3 because JavaScript / Java ( https://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/auth/STSAssumeRoleSessionCredentialsProvider.html ) has the appropriate built-in credential provider.). Then a large amount of code - repeated in relation to boto3 - regarding the use of credential management will become redundant.

Python Driver trace logs

Reproduction code

import redshift_connector

# Connects to Redshift cluster using AWS credentials
conn = redshift_connector.connect(
    host='examplecluster.abc123xyz789.us-west-1.redshift.amazonaws.com',
    database='dev',
    user='awsuser',
    role_arn='aws:...',
    external_id='...'
 )

CC: @podpio

[Brooke-white]

Hey @ad-m-ss

Thanks for providing this feedback. While I need to discuss with the Redshift driver team, I think he addition of external_id connection parameter is reasonable to allow users access to underlying boto3 functionality.

In my opinion, this library should only receive instance boto3.Session to build proper IAM credentials on its own.

I'll need to discuss this feedback further with the team, as this change would not be inline with how our JDBC driver handles IAM. We generally try to keep these inline with each other to reduce the complexity for customers working with more than one of our drivers, but there are differences that come up such as the Java SDK having the sts assume role credentials provider built in.

[Brooke-white]

Hi @ad-m-ss ,

If you authenticate with AWS profile, see the section on IAM in our tutorial, the external_id as well as role_arn can be specified within the profile. Please see boto3 docs on configuration profiles for more information.

[ad-m-ss]

@Brooke-white the use of profiles that must be saved to disk as a file is not an effective solution to the issue. This interferes with multi-threaded access to multiple AWS accounts (of multiple customers in a multi-tenant environment) via multiple AWS roles.

I work with an AWS partner (ISV) where we need to access a consumer AWS account. We use AWS account access to access the consumer's Redshift cluster. We provide a partner-hosted solution, so we are SaaS.

In our case, through our API, the consumer provides information about the role ARN that we need to use for the connection to their AWS account. We need to use an external ID for cross-account access. Information about the ARN role and external ID is stored in our internal database of consumers, so we do not save credentials to files on the disk, nor create any SDK profiles, because we rely on the API and database.

The current workaround we are using is to call AWS.STS.AssumeRole manually and then pass them to the driver. Fortunately, validation of credentials is only done once per connection creation, so we don't need to handle temporary credential renewal.