FileIO S3: Add support for Assume-Role-Arn and other AWS Client properties

[c-thiel]

Currently FileIO respects:

• s3.endpoint
• s3.access-key-id
• s3.secret-access-key
• s3.region

It would be great to also support additional client attributes which help with cross-account setups:

• client.assume-role.arn
• client.assume-role.external-id
• client.assume-role.region
• client.assume-role.session-name
• client.assume-role.tags.
• client.assume-role.timeout-sec

Java reference: https://iceberg.apache.org/javadoc/1.4.1/constant-values.html (AWSProperties section)

[Xuanwo]

Thank you for posting this. I have compared this list with the features supported by OpenDAL, and here are my findings.

• client.assume-role.arn: we can use role_arn
• client.assume-role.external-id: we can use external_id
• client.assume-role.region: not supported yet, should this usually be the same region with bucket?
• client.assume-role.session-name: created https://github.com/apache/opendal/issues/4974
• client.assume-role.tags: created https://github.com/apache/opendal/issues/4975
• client.assume-role.timeout-sec: created https://github.com/apache/opendal/issues/4976

[charlesdong1991]

Hi, I am new to the project, if nobody yet picks it up, can I give it a try to get to know the code base better?

[liurenjie1024]

Hi, I am new to the project, if nobody yet picks it up, can I give it a try to get to know the code base better?

Welcome to contribute!

[reneleonhardt]

@Xuanwo Which properties are still missing? https://iceberg.apache.org/javadoc/1.9.1/constant-values.html

org.apache.iceberg.aws.AwsClientProperties

modifier	field	value
protected	CLIENT_CREDENTIAL_PROVIDER_PREFIX	"client.credentials-provider."
public	CLIENT_CREDENTIALS_PROVIDER	"client.credentials-provider"
public	CLIENT_REGION	"client.region"
public	REFRESH_CREDENTIALS_ENABLED	"client.refresh-credentials-enabled"
public	REFRESH_CREDENTIALS_ENDPOINT	"client.refresh-credentials-endpoint"

org.apache.iceberg.aws.AwsProperties

modifier	field	value
public	CLIENT_ASSUME_ROLE_ARN	"client.assume-role.arn"
public	CLIENT_ASSUME_ROLE_EXTERNAL_ID	"client.assume-role.external-id"
public	CLIENT_ASSUME_ROLE_REGION	"client.assume-role.region"
public	CLIENT_ASSUME_ROLE_SESSION_NAME	"client.assume-role.session-name"
public	CLIENT_ASSUME_ROLE_TAGS_PREFIX	"client.assume-role.tags."
public	CLIENT_ASSUME_ROLE_TIMEOUT_SEC	"client.assume-role.timeout-sec"
public	CLIENT_ASSUME_ROLE_TIMEOUT_SEC_DEFAULT	3600
public	CLIENT_FACTORY	"client.factory"
public	DYNAMODB_ENDPOINT	"dynamodb.endpoint"
public	DYNAMODB_TABLE_NAME	"dynamodb.table-name"
public	DYNAMODB_TABLE_NAME_DEFAULT	"iceberg"
public	GLUE_ACCOUNT_ID	"glue.account-id"
public	GLUE_CATALOG_ENDPOINT	"glue.endpoint"
public	GLUE_CATALOG_ID	"glue.id"
public	GLUE_CATALOG_SKIP_ARCHIVE	"glue.skip-archive"
public	GLUE_CATALOG_SKIP_ARCHIVE_DEFAULT	true
public	GLUE_CATALOG_SKIP_NAME_VALIDATION	"glue.skip-name-validation"
public	GLUE_CATALOG_SKIP_NAME_VALIDATION_DEFAULT	false
public	GLUE_LAKEFORMATION_ENABLED	"glue.lakeformation-enabled"
public	GLUE_LAKEFORMATION_ENABLED_DEFAULT	false
public	LAKE_FORMATION_DB_NAME	"lakeformation.db-name"
public	LAKE_FORMATION_TABLE_NAME	"lakeformation.table-name"
public	REST_ACCESS_KEY_ID	"rest.access-key-id"
public	REST_SECRET_ACCESS_KEY	"rest.secret-access-key"
public	REST_SESSION_TOKEN	"rest.session-token"
public	REST_SIGNER_REGION	"rest.signing-region"
public	REST_SIGNING_NAME	"rest.signing-name"
public	REST_SIGNING_NAME_DEFAULT	"execute-api"