S3FileIO does not support Iceberg Cross-Region API Calls to Amazon S3 buckets

[elmehdibelgasmi]

Apache Iceberg version

None

Query engine

None

Please describe the bug 🐞

Issue Description

With S3FileIO implementation, Cross-Region API calls to S3 buckets (Iceberg read/write operations) fail with the errors below. With HadoopFileIO implementation, the same Iceberg table operations are successful.

Error 1:

software.amazon.awssdk.services.s3.model.S3Exception: The authorization header is malformed; the region 'us-east-1' is wrong; expecting 'eu-west-1' (Service: S3, Status Code: 400, Request ID: ..., Extended Request ID: ...)

Error 2:

software.amazon.awssdk.services.s3.model.S3Exception: The bucket you are attempting to access must be addressed using the specified endpoint. Please send all future requests to this endpoint. (Service: S3, Status Code: 301, Request ID: ..., Extended Request ID: ...)

Observations:

• From the S3FileIO source code, it appears that the S3ClientBuilder's crossRegionAccessEnabled property is not present.
• From Hadoop's S3A source code, crossRegionAccessEnabled has been implemented.

Note: This property has been added since AWS SDK version >= 2.20.111 (doc).

Suggestion:

Add Cross-Region bucket access support for S3FileIO (i.e. as a new Iceberg configuration property).

[nastra]

@elmehdibelgasmi would you be interested in opening up a PR and fixing this for S3FileIO?

[elmehdibelgasmi]

@elmehdibelgasmi would you be interested in opening up a PR and fixing this for S3FileIO?

@nastra sure, will do.