trino icon indicating copy to clipboard operation
trino copied to clipboard

Make region in S3 native file system optional

Open mosabua opened this issue 9 months ago • 3 comments

In some S3-compatible object storage systems the concept of a region as in S3 does not exist. One such example is MinIO. Currently users are forced to provide a fake value for region with these systems.

The desire is to make the region parameter optional. This could be done on the generic S3 level or potentially looking at creating separate levels for each "s3-compatible" system.

mosabua avatar May 01 '24 20:05 mosabua

Assigning to @electrum as the file system lead.

mosabua avatar May 01 '24 20:05 mosabua

The SDK v2 used to require region, but it looks like they have a cross region feature now: https://docs.aws.amazon.com/sdk-for-java/latest/developer-guide/s3-cross-region.html

electrum avatar May 01 '24 22:05 electrum

But even if the SDK requires it .. we could shadow this in Trino and fill it in with some random value if minio is used, right?

Or we could just have a random and invalid default value. Then its essentially optional.. problem of course is that it is then quietly invalid for s3 users unless they fill it to correct the default.

mosabua avatar May 01 '24 22:05 mosabua

Do we want to close this @electrum ?

mosabua avatar Aug 29 '24 19:08 mosabua

Region is required by the AWS SDK as it is part of the authorization header. Since we don't know if the storage system will validate the region, my thought is we should keep it as a requirement.

electrum avatar Aug 29 '24 21:08 electrum