Support Signature Version 4 only regions

[airizarryDB]

For S3 regions that only support Signature Version 4, a protocol for authenticating inbound API requests to AWS services, the default S3 service endpoint will not work, the S3A client needs to be given the endpoint to use via the fs.s3a.endpoint property. Any new Regions after January 30, 2014 will support only Signature Version 4 and therefore all requests to those Regions must be made with Signature Version 4. Older regions will still support Version 2, and are the only regions where Delta Sharing Server works. Example regions, that do not work because of only supporting Signature Version 4 include (S3 Frankfurt, Seoul, GovCloud); Example given below on how to fix this issue:

conf.set("fs.s3a.endpoint", "s3.us-gov-west-1.amazonaws.com")

We shoved that in server/src/main/scala/io/delta/standalone/internal/DeltaSharedTableLoader.scala and got it to work, but we had a problem validating tests when trying to contribute that as a PR, so opening an issue instead. Also, not sure if that's the best file location for that config, but it will solve the issue.

[zsxwing]

The doc in https://github.com/delta-io/delta-sharing#azure-blob-storage shows how to set a Hadoop configuration. Could you try to use the same approach to set fs.s3a.endpoint. It's talking about how to set Azure credentials but credentials are just Hadoop configurations.

[zsxwing]

Closing this. Setting this in a hadoop config file should work.