[Feature Request] Capability to specify the aws region while creating the snapshot repository

[navneet1v]

Is your feature request related to a problem? Please describe

I was trying to take a manual snapshot of my cluster which was in us-west-2 region and my bucket was also in the same region. But I kept on getting error because the default region provided which s3 client takes in us-east-1 and even when my machine was in us-west-2 it was not picking up that region.

"error":{"root_cause":[{"type":"repository_verification_exception","reason":"[test-navneet-us-west-2-repository] path [test-navneet-us-west-2/10m-cohere-217-snaphot] is not accessible on cluster-manager node"}],"type":"repository_verification_exception","reason":"[test-navneet-us-west-2-repository] path [test-navneet-us-west-2/10m-cohere-217-snaphot] is not accessible on cluster-manager node","caused_by":{"type":"i_o_exception","reason":"Unable to upload object [test-navneet-us-west-2/10m-cohere-217-snaphot/tests-QivGuhDBSAKfvsayxo2NLQ/master.dat] using a single upload","caused_by":{"type":"s3_exception","reason":"s3_exception: Please re-send this request to the specified temporary endpoint. Continue to use the original request endpoint for future requests. (Service: S3, Status Code: 307, Request ID:XXXXX, Extended Request ID: ZXXXX)"}}},"status":500}

Describe the solution you'd like

A solution I would like to see is one of the below:

1. Either Opensearch should pick the region from the machine region
2. Opensearch should start taking the region while setting up the snapshot repository.

Related component

Storage:Snapshots

Describe alternatives you've considered

The only alternative I had was the update the opensearch.yml file and restart the cluster. Restarting of a cluster might is not a right choice.

https://opensearch.org/docs/latest/tuning-your-cluster/availability-and-recovery/snapshots/snapshot-restore/#amazon-s3:~:text=(Optional)%20Add%20other%20settings%20to%20opensearch.yml%3A

Additional context

No response

[mgodwan]

@navneet1v Does setting the following help in getting rid of the error?

https://github.com/opensearch-project/OpenSearch/blob/031d8ae022e47aedc8563bb7735aa582c8145bde/plugins/repository-s3/src/main/java/org/opensearch/repositories/s3/S3ClientSettings.java#L253-L258

[navneet1v]

@mgodwan I ended up using the setting s3.client.default.region but this setting was not dynamic. I see what you provided as a setting is s3.client.region am I correct? and also can you tell me if s3.client.region is a dynamic setting? and also where this setting is provided in documentation.

[gbbafna]

[Storage Triage] - Attendees 1 2 3 4

@navneet1v Can you please try putting region - us-west-2 in the repository settings ? Currently repository doesn't read cluster settings dynamically .

[bowenlan-amzn]

Apart from s3.client.default.region, I also need to update s3.client.default.endpoint from s3.amazonaws.com to s3.us-west-2.amazonaws.com, because my s3 bucket was created less than 24h. It would be better we can dynamically detect the ec2 region and set this in the code https://repost.aws/knowledge-center/s3-http-307-response

[ICHx]

I am coming across the same issue upgrading from Opensearch 1.x to 2.x.

I noticed there is an issue being related to this https://github.com/opensearch-project/OpenSearch/issues/9265

So this might mean it is a regression from 2.9