make endpoints configurable

[PG2000]

Hello friends,

currently the minio-go client uses dualstack urls as default.

https://github.com/minio/minio-go/blob/f5f1604f764e0543943961efefac0e6dc56ee4b2/s3-endpoints.go#L36

Is that on purpose? Is it possible to disable this? I didn't found a possiblility to get rid of the "dualstack" host part.

[harshavardhana]

Why should we not use dualstack URLs? We have been using dualstack for the last 4-5yrs what is the benefit on removing it?

[PG2000]

Why should we not use dualstack URLs? We have been using dualstack for the last 4-5yrs what is the benefit on removing it?

Yeah, the title was mnisleading. We're working in a fully private environment and don't wan to have to many private endpoints. As we already have a private endpoint for s3.eu-central.amazonaws.com we now need a second one. with s3.dualstack.eu-central.amazonaws.com

We hoped to not rely on the dualstack endpoints

[PG2000]

Why should we not use dualstack URLs? We have been using dualstack for the last 4-5yrs what is the benefit on removing it?

Yeah, the title was mnisleading. We're working in a fully private environment and don't wan to have to many private endpoints. As we already have a private endpoint for s3.eu-central.amazonaws.com we now need a second one. with s3.dualstack.eu-central.amazonaws.com

We hoped to not rely on the dualstack endpoints

Additionally it is not easy to have a private endpoint for the s3 dualstack urls as you are forced to enable ip v6

[PG2000]

@harshavardhana I would like to open a PR for this and would like to ask you for a direction.

Ideas:

• have a flag like "disableDualstack" - to be backward compatible. If set then we would use the url which is configured for the bucket

[mss]

@harshavardhana if I may chime in with a use case where enforcing the dual stack DNS names is not a good idea or at least confusing: We are deploying Graphana Mimir (which uses this library for the S3 backend) in an enterprise environment. One of the policies (enforced by some central firewalling and/our routing) is that we can't connect to any public IPv4 addresses (IPv6 is completely unsupported) but must stay within the RFC1918 IP range. This includes the public AWS endpoints.

To be able to use them we deploy VPC Interface Endpoints and enable the Private DNS option which will override s3.region.amazonaws.com in the local resolver and point it to the local endpoint. But S3 VPC Endpoints do not support dualstack yet so the DNS name s3.dualstack.region.amazonaws.com is not overridden. Which means that our application still tries to connect to the public AWS IP addresses and fails.

This behaviour is not documented on the Graphana side, maybe because nobody noticed it yet and didn't even expect it (I shall raise a documentation issue there). I understand why the default to the dualstack DNS names may make sense in most cases and I think the proper fix for this would be an option similar to what @PG2000 suggested though I would call it something like disableHostnameMapping or the like; ie. when this option is set the library should just use the DNS name I specified, dualstack or not. It is then up to me to choose the proper regional name (as Mimir does already).

BTW, after digging through the code I found that the Local and Zonal names won't be changes (as long as they start with bucket. or accesspoint., no idea why that is checked) so for now our solution is to use the ugly Local Names which makes our IaC a quite more complex though because we now have to lookup that name for the current account and pass it on to our Mimir deployment.

[harshavardhana]

@harshavardhana I would like to open a PR for this and would like to ask you for a direction.

Ideas:

• have a flag like "disableDualstack" - to be backward compatible. If set then we would use the url which is configured for the bucket

Sure please send a PR