Host cannot be resolved when port is specified in endpointoverride for S3 CRT client

[MarcusHellqvist]

Describe the bug

Put/GetObjectAsync calls aws_s3_client_make_meta_request() which eventually ends up in a call to getaddrinfo(). hostname_cstr containes both the host and port and cannot be resolved.

https://github.com/awslabs/aws-c-io/blob/e538d31d458e25918eed892a998c697386f7800e/source/posix/host_resolver.c#L43

Expected behavior

That and endpoint with host:port can be resolved.

Current behavior

Relevant part of tracing:

dns: static: getaddrinfo failed with error_code -2 dns: static, resolving host 172.17.0.2:4566 failed, ec 1059 (aws-c-io: AWS_IO_DNS_INVALID_NAME, Host name was invalid for dns resolution.) S3MetaRequest: id=0x7fcb6013ce20 Meta request finished with error code 9229 (Invalid S3 endpoint) S3CrtClient: S3CrtClient received error response for s3_meta_request with response code: 0, with error_code: 9229

Steps to Reproduce

Set endpoint override in client configuration to contain a port.

Possible Solution

No response

AWS CPP SDK version used

1.9

Compiler and Version used

clang --10

Operating System and version

Ubuntu

[KaibaLopez]

Hi @MarcusHellqvist , I mean... you're right but it also kind of feels like a feature request rather than a bug, don't know if you mind it being classified like that but let me know and we'll see about making this work.

[MarcusHellqvist]

I would personally classify it as a bug but it's up to you. This works with the S3Client but not S3 crt client.

[lapointexavier]

I've also run into this when upgrading the sdk against a local setup using localstack. A work around was to bind localstack's port to :80 (default), but it would be great to improve this to allow specifying a different port.

[fboranek]

It is regression, in fact. I have test running against mock implementation running on localhost. Now I change S3 Client to Crt version and it stop working since the mock server run on custom port.

Basically just:

-    std::shared_ptr<Aws::S3::S3Client> s3Client;
+    std::shared_ptr<Aws::S3Crt::S3CrtClient> s3Client;
-    Aws::Client::ClientConfiguration configClient;
+    Aws::S3Crt::ClientConfiguration configClient;
-    Aws::S3::Model::PutObjectRequest request;
+    Aws::S3Crt::Model::PutObjectRequest request;

and using

    Aws::S3Crt::ClientConfiguration configClient;
    configClient.endpointOverride = "http://localhost:9888";

other opertaion prior to PutObjectRequest works fine. E.g.

• Aws::S3Crt::Model::HeadBucketRequest
• Aws::S3Crt::Model::CreateBucketRequest
• Aws::S3Crt::Model::PutBucketPolicyRequest
• Aws::S3Crt::Model::ListMultipartUploadsRequest

so it seems only problem in PutObjectRequest and I guess in GetObjectRequest since these methods are supposed to be improved.

[appu226]

Bug or feature, it’s definitely the most upvoted issue

[pnorbert]

Bug or feature, it’s definitely the most upvoted issue

I don't know how to upvote, but I just did run into this issue trying to run S3CrtClient against a local Minio server. Is there a workaround to make this work if this issue is not going to be solved?

[fboranek]

Bug or feature, it’s definitely the most upvoted issue

I don't know how to upvote, but I just did run into this issue trying to run S3CrtClient against a local Minio server. Is there a workaround to make this work if this issue is not going to be solved?

Build with patch or find someone to merge it into repo. Sadly, there is no other way.

[pnorbert]

Well, I can't do that for a dependency. So I will wait moving to S3Crt until this if fixed or until I find a workaround. Thanks.

[appu226]

Patch worked for us.

[TingDaoK]

For CRT S3, we do support endpoint override for request level, https://github.com/awslabs/aws-c-s3/blob/main/include/aws/s3/s3_client.h#L395-L401

The issue here is C++ SDK needs to store the client level endpoint override somewhere and feed it to the aws_s3_meta_request_options from here.

It should not be hard to add.

For more info: we had similar request from java side, here. And we fixed it for java. So, for C++, we just need to hook up the interface.

[koendelaat]

We also faced this issue. I've created a PR #2352 to address the changes suggested by @TingDaoK

[jmklix]

PR merged, closing this. Please let me know if you are still having problems when specifying the port with endpointoverride

[github-actions[bot]]

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see. If you need more assistance, please either tag a team member or open a new issue that references this one. If you wish to keep having a conversation with other community members under this issue feel free to do so.