Http Outbound request's header Host value comes with port number.

[seungjin]

Does outbound http request(with Rust)'s header HOST come with Port number?

Following request goes to AWS to upload S3 object.

let request = Request::builder()
        .method(Method::Put)
        .uri(target)
        .header("Authorization", authorization)
        .header("Content-Length", content_length)
        .header("Content-Type", content_type)
        .header("X-Amz-Content-Sha256", x_amz_content_sha256)
        .header("X-Amz-Date", x_amz_date)
        .body(file.to_vec())
        .build();

Got this from AWS:

content-length:70
content-type:image/png
host:seungjin.s3.amazonaws.com:443
x-amz-content-sha256:4d17e4d1ba1915f55196d1f9c135d2b6cc5cd3a4fd867b02f48e39bb5b37877e
x-amz-date:20240305T075204Z

Above saying host seungjin.s3.amazonaws.com:443 not seungjin.s3.amazonaws.com I actually trying to overwrite Host header value to test but Amazon is complaining it has a duplicated header.

1. Is that normal? Shipping Host value with port even it is well-known port(https, 443)?
2. Can I overwrite the Hearder (such as Host) ?

I checked with some 3rd party tool to catch what host name is. curl does not ship with port number but Spin's outbound request does. Also Rust X86 compiled binary using reqwest just ships hostname not with post number in Host header value. When i create a Mastodon's signing message few month ago, this host and port was a problem. - If hostname is used for signing http request, what I use for my hostname and server's received hostname are not matched(because port num) and signing can't be verified.

https://discord.com/channels/926888690310053918/1214479936396460063

# spin --version
spin 2.4.0-pre0 (8d67954d 2024-03-04)
suthep2~(:|✔)
# spin plugin list --installed
cloud 0.5.1 [installed]
platform 0.1.0 [installed]
pluginify 0.6.0 [installed]
py2wasm 0.3.2 [installed]

[lann]

I'm not sure where this behavior is coming from but in general it should be fine to include a default port number in an http host header.

If you manually set the host header on the request with just the hostname does it get replaced?

[seungjin]

If I force to add my own host header value with this: .header("Host", "seungjin.s3.amazonaws.com"), I got a following message from Amzaon. It is not overwriting but sending duplicated host header value:

<?xml version="1.0" encoding="UTF-8"?><Error><Code>DuplicateHeaderName</Code><Message>Your request contains duplicate headers.</Message>

[seungjin]

Okay. In this case, I am signing its request with the Host header value. Having a port or not is a BIG difference. The HTTP spec says it is okay to have it but also okay not to have it. But that's probably an old standard nobody needed to think about when signing a request with the hostname.

I sign the request with just the hostname (without the port), but AWS is using its host with the port because that's what AWS is getting. So the verification never happens. (Spin ships with a port, but curl and my Rust code with reqwest do not ship it.) To solve this for now (temporarily...), I also signed with the port number (forcefully adding {host}:{port}) and sent it to AWS. Of course, verification passed, and my image was up online with the trick.

The AWS API is a simpler case because my target is only one (AWS). But when I deal with Mastodon, it is not that simple. Some webservers (proxy, nginx, caddy, apache...) pass the port, and some don't. That's a configuration matter.

Anyway, I found a way I can work around it, but I don't think it is a permanent solution. Based on my search, Go/Java SDK developers have encountered similar cases.

I will get back and bash later when this issue becomes more critical. :-) Please don't close it until we clearly know what's going on. :-)

[lann]

Thanks for the update.

It is not overwriting but sending duplicated host header value:

This I think may be a bug, but will require some research to understand why its happening and how best to fix it.

[dicej]

@lann I suspect this is where the problem is: https://github.com/fermyon/spin/blob/7f55409a7b75c8dc52bc77fdaa5ba1e71a524b8d/crates/trigger-http/src/lib.rs#L493-L495. Should we instead check for an existing "host" header and only set one if none already exists?