Stop sending Host header to backend

[gfrankliu]

trafficstars

Is there a config option to tell mod_proxy_http2 not to sending Host header to backend/upstream, and only send http2 authority header?

In my test setup, apache virtualhost listens on port 8080, and h2c proxy to backend nginx on port 81.

ProxyPass "/" "h2c://backend:81/"

When I curl apache http://localhost:8080, nginx backend gets authority "localhost" and Host "localhost:8080" if apache ProxyPreserveHost is set to On. Otherwise authority "backend:81" and Host "localhost:8080".

In both cases, nginx returns http/400. https://trac.nginx.org/nginx/ticket/2268

[gfrankliu]

https://github.com/icing/mod_h2/issues/124 reported similar issue and was closed/fixed, but it seems not 100% fixed, when listening port is not standard port.

[icing]

Thanks, am working on test cases and a fix. I think mod_proxy_http2 should only forward :authority and no Host header, for one.

[gfrankliu]

Thanks @icing and look forward to the fix.

For now I am using http/1.1 to proxy but encountered an issue with connection reuse and dns. You can see the details in this post

Once you fix and stop sending Host header, I will switch to mod_proxy_http2 but will ProxyPassMatch support connection reuse in http2 with dollar substitution for backend host dns name?

[icing]

@gfrankliu could you give v2.0.7 a try?

[gfrankliu]

The change log for v2.0.7 only says connection idling fix, no mention of stop sending Host header. I will give it a try anyway today and report back.

[gfrankliu]

Tested and indeed v2.0.7 still sends Host header. Backend nginx returns 400.

[icing]

@gfrankliu the fix should be in release v2.0.8. My test case before did not correctly check the scenario. I enhanced that and then fixed the underlying issue.

[gfrankliu]

Just tried v2.0.8, indeed the http/400 error from backend nginx goes away. Thanks!

But I still see the same issues I observed in http/1.1 mod_proxy as mentioned above. The only fix is to remove connection reuse enablereuse flag, which increases the connections to backend from ~10 to ~80.

[icing]

If you could give a small config that reproduces the problems, I could work toward solving this.

[gfrankliu]

1. Having 2 VMs running as backends: host1.internal and host2.internal They simply return a static page with their hostname.
2. Front proxy server runs Apache on Debian 11 and listens 443/h2, with proxy config: ProxyPassMatch ^/(prod|dev)/([-a-z0-9]+)/(.*)$ h2://$2.internal/$1/$2/$3 enablereuse=on keepalive=on
3. Chrome browser connects http2 to the Apache. Request URI /prod/host1/something returns host1, and /prod/host2/something also returns host1 (though it should return host2).
4. Remove enablereuse=on in Apache proxy config and the request will be sent to host1 or host2 correctly.

[icing]

Thanks, will try to reproduce in a test.

[icing]

@gfrankliu I consulted with the other guys in the httpd team and it seems that connection reuse for such configurations is not implemented properly in general. The observer behaviour should also happen for mod_proxy_http. Not only the http2 one.

Each ProxyPass gets workers assigned and those re-use open connections when enabled. It was not foreseen that regex replacements are done for the host and port parts. If you think this is a bug, please open an issue at the httpd bugzilla.

In general, letting the client determine the hostname or port, seems a security risk, as your server could become open to someone probing your intranet this way. You know best what is applicable in your setup. Just as an explanation why this has not gotten more attention.

[icing]

@gfrankliu for a very detailed explanation, see also https://lists.apache.org/thread/tlzfbvopg5k61nz8mhjq518oowkmm43f

In short, for ProxyPassMatch the enablereuse is on by default unless substitutions happen in the authority part. Explicit enablereuse=on are not disabled, as we assume the user knows what they are doing.

[epsilon-phase]

This breaks http signatures as they generally require that the host header be passed through.

This will, for example, break authentication entirely for S3 implementations as the Host header must be the same on both sides of the connection for the signature to be calculated properly, so as far as default behavior is concerned, this is a very mixed bag.

[icing]

@epsilon-phase this is a configuration choice by a setup. I agree that it is inappropriate and probably breaking in situations as you describe. In which way do you think defaults should be different?