Does reverse_proxy handler support proxying H2C traffic to a UDS?

[fjn308]

To proxy gRPC traffic to a gRPC inbound of Xray (similar to V2Ray), I used to let Xray listen on a port, like 4443, and let Caddy pick gRPC traffic with right Path (here it is /example/TunMulti) out, then proxy it to an upstream (here it is :4443) . The config file reads:

{
    "apps": {
        "http": {
            "servers": {
                "fall": {
                    "listen": [
                        "unix//dev/shm/fall.sock"
                    ],
                    "allow_h2c": true,
                    "routes": [
                        {
                            "handle": [
                                {
                                    "handler": "subroute",
                                    "routes": [
                                        {
                                            "match": [
                                                {
                                                    "protocol": "grpc",
                                                    "path": [
                                                        "/example/TunMulti"
                                                    ]
                                                }
                                            ],
                                            "handle": [
                                                {
                                                    "handler": "reverse_proxy",
                                                    "transport": {
                                                        "protocol": "http",
                                                        "versions": [
                                                            "h2c"
                                                        ]
                                                    },
                                                    "upstreams": [
                                                        {
                                                            "dial": ":4443"
                                                        }
                                                    ]
                                                }
                                            ]
                                        }
                                    ]
                                }
                            ]
                        }
                    ]
                }
            }
        }
    }
}

There is something else I need to explain: Here a VLESS (a protocol) inbound of Xray is working behind :443 (not :4443), it decrypts all TLS traffic and foward everything non-VLESS to a fallback, which can be a port or a UDS (here it is fall.sock, on which Caddy is listening). The traffic flows through Caddy has to be H2C because it has been decrypted, and the upstream (:4443 or grpc.sock in the following) is listened by a gRPC inbound of Xray.

Everything works fine until I try to replace :4443 (a port) to grpc.sock (a UDS), that is, changing

"dial": ":4443"

to

"dial": "unix//dev/shm/grpc.sock"

A similar change is made in the config file of Xray. Such configs do not work, and Caddy reports some error like:

"msg": "dial tcp: lookup /dev/shm/grpc.sock: no such host"

My question is: Does reverse_proxy handler support proxying H2C traffic to a UDS? If it does, I hope to get some help from you. Many thanks.

P.S. I tried "h2c_unix" mentioned here, but seems not working.

[francislavoie]

"versions" needs to contain both "h2c" and "2" to work properly. The "h2c" part is a hack, but "2" is necessary so the codepath for the H2 transport (which is used for regular HTTPS) to work.

[ghost]

Sorry to forget mentioning, I tried to contain both of them, too, but still not working.

Edit: It's important to point out that, when Caddy dials to a port, even only "h2c" is contained, it still works fine.

[francislavoie]

Are you able to build Caddy and add some debug lines? I'm not sure I'll be able to replicate the issue myself right now. (tldr checkout the repo, cd cmd/caddy then run go build, which will output a caddy binary you can use)

First, add just below this chunk of code, a log statement:

https://github.com/caddyserver/caddy/blob/5333c3528bd2badf1676efcced322d151e3706c8/modules/caddyhttp/reverseproxy/reverseproxy.go#L430-L436

Put on the following line:

fmt.Printf("\ndialInfo: %+v\n", dialInfo)

This should show what the dial info has been parsed as.

Then later it gets used by the dialer:

https://github.com/caddyserver/caddy/blob/5333c3528bd2badf1676efcced322d151e3706c8/modules/caddyhttp/reverseproxy/httptransport.go#L132-L136

You can add a log line here as well (on the line before net.Dial()) to make sure the network and address still look right.

fmt.Printf("\ndialNet: %+v\ndialAddr: %+v\n", network, address)

[ghost]

OK, thank you. I'm a newbie though, I'll take a try when my computer at hand later.

[francislavoie]

Reading the code more closely, I think the // TODO: no context, thus potentially wrong dial info is the issue, essentially.

Go's http2 package doesn't provide a DialTLSContext closure that would be what we need to pass through the network the way we do for for the http v1 transport:

https://github.com/caddyserver/caddy/blob/5333c3528bd2badf1676efcced322d151e3706c8/modules/caddyhttp/reverseproxy/httptransport.go#L196-L210

So I think this is more of an upstream issue at this point, Go doesn't really give us the facilities to do this correctly. We could hack in some stuff to pipe in the network, but it would be messy and temporary, and probably not work in all cases.

[francislavoie]

I opened a bug report upstream to see if we can resolve this TODO. But I guess for now, you'll have to stick with TCP 😞

[ghost]

I see (I'm not a pro as you guys so I can only understand your point roughly :), and thank you all the same. Earlier I have been told that Caddy2 does not support h2c to UDS elsewhere (a repo in Chinese), and I think I need to confirm it (essentially, in your word) so I wrote this issue. Does it mean that I do not need to debug as you told any more? And again, thank you for your kindly reply, wish Caddy being more powerful with your continuously development.

[francislavoie]

Does it mean that I do not need to debug as you told any more?

Yeah, I think I understand the way things are wired up enough that it's not necessary anymore.

[ghost]

OK, thank you!

[ghost]

I just noticed that https://github.com/golang/net/pull/123 might fix this issue, so may I ask in advance that, if I want to set the reverse proxy to some h2c UDS in Caddyfile within one line later, should the address "h2c://unix//path/to/the.sock" work? Thank you.

[francislavoie]

h2c:// is a Caddyfile-only shortcut for setting the versions to "versions": ["h2c", "2"]. It seems like you're using JSON, so you wouldn't use that. But yes, the upstream dial address would be unix//path/to/sock

[ghost]

You're right. Recently I have switched to Caddyfile for its readability. I just feel a little unsure about the somehow strange prefix "h2c://unix//" (so many slashes) so I need a certain answer from a pro. Thank you!

[PaTTeeL]

Recently I've read about golang release notes. In https://go.dev/doc/go1.18 , it's seems DialTLS & DialTLSContext is supported by golang :

net/http
    On WebAssembly targets, the Dial, DialContext, DialTLS and DialTLSContext method fields in Transport will now be correctly used, if specified, for making HTTP requests. 

Then I've build a new caddy by using xcaddy

xcaddy build HEAD --with github.com/mastercactapus/caddy2-proxyprotocol --with github.com/caddy-dns/cloudflare --with github.com/mholt/caddy-l4

so caddy version is now v2.5.3-0.20220803170451-1960a0dc117d h1:kIBsNVyxn9SPJ44s6UlljPK7pWjxgCU3LVKPz0+GyVY=

but reverse proxy dial by using UDS still call error , logs here : Aug 03 01:49:12 server-01 caddy[559725]: DEBUG http.log.error.log1 dial tcp: lookup /dev/shm/grpc: no such host {"request": {"remote_ip": "remote ip", "remote_port": "61676", "proto": "HTTP/2.0", "method": "POST", "host": "server-01:443", "uri": "/grpc/TunMulti", "headers": {"Content-Type": ["application/grpc"], "User-Agent": ["grpc-go/1.45.0"], "Te": ["trailers"]}}, "duration": 0.000145173, "status": 502, "err_id": "0if2af75g", "err_trace": "reverseproxy.statusError (reverseproxy.go:1201)"} Aug 03 01:49:12 server-01 caddy[559725]: ERROR http.log.access.log1 handled request {"request": {"remote_ip": "remote ip", "remote_port": "61676", "proto": "HTTP/2.0", "method": "POST", "host": "server-01:443", "uri": "/grpc/TunMulti", "headers": {"Content-Type": ["application/grpc"], "User-Agent": ["grpc-go/1.45.0"], "Te": ["trailers"]}}, "user_id": "", "duration": 0.000145173, "size": 7021, "status": 502, "resp_headers": {"Accept-Ranges": ["bytes"], "Content-Length": ["7021"], "Server": ["Caddy"], "Etag": [""], "Content-Type": ["text/html; charset=utf-8"]}}

configurations here

          { "handle":[{
              "handler":"reverse_proxy",
              "flush_interval":-1,
              "upstreams":[{"dial":"unix//dev/shm/grpc"}],
              "transport":{"protocol":"http","versions":["h2c"]}}],
            "match":[{"header":{"Content-Type":["application/grpc"]},"path":["/grpc/*"]}]},

I've read https://github.com/caddyserver/caddy/raw/a379fa4c6c5d58aa6b812bb21a879ce0944c1ccb/modules/caddyhttp/reverseproxy/httptransport.go

	if sliceContains(h.Versions, "h2c") {
		// crafting our own http2.Transport doesn't allow us to utilize
		// most of the customizations/preferences on the http.Transport,
		// because, for some reason, only http2.ConfigureTransport()
		// is allowed to set the unexported field that refers to a base
		// http.Transport config; oh well
		h2t := &http2.Transport{
			// kind of a hack, but for plaintext/H2C requests, pretend to dial TLS
			DialTLS: func(network, addr string, _ *tls.Config) (net.Conn, error) {
				// TODO: no context, thus potentially wrong dial info
				return net.Dial(network, addr)
			},
			AllowHTTP: true,
		}
		if h.Compression != nil {
			h2t.DisableCompression = !*h.Compression
		}
		h.h2cTransport = h2t
	}

and https://pkg.go.dev/net/http

	// DialTLS specifies an optional dial function for creating
	// TLS connections for non-proxied HTTPS requests.
	//
	// Deprecated: Use DialTLSContext instead, which allows the transport
	// to cancel dials as soon as they are no longer needed.
	// If both are set, DialTLSContext takes priority.
	DialTLS func(network, addr string) (net.Conn, error)

which seems caddy already using DialTLS for reverse proxy in h2c mode, so what can I do to make it works ?

[francislavoie]

@PaTTeeL no, we still don't have it. We need https://github.com/golang/net/pull/123 to be merged. We need DialTLSContext in http2.Transport, not http.Transport.

[PaTTeeL]

@PaTTeeL no, we still don't have it. We need golang/net#123 to be merged. We need DialTLSContext in http2.Transport, not http.Transport.

got it, many thanks for replying.

[francislavoie]

@fjn308 @PaTTeeL The change we needed was finally merged to stdlib! I opened a PR to fix it https://github.com/caddyserver/caddy/pull/4951, please try it out to confirm that it does indeed work!

[PaTTeeL]

@francislavoie it works! build with xcaddy build fix-h2c-dial --output /tmp/ the version is v2.5.3-0.20220812171626-7c3411e435e3 h1:Lq4/0CP4akptUojOhNfAnSZr5BrB16YyZ/7jOvctNGw= logs here:

caddy[617846]: {"level":"DEBUG","ts":"2022/08/12 18:29:28.268","logger":"tls.handshake","msg":"choosing certificate","identifier":"HostName.com","num_choices":1}
caddy[617846]: {"level":"DEBUG","ts":"2022/08/12 18:29:28.268","logger":"tls.handshake","msg":"default certificate selection results","identifier":"HostName.com","subjects":"HostName.com","managed":true,"issuer_key":"acme.zerossl.com-v2-DV90","hash":"90125a36b1cc6de8b88e4a17894d4bd93493b4d5896fe354eb6eb7b187d52696"}
caddy[617846]: {"level":"DEBUG","ts":"2022/08/12 18:29:28.268","logger":"tls.handshake","msg":"matched certificate in cache","subjects":"HostName.com","managed":true,"expiration":"2022/10/10 23:59:59.000","hash":"90125a36b1cc6de8b88e4a17894d4bd93493b4d5896fe354eb6eb7b187d52696"}
caddy[617846]: {"level":"DEBUG","ts":"2022/08/12 18:29:28.911","logger":"http.handlers.reverse_proxy","msg":"selected upstream","dial":"/dev/shm/grpc","total_upstreams":1}
caddy[617846]: {"level":"DEBUG","ts":"2022/08/12 18:29:28.931","logger":"http.handlers.reverse_proxy","msg":"upstream roundtrip","upstream":"unix//dev/shm/grpc","duration":0.019909911,"request":{"remote_ip":"remote_ip","remote_port":"54105","proto":"HTTP/2.0","method":"POST","host":"HostName.com","uri":"34b6b312","headers":{"X-Forwarded-For":"remote_ip","X-Forwarded-Proto":["https"],"X-Forwarded-Host":"HostName.com","Te":["trailers"],"Content-Type":["application/grpc"],"User-Agent":["grpc-go/1.46.0"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"h2","server_name":"HostName.com"}},"headers":{"Content-Type":["application/grpc"]},"status":200}

And it will be perfect if the small bug with Caddyfile configuration could be fixed: reverse_proxy @gRPC h2c://unix//dev/shm/grpc cannot be reconized by caddy while reverse_proxy @gRPC h2c://localhost:8003 works caddy validate --config /etc/caddy/Caddyfile says:

2022/08/12 18:46:55.118 INFO    using provided configuration    {"config_file": "/etc/caddy/Caddyfile", "config_adapter": ""}
validate: adapting config using caddyfile: parsing caddyfile tokens for 'route': /etc/caddy/Caddyfile:143 - Error during parsing: parsing caddyfile tokens for 'reverse_proxy': /etc/caddy/Caddyfile:86 - Error during parsing: for now, URLs for proxy upstreams only support scheme, host, and port components

[mholt]

Woooah cool. Thanks for verifying!

[francislavoie]

@PaTTeeL try this instead:

reverse_proxy @gRPC unix//dev/shm/grpc {
	transport http {
		versions h2c 2
	}
}

The h2c:// scheme is just a shortcut to setting the transport versions.

[PaTTeeL]

@PaTTeeL try this instead:

reverse_proxy @gRPC unix//dev/shm/grpc {
	transport http {
		versions h2c 2
	}
}

The h2c:// scheme is just a shortcut to setting the transport versions.

yes, I'm now using the configuration you've mentioned for working currectly. the configuration seems short and easyily reading if h2c:// scheme with unix// can be fixed.

[ghost]

That's cool! Thank you! @PaTTeeL @francislavoie @mholt