Dynamic proxy based on SNI

[vk496]

Hello.

Related to #240, there are any way to route traffic based on the SNI? For example:

Instead of static configuration like this:

	layer4 {
		:18000 {
			@app1 tls sni app1.internal
			route @app1 {
				tls
				proxy 192.168.1.155:3333
			}

			@app2 tls sni app2.internal
			route @app2 {
				tls
				proxy 192.168.1.156:3333
			}

			@app3 tls sni app3.internal
			route @app3 {
				tls
				proxy 192.168.1.157:3333
			}

			@insecure http
			@secure tls

			route @insecure @secure {
				proxy localhost:443
			}
		}
	}

Have the app routed directly based on the SNI. Maybe similar to this:

	layer4 {
		:18000 {
                        #regex or simple wildcard
			@app tls sni app[0-9]{1,2}.internal
			route @app {
				tls
				proxy ${sni}:3333 # or maybe object access ${app.tls.sni}
			}

			@insecure http
			@secure tls

			route @insecure @secure {
				proxy localhost:443
			}
		}
	}

The idea behind is that there will be multiple instances of the app, and the client could route based on the SNI value. Im not sure how ports treatement could be possible (or regex/parser of the sni), but at least using the sni directly could reduce a lot the config redundancy.

[mholt]

So basically the SNI matcher needs to support regex, I guess? Or a new sni_regexp matcher?

[vk496]

So basically the SNI matcher needs to support regex, I guess? Or a new sni_regexp matcher?

That's one of the things. And the other, reference those attributes, so it can be "reused". Even better, if we're able to use basic functions like split or array acces, we could "construct" proxy destination from sni string. Something like:

# sni example: app02_3333_.internal
@app tls sni app[0-9]{1,2}_[0-9]{1,4}_.internal
	route @app {
		tls
		proxy ${app.tls.sni.split("_")[0]}:${app.tls.sni.split("_")[1]}
	}

Im not fluent in Go, but the idea is simple: regex on SNI for matching and then accessing what was matched (and if possible, manipulate/parse it)

[vnxme]

Hi! My 2 cents:

• proxy ${sni}:3333 # or maybe object access ${app.tls.sni} is already implemented by tls matcher, use {l4.tls.server_name}.
• sni (sub)matcher is a part of Caddy, not caddy-l4. Whether it is an expanded syntax for sni matcher or a new sni_regexp matcher, it would be more logical to put it to modules/caddytls/matchers.go of the mainline.

Even better, if we're able to use basic functions like split or array acces, we could "construct" proxy destination from sni string.

You may also implement your own sni_advanced matcher with any syntax you like that perfectly satisfies your needs and build Caddy with it.

[sorenisanerd]

I'm using something like this in my Caddyfile to perform SRV lookups to identify backends for HTTP:

*.srv.example.com {
	map {host} {consul_service} {
		~(.*)\.srv\.example\.com$ "${1}.service.consul"
	}
	reverse_proxy {
		dynamic srv {consul_service} {
			resolvers "10.10.10.10"
		}
	}
}

Being able to do the same for L4 would be amazing.

[lelemka0]

I'm trying out the new sni_regexp matcher, but I ran into 2 problems.

1. Unable to obtain matching results

layer4 {
	:2000 {
		@test tls sni_regexp ^([0-9]{1})\.test$
		route @test {
			proxy AAA{tls.regexp.1}:9000
		}
	}
}

{"level":"error","ts":1737391062.3122277,"logger":"layer4","msg":"handling connection","remote":"****","error":"dial tcp: lookup AAA on 127.0.0.53:53: no such host"}

2. proxy cannot accept a dynamic port number (placeholders in port)

@test tls sni_regexp ^([0-9]{1})\.test$
route @test {
	proxy localhost:900{tls.regexp.1}
}

Error: loading initial config: loading new config: loading layer4 app module: provision layer4: server 'srv0': route 0: position 0: loading module 'proxy': provision layer4.handlers.proxy: upstream 0: invalid start port: strconv.ParseUint: parsing "900{tls.regexp.1}": invalid syntax

[vnxme]

1. Unable to obtain matching results

I think you should try double escaping as suggested in caddyserver/caddy#6569, i.e. @test tls sni_regexp ^([0-9]{1})\\.test$

2. proxy cannot accept a dynamic port number (placeholders in port)

And this is a current design limitation, since the port is parsed and converted from string to integer after static placeholders are replaced (e.g. {env.XYZ}, but before dynamic placeholders are replaced (e.g. {tls.regexp.1}).

It's definitely possible to implement the dynamic port number feature, but it will require many changes to the code. PRs are welcome.

[lelemka0]

I think you should try double escaping as suggested in https://github.com/caddyserver/caddy/pull/6569, i.e. @test tls sni_regexp ^([0-9]{1})\.test$

I have tried this, but double escaping will cause the match to fail. The log shows dial tcp: lookup AAA, which proves that the match has succeeded but the placeholder has not been replaced correctly. I also tried the top level domain name which don't need escaping, the problem remains.

layer4 {
	:2000 {
		@test tls sni_regexp ^([0-9]{1})-test$
		route @test {
			proxy AAA{tls.regexp.1}:9000
		}
	}
}

{"level":"debug","ts":1737621980.189211,"logger":"layer4.handlers.proxy","msg":"dial upstream","remote":"****","upstream":"AAA:9000","error":"dial tcp: lookup AAA on 127.0.0.53:53: no such host"}
{"level":"error","ts":1737621980.1892602,"logger":"layer4","msg":"handling connection","remote":"****","error":"dial tcp: lookup AAA on 127.0.0.53:53: no such host"}

~~Regarding the dynamic port number feature, I think we just need to move the address parsing and validation to after repl.ReplaceAll of dialPeers. But now I'm stuck on healthchecks, no matter it's dynamic hostname or dynamic port, health check doesn't make sense, but abandoning it will make the health check of fixed upstream invalid.~~ ~~Maybe we need a new l4proxydynamic ?~~

Adding an option to recognize that the upstream is a dynamic address and make changes accordingly might be a good solution with less changes.

[vnxme]

@lelemka0 Let's make sure we are testing the same software. What version of Caddy do you use? I suppose sni_regexp has been released with 2.9.0-beta3.

Update: yes, you are right, I see where the problem is.

[vnxme]

This is why sni_regexp doesn't work for layer4 connections when it comes to {tls.regexp.*} placeholders:

• MatchServerNameRE.Match() tries to load the context from tls.ClientHelloInfo here. It works fine for a standard TLS connection, so sni_regexp should work fine as a part of Caddy mainline. But it doesn't work for L4 connections because of an empty context.
• MatchTLS.Match() calls submatchers, including MatchServerNameRE.Match(), with an artificially composed tls.ClientHelloInfo here. Since tls.ClientHelloInfo.ctx is non-public, we can't set it from MatchTLS.Match(), so it's empty. We do set tls.ClientHelloInfo.Conn, but MatchServerNameRE.Match() only knows it is a net.Conn, so we can't easily obtain the context from it.
• In order to make MatchServerNameRE.Match() obtain the context passed from caddy-l4, we have to import it as a Caddy dependency. I doubt this is what we would like to do now. Nevertheless, it will be possible once we merge Caddy and caddy-l4.
• Another option I can see is to change MatchServerNameRE.Match() signature, so that we could pass the L4 context along with tls.ClientHelloInfo. Yet this option doesn't seem easy, as it requires changing all TLS matchers at the same time.

@mholt, Hi, What do you think about it?

[lelemka0]

It's definitely possible to implement the dynamic port number feature, but it will require many changes to the code. PRs are welcome.

I think this PR #289 will support dynamic port numbers.

[mholt]

Good analysis @vnxme -- thanks.

In order to make MatchServerNameRE.Match() obtain the context passed from caddy-l4, we have to import it as a Caddy dependency. I doubt this is what we would like to do now.

Yeah, in fact, this would result in an import cycle which wouldn't compile.

There is a way to add indirection though, by an interface type for a method like GetContext() context.Context or something. So if the net.Conn satisfied that interface, maybe the matcher could type-assert that if it can't get a context from the ClientHelloInfo directly.

Still not my favorite.

I wonder if we should petition the Go team for a way to set that unexported context, like you can with http.Request.

[vnxme]

There is a way to add indirection though, by an interface type for a method like GetContext() context.Context or something. So if the net.Conn satisfied that interface, maybe the matcher could type-assert that if it can't get a context from the ClientHelloInfo directly.

Nice idea, thanks. Will require changes in both caddytls.MatchServerNameRE.Match() of the mainline code and layer4.Connection of caddy-l4. I will make sure that it works and submit the corresponding PRs.

[mholt]

Sounds good for now. Thank you 😃

[vnxme]

@lelemka0 Please try https://github.com/caddyserver/caddy/pull/6804 and https://github.com/mholt/caddy-l4/pull/290.

This is the config I've tested locally.

{
	layer4 {
		:2000 {
			@test tls sni_regexp ^([0-9]{1})\.test$
			route @test {
				proxy 127.0.0.{tls.regexp.1}:443
			}
		}
	}
}

*.test {
	respond OK 200
}

[lelemka0]

@vnxme it works.

[sandros94]

@lelemka0 Please try caddyserver/caddy#6804 and #290.

~~Any tips on building a docker image with caddy's pre-releases? With xcaddy is quite easy to specify which caddy-l4 commit to use, but what about caddy itself?~~

Ended up fully rebuilding it from source

[lelemka0]

@sandros94 Maybe you have already solved it. Here is a Dockerfile for building any version of the caddy container including caddy-l4.

FROM golang AS builder
WORKDIR /usr/bin
RUN go install github.com/caddyserver/xcaddy/cmd/xcaddy@latest

ARG CADDY_TAG=master            #caddy version; it can be a tag, a branch or a commit hash
ARG L4_TAG=master               #caddy-l4 version
RUN xcaddy build ${CADDY_TAG} \
    --with github.com/mholt/caddy-l4@${L4_TAG}

FROM caddy:latest AS dist
COPY --from=builder /usr/bin/caddy /usr/bin/caddy

Because the current official caddy builder container contains a version of go that is not sufficient to compile the pre-release caddy, golang is used as the compilation container. For release version, the builder container can be set to caddy:builder and the xcaddy build can be removed.

[sandros94]

@lelemka0 thank you! much cleaner than my approach!