caddy

caddy copied to clipboard

I'm developing a caddy listener plugin that multiplex connection over tls, and dispatch them to different backend, ie, mysql over tls, ssh over tls etc. But unlike caddy-l4, this works as a listener wrapper plugin.

However because golang http.Server only directly support h2 when listener returns a *tls.Conn, I have to use h2c. Now http2 request has request.TLS field filled because h2 server uses an interface to sniff tls related information. But http1.1 request is not, I wonder if caddy can set the field when the underlying conn implements the same interface.

Sounds interesting. Can you help me understand a little better by showing some relevant code? Sounds like you need the http.Request.TLS field filled out even though TLS is not used?

TLS is used, like caddy-l4. But I make it a listener wrapper to simplify caddyfile parsing. I can simply add a domain and it will have a tls certificate, no need to modify json configure to add domain. That domain will simply return 200 when opened in browser, but because it's not relevant to its functionality, I can name it sql.example.com to add mysql ssl support etc. Neccessary sni match can be done in my plugin too.

However, because go http server code, http1.1 will only have request.tls field filled when listener returns a tls.Conn.

This will also interfere with strict_sni_check

The main listener is like this

type Listener struct {
	net.Listener
}

func (l *Listener) Accept() (net.Conn, error) {
	conn, err := l.Listener.Accept()
	if err != nil {
		return nil, err
	}

	return &Conn{
		Conn: conn,
	}, nil
}

The Conn definition is

type Conn struct {
	net.Conn
	br   *bufio.Reader
	once sync.Once
}

func (c *Conn) Read(p []byte) (int, error) {
	var once bool
	c.once.Do(func() {
		c.br = handleConn(c.Conn)
		once = true
	})

	if once && c.br == nil {
		c.Conn.Close()
		return 0, net.ErrClosed
	}

	return c.br.Read(p)
}

Embedded net.Conn may be a tls.Conn, and when it is, the returned conn will have a method to extract tls connection information.

Here is a demo that shows what I want to achive and how I achieved it

package main

import (
	"bufio"
	"context"
	"crypto/tls"
	"github.com/caddyserver/certmagic"
	"golang.org/x/net/http2"
	"log"
	"net"
	"net/http"
	"sync"
)

func main() {
	server := new(http.Server)

	h2server := &http2.Server{
		NewWriteScheduler: func() http2.WriteScheduler {
			return http2.NewPriorityWriteScheduler(nil)
		},
	}
	http2.ConfigureServer(server, h2server)

	server.Handler = http.HandlerFunc(func(writer http.ResponseWriter, request *http.Request) {
		conn := request.Context().Value("conn").(net.Conn)
		if request.TLS == nil {
			log.Println("tls sniff")
			sniffer := conn.(connectionStater)
			state := sniffer.ConnectionState().NegotiatedProtocol
			request.TLS = new(tls.ConnectionState)
			*request.TLS = sniffer.ConnectionState()
			if state == http2.NextProtoTLS {
				sniffer.Discard()
				h2server.ServeConn(conn, &http2.ServeConnOpts{
					Context:    request.Context(),
					BaseConfig: server,
					Handler:    server.Handler,
				})
				return
			}
		}
		log.Println(request.TLS)
		log.Println(conn.RemoteAddr(), conn.LocalAddr())
		log.Println(request.Context().Value(http.ServerContextKey) == server)

		log.Println(request.Header)
		log.Println(request.Proto, request.ProtoMajor, request.ProtoMinor)
		log.Println(request.Method)
		log.Println(request.URL.Path)
	})
	server.ConnContext = func(ctx context.Context, c net.Conn) context.Context {
		return context.WithValue(ctx, "conn", c)
	}

	certmagic.Default.Storage = &certmagic.FileStorage{Path: "redacted"}
	certmagic.HTTPSPort = 8443
	//certmagic.HTTPPort = 8444
	certmagic.DefaultACME.Agreed = true
	certmagic.DefaultACME.Email = "redacted"
	certmagic.DefaultACME.CA = certmagic.ZeroSSLProductionCA
	config, _ := certmagic.TLS([]string{"redacted"})
	config.NextProtos = append(config.NextProtos, "h2", "http/1.1")
	l, _ := tls.Listen("tcp", ":8443", config)
	log.Fatal(server.Serve(&Listener{l}))
}

type Listener struct {
	net.Listener
}

func (l *Listener) Accept() (net.Conn, error) {
	conn, err := l.Listener.Accept()
	if err != nil {
		return nil, err
	}

	return &Conn{
		Conn: conn,
	}, nil
}

type Conn struct {
	net.Conn
	br     *bufio.Reader
	once   sync.Once
	buffer bool
}

func (c *Conn) Read(p []byte) (int, error) {
	var once bool
	c.once.Do(func() {
		c.br = handleConn(c.Conn)
		once = true
	})

	if once && c.br == nil {
		c.Conn.Close()
		return 0, net.ErrClosed
	}

	if once {
		preface, err := c.br.Peek(len(http2.ClientPreface))
		if err == nil && string(preface) == http2.ClientPreface {
			c.buffer = true
		}
	}

	if c.buffer {
		data, _ := c.br.Peek(c.br.Buffered())
		return copy(p, data), nil
	}
	return c.br.Read(p)
}

func (c *Conn) ConnectionState() tls.ConnectionState {
	return c.Conn.(*tls.Conn).ConnectionState()
}

func (c *Conn) Discard() {
	c.buffer = false
}

type connectionStater interface {
	ConnectionState() tls.ConnectionState
	Discard()
}

It's actually more difficult than it seems, golang http.server will do background read one byte from underlying net.Conn unless hijacked. Unless there are listener related caddy namespace added, it will not be easy to solve.

I'll try to catch up on this soon, thanks! In the meantime, do you want an invite to our developer Slack? If so just let me know which email to invite.

Sure, thanks for the invitation, my email is [email protected]

Turns out not that difficult either. Underlying conn need to implement a similar method like http.Hijacker to prevent orginal http handler goroutine to modify its state.

It's not that good actually, this prevents multiple listener wrapper to be used together. i.e, because conn should buffer http2 client preface and settings frames, when next listener wraps it, it should calll hijack to read the buffered bytes which is a pain.

I tested bradfitz's suggestion locally, and it seems to work fine. Recording the underlying read and written bytes and corresponding tls.Config rand.Reader is the way to go. But this means a new possible interface implemented by caddy listener modules that takes a tls.Config as parameter.