Improve performance of FastCGI transport (and clean up code)

[mholt]

The current fastcgi implementation is functional, but clunky and slow.

Go does not have a standard FastCGI client implementation (note that we are not a responder we are a client), so in January 2015 we forked the code from http://bitbucket.org/PinIdea/fcgi_client (which is forked from https://code.google.com/p/go-fastcgi-client/) -- and I am very grateful to the original authors for publishing it under a liberal license, so I did not have to go learn the nitty-gritty of the FastCGI protocol and implement it myself. However, the code was never really intended for heavy use.

But now, Caddy is used in heavy production environments where performance matters. Most of our code base could benefit from optimizing, but the FastCGI client should still have lots of low-hanging fruit that makes significant optimizations without too much trouble. For example, we could reduce buffering and pool buffers.

The client.go file needs the most attention.

This file would benefit greatly from a refactoring to clean it up, and I've determined that a refactor will be necessary to support optimizations like buffer pooling.

I don't know how soon I will be able to get around to it, so if someone wants to take up the challenge of working on this file, please feel free to comment below and discuss your plans. The FastCGI protocol is not particularly complex in its most basic form, so you may find that rewriting the client from scratch and using the existing code (and the spec) as a guide could be the most straightforward solution.

This code is used by a lot of people in a lot of important environments. Benchmark tests (and regular unit tests) should support any changes.

Thank you to anyone who helps!

[mholt]

For instance, the newWriter() function is extremely inefficient currently:

https://github.com/caddyserver/caddy/blob/97caf368eea8d2c33a7786fbe3471b83b5b294dc/modules/caddyhttp/reverseproxy/fastcgi/client.go#L310-L314

It appears at the top of all memory profiles as a huge allocator, because the buffer isn't pooled. (Look closely and you'll see why using sync.Pool is impossible without a refactor.)

[raygervais]

Hey 🤗

I've been following caddy since the release of 2.0, hoping to contribute here over time and where possible.

Would love to start with the low hanging fruit and work my way through with guidance. 💻⚙️📍

[raygervais]

First question, how exactly are we profiling Caddy?

[hairyhenderson]

@raygervais there's a pprof endpoint that's built in to the admin endpoint - at http://localhost:2019/debug/pprof by default

[mholt]

Yep, using that would be easiest; and it might also not hurt to write benchmark tests as well: https://golang.org/pkg/testing/#hdr-Benchmarks

Doing a profile and benchmark before changes, then a profile after them (doing identical things) should be a good indication as to the improvement.

[raygervais]

Thanks for the advice, going to look into how the code works and will try to improve it starting next week (currently AFK). if you take it on before I do, I'd be happy to help in anyway and learn as I go.

[jasonmccallister]

Happy to assist in anyway I can, I'm not very familiar with fastcgi but am willing to help with implementation and testing!

I have looked at this in the past and found a few interesting projects that do something similar:

1. https://golang.org/pkg/net/http/fcgi/ which links to this page with a nicely formatted spec
2. roadrunner.dev is a Go project for a PHP runtime with a lot of functionality. Not quite sure if it uses a FastCGI or a PSR-7 implementation
3. https://github.com/yookoala/gofast is another FastCGI implementation in Go

We might be able find some inspiration in one of these projects?

[mholt]

@jasonmccallister Thanks for the links!

https://golang.org/pkg/net/http/fcgi/

Note that this is a responder, not a client.

roadrunner.dev is a Go project for a PHP runtime with a lot of functionality. Not quite sure if it uses a FastCGI or a PSR-7 implementation

It's PSR-7.

https://github.com/yookoala/gofast is another FastCGI implementation in Go

That's good to know, I haven't seen that one.

[jasonmccallister]

@mholt I did some "light" reading on the FastCGI spec last night, did not realize the difference between a responder and client.

Reading the net/http/fcgi and seeing that FastCGI is offline and the spec is offline. Makes me wonder the future of FastCGI or if I should even worry/care :).

Regardless, hope those help!

[francislavoie]

Actually yookoala/gofast looks very interesting, it has some features some people have asked for like FastCGI Authorizers. Since it's basically just a handler, maybe we could swap it in? We'd need to make sure it conforms to the assumptions we've made in our implementation though (note the path manipulation logic in the handler code)

/cc @yookoala FYI, if you'd be interested in helping out!

[mholt]

Ideally, what we want is a RoundTripper from a FastCGI client library. All it has to do is take a request and give us a response.

[yookoala]

Actually yookoala/gofast looks very interesting, it has some features some people have asked for like FastCGI Authorizers. Since it's basically just a handler, maybe we could swap it in? We'd need to make sure it conforms to the assumptions we've made in our implementation though (note the path manipulation logic in the handler code)

/cc @yookoala FYI, if you'd be interested in helping out!

I'd love to. The only catch was I couldn't find any FOSS implementation that uses the authoizer with. We'd need some way to verify our implementation. (I could have mis-read the specification)

[francislavoie]

Oh, okay; that's fine, I wasn't trying to be specific about the authorizers, just an example of something your implementation has that ours doesn't. Thought we could compare notes and hoping that you might be interested in helping us improve our implementation 🙂

[raygervais]

Apologies for delay, work and other obligations are taking quite a bit of time. If someone else wants to take it, by all means. I'll try to contribute to it if I get a chance

[MaxiMii05]

I don't intend to put anyone under pressure but I just wanted to point out that there are many users who would benefit from a nice and fast FastCGI implementation - especially with services such as Nextcloud, Moodle or really any large PHP program.

It's been more than a year and a half and while not being able to contribute myself, I would love to see progress with this issue. I understand if this is not a priority. Also, thank you to all contributors of this awesome reverse proxy.

[mholt]

I too would like to see improvements here, even do it myself -- but I've been too busy with other tasks for the project. I would be happy to schedule it in soon if a company could sponsor the work. :+1: (Contact me if interested.)

[mholt]

@WeidiDeng has a good start on this if anyone would like to try it out. It could use extensive testing: #4978

[mattvb91]

I have 2 seperate projects with pretty big cypress pipelines I can throw at this for an initial look from Tuesday onwards. Is there any specific logging I should enable? Im assuming just debug for a start and see if I can spot anything.

[mholt]

@mattvb91 Definitely enable debug logging. Thanks for being willing to test this! Perhaps @WeidiDeng has suggestions for how to best test it.

[WeidiDeng]

Just using wget to pull http://localhost:2019/debug/pprof/heap and http://localhost:2019/debug/pprof/profile?seconds=30 at an interval and post the resulting file here, I haven't added any debug output yet.

[mattvb91]

Sorry for the dumb question.

Im trying to build (?) @WeidiDeng branch with xcaddy am I going in the right direction?

RUN xcaddy build --with github.com/WeidiDeng/caddy/tree/fastcgi-improve

Results in:

2022/08/28 08:16:41 [INFO] exec (timeout=0s): /usr/local/go/bin/go get -d -v github.com/WeidiDeng/caddy/tree/fastcgi-improve github.com/caddyserver/caddy/[email protected] 
go: downloading github.com/WeidiDeng/caddy v1.0.5
go: module github.com/WeidiDeng/caddy@upgrade found (v1.0.5), but does not contain package github.com/WeidiDeng/caddy/tree/fastcgi-improve
2022/08/28 08:17:01 [FATAL] exit status 1

Im assuming im doing something wrong with my build process?

[francislavoie]

Yeah, it's not super obvious; this should do the trick:

xcaddy build --with github.com/caddyserver/caddy/v2=github.com/WeidiDeng/caddy/v2@fastcgi-improve

[mattvb91]

Thank you @francislavoie appreciated! Got it working.

@WeidiDeng here is the initial heap for pprof for http://localhost:2019/debug/pprof/heap. The 30 second interval heap seems to be corrupted or just return an empty response as it ends up only being 205 bytes?

heap.zip

(had to append .zip to filename to attach on github)

[mholt]

@mattvb91 Odd... I've never heard of that before. Do other profiles work? What kinds of actions cause it to break?

[mattvb91]

yea the one above works fine, only the interval comes back empty

[mholt]

@mattvb91 Gotcha. So that heap doesn't seem to have any fastcgi code in it, but I see caching-related allocations. Is that from a caddy instance that is using the fastcgi transport?

[mattvb91]

Whoops my bad now that I think about it I may have just created that heap right after launching the server. Im assuming now that you say that I need to launch a few requests that hit my fastcgi backend & then run pprof?

[mholt]

@mattvb91 Yeah :)

Ideally, do a profile without the patch (old binary) then do a profile with the same load with the patch (new binary) and then we can compare :+1:

[mattvb91]

seems obvious in hindsight :laughing:

So now when I try and hit my php backend in this case I get the following:

caddy_1                | {"level":"debug","ts":1662138690.5002112,"logger":"http.stdlib","msg":"http: panic serving 172.18.0.1:38836: interface conversion: io.ReadCloser is fastcgi.clientCloser, not *fastcgi.clientCloser\ngoroutine 201 [running]:\nnet/http.(*conn).serve.func1()\n\tnet/http/server.go:1825 +0xbf\npanic({0x1db7aa0, 0xc0007700c0})\n\truntime/panic.go:844 +0x258\ngithub.com/caddyserver/caddy/v2/modules/caddyhttp/reverseproxy/fastcgi.Transport.RoundTrip({{0xc0004fc138, 0x15}, {0xc0004225c0, 0x1, 0x4}, 0x0, 0x0, 0xb2d05e00, 0x0, 0x0, ...}, ...)\n\tgithub.com/caddyserver/caddy/[email protected]/modules/caddyhttp/reverseproxy/fastcgi/fastcgi.go:196 +0xd9b\ngithub.com/caddyserver/caddy/v2/modules/caddyhttp/reverseproxy.(*Handler).reverseProxy(0xc00001a680, {0x7fd64ac37fe8?, 0xc000850240}, 0xc0004f9400, 0xc0004f8f00, 0xc00009abc0, {0xc0002facc0, {0x204d31f, 0x3}, {0xc0002d8ee0, ...}, ...}, ...)\n\tgithub.com/caddyserver/caddy/[email protected]/modules/caddyhttp/reverseproxy/reverseproxy.go:784 +0x8d7\ngithub.com/caddyserver/caddy/v2/modules/caddyhttp/reverseproxy.(*Handler).proxyLoopIteration(0xc00001a680, 0xc0004f9400, 0x4?, {0x7fd64ac37fe8, 0xc000850240}, {0x0, 0x0}, {0xc0bcac14e5bc6191, 0x6d3388bd2, 0x3401000}, ...)\n\tgithub.com/caddyserver/caddy/[email protected]/modules/caddyhttp/reverseproxy/reverseproxy.go:546 +0xf10\ngithub.com/caddyserver/caddy/v2/modules/caddyhttp/reverseproxy.(*Handler).ServeHTTP(0xc00001a680, {0x7fd64ac37fe8, 0xc000850240}, 0xc0004f8f00, {0x24835c0, 0xc00009ad20})\n\tgithub.com/caddyserver/caddy/[email protected]/modules/caddyhttp/reverseproxy/reverseproxy.go:454 +0x3dc\ngithub.com/caddyserver/caddy/v2/modules/caddyhttp.(*metricsInstrumentedHandler).ServeHTTP(0xc00041b020, {0x7fd64ac37fe8?, 0xc0000cfe60}, 0xc0004f8f00, {0x24835c0, 0xc00009ad20})\n\tgithub.com/caddyserver/caddy/[email protected]/modules/caddyhttp/metrics.go:132 +0x53b\ngithub.com/caddyserver/caddy/v2/modules/caddyhttp.wrapMiddleware.func1.1({0x7fd64ac37fe8?, 0xc0000cfe60?}, 0xc000882a30?)\n\tgithub.com/caddyserver/caddy/[email protected]/modules/caddyhttp/routes.go:276 +0x3b\ngithub.com/caddyserver/caddy/v2/modules/caddyhttp.HandlerFunc.ServeHTTP(0xc00055e860?, {0x7fd64ac37fe8?, 0xc0000cfe60?}, 0xc0008166c0?)\n\tgithub.com/caddyserver/caddy/[email protected]/modules/caddyhttp/caddyhttp.go:58 +0x2f\ngithub.com/darkweak/souin/plugins/caddy.(*SouinCaddyPlugin).ServeHTTP(0xc00001a000, {0x7fd64ac37fe8?, 0xc0000cfe60}, 0xc0004f8f00, {0x24835c0, 0xc00009ae40})\n\tgithub.com/darkweak/souin/plugins/[email protected]/httpcache.go:106 +0x8a8\ngithub.com/caddyserver/caddy/v2/modules/caddyhttp.(*metricsInstrumentedHandler).ServeHTTP(0xc00041b000, {0x7fd64ac37fe8?, 0xc0000cfda0}, 0xc0004f8f00, {0x24835c0, 0xc00009ae40})\n\tgithub.com/caddyserver/caddy/[email protected]/modules/caddyhttp/metrics.go:132 +0x53b\ngithub.com/caddyserver/caddy/v2/modules/caddyhttp.wrapMiddleware.func1.1({0x7fd64ac37fe8?, 0xc0000cfda0?}, 0x24835c0?)\n\tgithub.com/caddyserver/caddy/[email protected]/modules/caddyhttp/routes.go:276 +0x3b\ngithub.com/caddyserver/caddy/v2/modules/caddyhttp.HandlerFunc.ServeHTTP(0x24835c0?, {0x7fd64ac37fe8?, 0xc0000cfda0?}, 0xc000422940?)\n\tgithub.com/caddyserver/caddy/[email protected]/modules/caddyhttp/caddyhttp.go:58 +0x2f\ngithub.com/caddyserver/caddy/v2/modules/caddyhttp.wrapRoute.func1.1({0x7fd64ac37fe8, 0xc0000cfda0}, 0xc0004f8f00)\n\tgithub.com/caddyserver/caddy/[email protected]/modules/caddyhttp/routes.go:248 +0x3a8\ngithub.com/caddyserver/caddy/v2/modules/caddyhttp.HandlerFunc.ServeHTTP(0x1d7dd60?, {0x7fd64ac37fe8?, 0xc0000cfda0?}, 0xe?)\n\tgithub.com/caddyserver/caddy/[email protected]/modules/caddyhttp/caddyhttp.go:58 +0x2f\ngithub.com/caddyserver/caddy/v2/modules/caddyhttp.wrapRoute.func1.1({0x7fd64ac37fe8, 0xc0000cfda0}, 0xc0004f8f00)\n\tgithub.com/caddyserver/caddy/[email protected]/modules/caddyhttp/routes.go:216 +0x336\ngithub.com/caddyserver/caddy/v2/modules/caddyhttp.HandlerFunc.ServeHTTP(0xc00084c000?, {0x7fd64ac37fe8?, 0xc0000cfda0?}, 0x24835c0?)\n\tgithub.com/caddyserver/caddy/[email protected]/modules/caddyhttp/caddyhttp.go:58 +0x2f\ngithub.com/caddyserver/caddy/v2/modules/caddyhttp.(*Subroute).ServeHTTP(0xc00041ac80, {0x7fd64ac37fe8, 0xc0000cfda0}, 0x6?, {0x24835c0, 0x2190960})\n\tgithub.com/caddyserver/caddy/[email protected]/modules/caddyhttp/subroute.go:74 +0x6d\ngithub.com/caddyserver/caddy/v2/modules/caddyhttp.(*metricsInstrumentedHandler).ServeHTTP(0xc00041b2e0, {0x7fd64ac37fe8?, 0xc0000cfc20}, 0xc0004f8f00, {0x24835c0, 0x2190960})\n\tgithub.com/caddyserver/caddy/[email protected]/modules/caddyhttp/metrics.go:132 +0x53b\ngithub.com/caddyserver/caddy/v2/modules/caddyhttp.wrapMiddleware.func1.1({0x7fd64ac37fe8?, 0xc0000cfc20?}, 0xc0008829c0?)\n\tgithub.com/caddyserver/caddy/[email protected]/modules/caddyhttp/routes.go:276 +0x3b\ngithub.com/caddyserver/caddy/v2/modules/caddyhttp.HandlerFunc.ServeHTTP(0xc00055eeb0?, {0x7fd64ac37fe8?, 0xc0000cfc20?}, 0xc000816240?)\n\tgithub.com/caddyserver/caddy/[email protected]/modules/caddyhttp/caddyhttp.go:58 +0x2f\ngithub.com/darkweak/souin/plugins/caddy.(*SouinCaddyPlugin).ServeHTTP(0xc00042f860, {0x7fd64ac37fe8?, 0xc0000cfc20}, 0xc0004f8f00, {0x24835c0, 0xc00009ace0})\n\tgithub.com/darkweak/souin/plugins/[email protected]/httpcache.go:106 +0x8a8\ngithub.com/caddyserver/caddy/v2/modules/caddyhttp.(*metricsInstrumentedHandler).ServeHTTP(0xc00041b280, {0x2494840?, 0xc00043c1c0}, 0xc0004f8f00, {0x24835c0, 0xc00009ace0})\n\tgithub.com/caddyserver/caddy/[email protected]/modules/caddyhttp/metrics.go:132 +0x53b\ngithub.com/caddyserver/caddy/v2/modules/caddyhttp.wrapMiddleware.func1.1({0x2494840?, 0xc00043c1c0?}, 0x24835c0?)\n\tgithub.com/caddyserver/caddy/[email protected]/modules/caddyhttp/routes.go:276 +0x3b\ngithub.com/caddyserver/caddy/v2/modules/caddyhttp.HandlerFunc.ServeHTTP(0x24835c0?, {0x2494840?, 0xc00043c1c0?}, 0xc000422c00?)\n\tgithub.com/caddyserver/caddy/[email protected]/modules/caddyhttp/caddyhttp.go:58 +0x2f\ngithub.com/caddyserver/caddy/v2/modules/caddyhttp.wrapRoute.func1.1({0x2494840, 0xc00043c1c0}, 0xc0004f8f00)\n\tgithub.com/caddyserver/caddy/[email protected]/modules/caddyhttp/routes.go:248 +0x3a8\ngithub.com/caddyserver/caddy/v2/modules/caddyhttp.HandlerFunc.ServeHTTP(0x1d7dd60?, {0x2494840?, 0xc00043c1c0?}, 0xe?)\n\tgithub.com/caddyserver/caddy/[email protected]/modules/caddyhttp/caddyhttp.go:58 +0x2f\ngithub.com/caddyserver/caddy/v2/modules/caddyhttp.wrapRoute.func1.1({0x2494840, 0xc00043c1c0}, 0xc0004f8f00)\n\tgithub.com/caddyserver/caddy/[email protected]/modules/caddyhttp/routes.go:216 +0x336\ngithub.com/caddyserver/caddy/v2/modules/caddyhttp.HandlerFunc.ServeHTTP(0x2500?, {0x2494840?, 0xc00043c1c0?}, 0x1f8d4c0?)\n\tgithub.com/caddyserver/caddy/[email protected]/modules/caddyhttp/caddyhttp.go:58 +0x2f\ngithub.com/caddyserver/caddy/v2/modules/caddyhttp.(*Server).enforcementHandler(0x0?, {0x2494840?, 0xc00043c1c0?}, 0xc000882990?, {0x24835c0?, 0xc00041b5c0?})\n\tgithub.com/caddyserver/caddy/[email protected]/modules/caddyhttp/server.go:352 +0x252\ngithub.com/caddyserver/caddy/v2/modules/caddyhttp.(*Server).wrapPrimaryRoute.func1({0x2494840?, 0xc00043c1c0?}, 0x4c6ab7?)\n\tgithub.com/caddyserver/caddy/[email protected]/modules/caddyhttp/server.go:328 +0x3b\ngithub.com/caddyserver/caddy/v2/modules/caddyhttp.HandlerFunc.ServeHTTP(0xc0005f55f0?, {0x2494840?, 0xc00043c1c0?}, 0xc0004f8f00?)\n\tgithub.com/caddyserver/caddy/[email protected]/modules/caddyhttp/caddyhttp.go:58 +0x2f\ngithub.com/caddyserver/caddy/v2/modules/caddyhttp.(*Server).ServeHTTP(0xc000178e00, {0x2494840, 0xc00043c1c0}, 0xc0004f8f00)\n\tgithub.com/caddyserver/caddy/[email protected]/modules/caddyhttp/server.go:264 +0xb46\nnet/http.serverHandler.ServeHTTP({0x248d328?}, {0x2494840, 0xc00043c1c0}, 0xc0004f8d00)\n\tnet/http/server.go:2916 +0x43b\nnet/http.(*conn).serve(0xc00020de00, {0x24963f0, 0xc0005d55c0})\n\tnet/http/server.go:1966 +0x5d7\ncreated by net/http.(*Server).Serve\n\tnet/http/server.go:3071 +0x4db"}

And just to make sure ive compiled it properly hers a grep of my xcaddy build for 'weidideng' to make sure im actually getting the correct module:

2022/09/02 17:15:15 [INFO] Replace github.com/caddyserver/caddy/v2 => github.com/WeidiDeng/caddy/v2@fastcgi-improve
2022/09/02 17:15:15 [INFO] exec (timeout=10s): /usr/local/go/bin/go mod edit -replace github.com/caddyserver/caddy/v2=github.com/WeidiDeng/caddy/v2@fastcgi-improve 
go: downloading github.com/WeidiDeng/caddy/v2 v2.5.2-0.20220902113148-74275bfe888e

So that looks alright

edit:

just noticed in the log that it seems to be crashing in the caddy cache module (souin) let me disable that and try again, still relevant im assuming tho if its crashing plugins?

Edit2:

Nope still the same with cache disabled:

caddy_1                | {"level":"debug","ts":1662139489.1957278,"logger":"http.stdlib","msg":"http: panic serving 172.20.0.1:36524: interface conversion: io.ReadCloser is fastcgi.clientCloser, not *fastcgi.clientCloser\ngoroutine 25 [running]:\nnet/http.(*conn).serve.func1()\n\tnet/http/server.go:1825 +0xbf\npanic({0x1db7aa0, 0xc00045c510})\n\truntime/panic.go:844 +0x258\ngithub.com/caddyserver/caddy/v2/modules/caddyhttp/reverseproxy/fastcgi.Transport.RoundTrip({{0xc0002dafd8, 0x15}, {0xc000649740, 0x1, 0x4}, 0x0, 0x0, 0xb2d05e00, 0x0, 0x0, ...}, ...)\n\tgithub.com/caddyserver/caddy/[email protected]/modules/caddyhttp/reverseproxy/fastcgi/fastcgi.go:196 +0xd9b\ngithub.com/caddyserver/caddy/v2/modules/caddyhttp/reverseproxy.(*Handler).reverseProxy(0xc00064a820, {0x7f09f4e45058?, 0xc000a02360}, 0xc000933000, 0xc000932f00, 0xc0007560a0, {0xc000640720, {0x204d31f, 0x3}, {0xc00066c9f0, ...}, ...}, ...)\n\tgithub.com/caddyserver/caddy/[email protected]/modules/caddyhttp/reverseproxy/reverseproxy.go:784 +0x8d7\ngithub.com/caddyserver/caddy/v2/modules/caddyhttp/reverseproxy.(*Handler).proxyLoopIteration(0xc00064a820, 0xc000933000, 0x4?, {0x7f09f4e45058, 0xc000a02360}, {0x0, 0x0}, {0xc0bcaceca1df9455, 0x11cdf85d9b, 0x3400fe0}, ...)\n\tgithub.com/caddyserver/caddy/[email protected]/modules/caddyhttp/reverseproxy/reverseproxy.go:546 +0xf10\ngithub.com/caddyserver/caddy/v2/modules/caddyhttp/reverseproxy.(*Handler).ServeHTTP(0xc00064a820, {0x7f09f4e45058, 0xc000a02360}, 0xc000932f00, {0x24835c0, 0xc000756160})\n\tgithub.com/caddyserver/caddy/[email protected]/modules/caddyhttp/reverseproxy/reverseproxy.go:454 +0x3dc\ngithub.com/caddyserver/caddy/v2/modules/caddyhttp.(*metricsInstrumentedHandler).ServeHTTP(0xc00009b560, {0x7f09f4e45058?, 0xc000a022a0}, 0xc000932f00, {0x24835c0, 0xc000756160})\n\tgithub.com/caddyserver/caddy/[email protected]/modules/caddyhttp/metrics.go:132 +0x53b\ngithub.com/caddyserver/caddy/v2/modules/caddyhttp.wrapMiddleware.func1.1({0x7f09f4e45058?, 0xc000a022a0?}, 0x24835c0?)\n\tgithub.com/caddyserver/caddy/[email protected]/modules/caddyhttp/routes.go:276 +0x3b\ngithub.com/caddyserver/caddy/v2/modules/caddyhttp.HandlerFunc.ServeHTTP(0x24835c0?, {0x7f09f4e45058?, 0xc000a022a0?}, 0xc0006493c0?)\n\tgithub.com/caddyserver/caddy/[email protected]/modules/caddyhttp/caddyhttp.go:58 +0x2f\ngithub.com/caddyserver/caddy/v2/modules/caddyhttp.wrapRoute.func1.1({0x7f09f4e45058, 0xc000a022a0}, 0xc000932f00)\n\tgithub.com/caddyserver/caddy/[email protected]/modules/caddyhttp/routes.go:248 +0x3a8\ngithub.com/caddyserver/caddy/v2/modules/caddyhttp.HandlerFunc.ServeHTTP(0x1d7dd60?, {0x7f09f4e45058?, 0xc000a022a0?}, 0xe?)\n\tgithub.com/caddyserver/caddy/[email protected]/modules/caddyhttp/caddyhttp.go:58 +0x2f\ngithub.com/caddyserver/caddy/v2/modules/caddyhttp.wrapRoute.func1.1({0x7f09f4e45058, 0xc000a022a0}, 0xc000932f00)\n\tgithub.com/caddyserver/caddy/[email protected]/modules/caddyhttp/routes.go:216 +0x336\ngithub.com/caddyserver/caddy/v2/modules/caddyhttp.HandlerFunc.ServeHTTP(0xc000610900?, {0x7f09f4e45058?, 0xc000a022a0?}, 0x24835c0?)\n\tgithub.com/caddyserver/caddy/[email protected]/modules/caddyhttp/caddyhttp.go:58 +0x2f\ngithub.com/caddyserver/caddy/v2/modules/caddyhttp.(*Subroute).ServeHTTP(0xc00009b1e0, {0x7f09f4e45058, 0xc000a022a0}, 0x6?, {0x24835c0, 0x2190960})\n\tgithub.com/caddyserver/caddy/[email protected]/modules/caddyhttp/subroute.go:74 +0x6d\ngithub.com/caddyserver/caddy/v2/modules/caddyhttp.(*metricsInstrumentedHandler).ServeHTTP(0xc00009b640, {0x2494840?, 0xc00013c9a0}, 0xc000932f00, {0x24835c0, 0x2190960})\n\tgithub.com/caddyserver/caddy/[email protected]/modules/caddyhttp/metrics.go:132 +0x53b\ngithub.com/caddyserver/caddy/v2/modules/caddyhttp.wrapMiddleware.func1.1({0x2494840?, 0xc00013c9a0?}, 0x24835c0?)\n\tgithub.com/caddyserver/caddy/[email protected]/modules/caddyhttp/routes.go:276 +0x3b\ngithub.com/caddyserver/caddy/v2/modules/caddyhttp.HandlerFunc.ServeHTTP(0x24835c0?, {0x2494840?, 0xc00013c9a0?}, 0xc00072a6c0?)\n\tgithub.com/caddyserver/caddy/[email protected]/modules/caddyhttp/caddyhttp.go:58 +0x2f\ngithub.com/caddyserver/caddy/v2/modules/caddyhttp.wrapRoute.func1.1({0x2494840, 0xc00013c9a0}, 0xc000932f00)\n\tgithub.com/caddyserver/caddy/[email protected]/modules/caddyhttp/routes.go:248 +0x3a8\ngithub.com/caddyserver/caddy/v2/modules/caddyhttp.HandlerFunc.ServeHTTP(0x1d7dd60?, {0x2494840?, 0xc00013c9a0?}, 0xe?)\n\tgithub.com/caddyserver/caddy/[email protected]/modules/caddyhttp/caddyhttp.go:58 +0x2f\ngithub.com/caddyserver/caddy/v2/modules/caddyhttp.wrapRoute.func1.1({0x2494840, 0xc00013c9a0}, 0xc000932f00)\n\tgithub.com/caddyserver/caddy/[email protected]/modules/caddyhttp/routes.go:216 +0x336\ngithub.com/caddyserver/caddy/v2/modules/caddyhttp.HandlerFunc.ServeHTTP(0x1b00?, {0x2494840?, 0xc00013c9a0?}, 0x1f8d4c0?)\n\tgithub.com/caddyserver/caddy/[email protected]/modules/caddyhttp/caddyhttp.go:58 +0x2f\ngithub.com/caddyserver/caddy/v2/modules/caddyhttp.(*Server).enforcementHandler(0x0?, {0x2494840?, 0xc00013c9a0?}, 0xc00067b680?, {0x24835c0?, 0xc00009bc60?})\n\tgithub.com/caddyserver/caddy/[email protected]/modules/caddyhttp/server.go:352 +0x252\ngithub.com/caddyserver/caddy/v2/modules/caddyhttp.(*Server).wrapPrimaryRoute.func1({0x2494840?, 0xc00013c9a0?}, 0x4c6ab7?)\n\tgithub.com/caddyserver/caddy/[email protected]/modules/caddyhttp/server.go:328 +0x3b\ngithub.com/caddyserver/caddy/v2/modules/caddyhttp.HandlerFunc.ServeHTTP(0xc0000b1040?, {0x2494840?, 0xc00013c9a0?}, 0xc000932f00?)\n\tgithub.com/caddyserver/caddy/[email protected]/modules/caddyhttp/caddyhttp.go:58 +0x2f\ngithub.com/caddyserver/caddy/v2/modules/caddyhttp.(*Server).ServeHTTP(0xc0002e4400, {0x2494840, 0xc00013c9a0}, 0xc000932f00)\n\tgithub.com/caddyserver/caddy/[email protected]/modules/caddyhttp/server.go:264 +0xb46\nnet/http.serverHandler.ServeHTTP({0xc0004e5cb0?}, {0x2494840, 0xc00013c9a0}, 0xc0000c1700)\n\tnet/http/server.go:2916 +0x43b\nnet/http.(*conn).serve(0xc00004e460, {0x24963f0, 0xc000672180})\n\tnet/http/server.go:1966 +0x5d7\ncreated by net/http.(*Server).Serve\n\tnet/http/server.go:3071 +0x4db"}

[mholt]

@mattvb91 Nice catch, thanks. @WeidiDeng I pushed a commit to your branch that should fix this panic: https://github.com/caddyserver/caddy/pull/4978/commits/ce2018222990b2b84683a767c61aaeece4748099