Abilility for http middleware handler to configure http.server

[WeidiDeng]

Right now http.handler only needs to implement servehttp method, which is fine in most cases. However in some cases, for example, custom caddy app which also implements middleware may need to configure http.Server such as BaseContext, ConnState etc.

I have written a plugin which is similar to forward proxy. And since most of proxy connection is long lived, I have to register a http.Server OnShutdown function to close those when server configuration changes, or have to set a grace period which had lingering connection bug.

Right now I can access http.server instance through http.ServerContextKey and configure it there. But I think setting up a server is best done through an optional configure server method and before servehttp.

[mholt]

Interesting use case; usually, we'd probably want to configure the server by making the server itself pluggable. For example, we have a separate namespace for listener wrappers that configures the server's socket/listener. I think it's best for the http.handlers namespace to stay only HTTP handlers.

So maybe if we need to extend its BaseContext or ConnState, maybe we need a new plugin namespace (or multiple?) to set those up.

If you want to do something when the server is shutting down, you can implement the caddy.CleanerUpper interface: https://pkg.go.dev/github.com/caddyserver/caddy/v2#CleanerUpper

The CleanUp() method will be called when the config is reloaded or shutting down.

[WeidiDeng]

Interesting use case; usually, we'd probably want to configure the server by making the server itself pluggable. For example, we have a separate namespace for listener wrappers that configures the server's socket/listener. I think it's best for the http.handlers namespace to stay only HTTP handlers.

So maybe if we need to extend its BaseContext or ConnState, maybe we need a new plugin namespace (or multiple?) to set those up.

If you want to do something when the server is shutting down, you can implement the caddy.CleanerUpper interface: https://pkg.go.dev/github.com/caddyserver/caddy/v2#CleanerUpper

The CleanUp() method will be called when the config is reloaded or shutting down.

I tried to cleanup, but when caddy configuration changes, old config will shutdown. First all apps will shutdown, then cleanup will be called, http app shutdown involves shutting down http.server, which will wait indefinitely for request to complete or context error. But in practice, golang http.server has lingering connection bug, and it basically stuck even after context error. Cleanup will only be called after shutdown complete.

[WeidiDeng]

By the way, the lingering connection bug is like this. It causes remote api to be unresponsive. The server will behave like configuration not changed, and I can see from logs deadline excceeded. The only way is to kill caddy to unlock admin api. So grace_period is not an option.

The only way I found out is to register a server shutdown func to close those connection, which has worked great and there is no problem since.

Speaking of waiting for request to complete, because I am using http2 bidirectional streams, they're still managed by golang http.server activeConn, which unlike doing hijack, removes underlying connection from activeConn.

In this particular use case, I have no problem registering shutdown func, but I think when someone wants to use caddy as a platform to serve his own http apps, he can write a http.handler, doing his own routing logic, and just write http config parser, and it's done.

[mholt]

By the way, the lingering connection bug is like https://github.com/golang/go/issues/36093. It causes remote api to be unresponsive. The server will behave like configuration not changed, and I can see from logs deadline excceeded.

I've heard of this kind of bug before but never been able to see it myself. I'm not sure if the bug is in Go or in Caddy, or a little bit of both.

Actually, can you try Caddy on this branch? https://github.com/caddyserver/caddy/pull/4705 -- I am wondering if this will have any effect on the bug.