proposal: remove `{add,remove}-package` and `upgrade` commands from core

[mohammed90]

The {add,remove}-package and upgrade commands were added to experiment with providing users the ability to modify their executable as needed. Those commands call our build server, which operates the /download page, requesting a new binary with the requested modules. We typically see a huge uptick in requests to the build server soon after new releases which is assumed to be by users running caddy upgrade, though we cannot provide the data to back this claim. The build server presence is good for one-off requests where you need custom Caddy binary on the go, but it shouldn't be the main go-to source for CI/CD. However, we see references to it being part of CI/CD with periodic calls, which appears to be users' way of getting the latest version. This is an anti-pattern.

I propose we remove these commands from core and make them plugins, mostly to discourage their use. For three reasons:

1. They rely on our build server and encourage using it, which comes without SLA.
2. They centralize distribution to us, which is contrary to how optimal package distribution operates.
3. Their presence encourages mutable infrastructure, which is discouraged due to drift, lack of attestation, lack of reproducibility

The commands can be split into a plugin. I validated this with a PoC. The deprecation can be done in phases:

• v2.11: remove the commands from core Caddy, and make it only available through custom build with the explicit request of the plugin
• v2.12: full removal of the commands, plugin removed from site

From v2.12 onward, users who desire those functions must develop their own plugin and point them at their own build server. That said, the recommendation has always been to rely on xcaddy rather than the build server.

Previous conversation on the subject:

• State of add-packages and xcaddy

[mholt]

I'm probably good with this, since we can't guarantee the commands will work in the first place.

What if we found a sponsor to fund really solid build infrastructure?

[mohammed90]

What if we found a sponsor to fund really solid build infrastructure?

I don't know. I feel like it's a distraction from our main focus, which is building Caddy and the ecosystem. I think enhancing the documentation and the module registry is more valuable to the community and the ecosystem than the build server.

[Compy]

@mohammed90 Thanks for opening this up for discussion! I saw your forum post the other day and was meaning to inquire about it.

I'm building and testing a distributed Caddy build system.

Basically its a Go binary that can be run in one of the following modes:

• Standalone - One executable that serves the UI/API/runs xcaddy builds in separate goroutines. (wraps the xcaddy library)
• Hub - Serves the UI/API but dispatches builds to a redis queue
• Agent - Gets jobs from the redis queue and runs builds (wraps the xcaddy library)

A redis pubsub queue is used to relay live build log output back to the web interface via websocket.

This is all API compatible with the server you run at caddyserver.com, so it has an /api/download endpoint that accepts the same arguments, the same with /api/packages Even with distributed builds, the process will hold the HTTP request open and return the binary. The system also calculates a cache key based on the following:

1. OS/Architecture selected.
2. Caddy version (it gets the remote SHA of the current master head)
3. Packages selected (sorted)

Then each build is either stored on the local filesystem or in S3 compatible storage, so it can sling builds back quickly. The only real dependency it has directly on caddyserver.com is that it hits /api/packages once an hour to update its internal mirror of that package list.

The plan is to open source this so that people can either host it themselves as a reference implementation, or run a corporate sponsored build server using this system.

So what is the current Achilles heel of the build server at caddyserver.com? I imagine people running their CI builds against it is the bulk of it, which I imagine destroys your foot traffic for people using the web interface and wanting to quickly cobble a build together.

To @mholt 's point, I might be able to find a corporate sponsor, but I'd want some metrics on the current traffic that the build server is crumbling under. From what I gather, its a single (maybe fat) DigitalOcean box that's calling xcaddy library functions directly with each HTTP request (with maybe some sort of local caching option).

When this build server gets released, and provided it scratches a sufficient itch, perhaps a path forward would be to require the user to specify a build server endpoint with these commands. This allows users a path forward with minimal disruption to their current CI stack, and potentially reduces the cognitive load that Caddy maintainers have on keeping the build server running, and potentially still gives your users a quick and easy way to download builds using the web UI.

Jimmy

[mholt]

Thanks so much @Compy -- currently our build server is bottlenecked by CPU. Occasionally it's memory, but only the ridiculously-configured builds.

The build server can scale horizontally too... we use Caddy's reverse_proxy, so we can just add more smaller backends if that's cheaper and more reliable than one fat one.

Basically: website server looks in cache for requested build, if it doesn't exist, proxies a build request to one of the backends, which streams the artifact in the HTTP response body. The site server then caches it and serves it.

[mohammed90]

The module for the extracted commands is pushed here: https://github.com/caddyserver/mutable-caddy