caddy `systemctl reload caddy.service` fails with log output `dial fd: unknown network fd` when using socket activation

I built git commit a211c656f12bcab73df0de114f2b6100ee5a0fe4 (master branch as of today)

Install caddy executable to /usr/local/bin/caddy
Run command
```
sudo groupadd --system caddy
```

Run command

sudo useradd --system \
   --gid caddy \
   --create-home \
   --home-dir /var/lib/caddy \
   --shell /usr/sbin/nologin \
   --comment "Caddy web server" \
   caddy`

Create file /etc/caddy/Caddyfile containing

{
	auto_https disable_redirects
	admin fd/4
}

http://test.example.com {
	bind fd/3 {
		protocols h1
	}
	log
	respond "hello world
"
}

Create file /etc/systemd/system/caddy.service containing

[Unit]
Description=Caddy
Documentation=https://caddyserver.com/docs/
After=network.target network-online.target
Requires=network-online.target

[Service]
Type=notify
User=caddy
Group=caddy
ExecStart=/usr/local/bin/caddy run --environ --config /etc/caddy/Caddyfile
ExecReload=/usr/local/bin/caddy reload --config /etc/caddy/Caddyfile --force
TimeoutStopSec=5s
LimitNOFILE=1048576
PrivateTmp=true
ProtectSystem=full
AmbientCapabilities=CAP_NET_ADMIN CAP_NET_BIND_SERVICE

[Install]
WantedBy=multi-user.target

It's the same content as https://github.com/caddyserver/dist/blob/34f51d87d16c3da70156edffdf7c52a6916f48e6/init/caddy.service#L16-L35 but /usr/bin/caddy replaced with /usr/local/bin/caddy

Create file /etc/systemd/system/caddy.socket

[Socket]
ListenStream=0.0.0.0:8077
ListenStream=%t/caddy.sock
[Install]
WantedBy=sockets.target

Run command
```
sudo systemctl daemon-reload
```
Run command
```
sudo systemctl start caddy.socket`
```

Run command

curl --resolve test.example.com:8077:127.0.0.1 http://test.example.com:8077

The following output is printed

hello world

Run command

sudo systemctl reload caddy.service

The following output is printed

Job for caddy.service failed.
See "systemctl status caddy.service" and "journalctl -xeu caddy.service" for details.

Run command

 sudo journalctl -xeu caddy.service --no-pager | tail -79

The following output is printed

Oct 15 17:26:25 fcos-next5 systemd[1]: Starting caddy.service - Caddy...
░░ Subject: A start job for unit caddy.service has begun execution
░░ Defined-By: systemd
░░ Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel
░░ 
░░ A start job for unit caddy.service has begun execution.
░░ 
░░ The job identifier is 44912.
Oct 15 17:26:26 fcos-next5 caddy[43845]: caddy.HomeDir=/var/lib/caddy
Oct 15 17:26:26 fcos-next5 caddy[43845]: caddy.AppDataDir=/var/lib/caddy/.local/share/caddy
Oct 15 17:26:26 fcos-next5 caddy[43845]: caddy.AppConfigDir=/var/lib/caddy/.config/caddy
Oct 15 17:26:26 fcos-next5 caddy[43845]: caddy.ConfigAutosavePath=/var/lib/caddy/.config/caddy/autosave.json
Oct 15 17:26:26 fcos-next5 caddy[43845]: caddy.Version=(devel)
Oct 15 17:26:26 fcos-next5 caddy[43845]: runtime.GOOS=linux
Oct 15 17:26:26 fcos-next5 caddy[43845]: runtime.GOARCH=arm64
Oct 15 17:26:26 fcos-next5 caddy[43845]: runtime.Compiler=gc
Oct 15 17:26:26 fcos-next5 caddy[43845]: runtime.NumCPU=2
Oct 15 17:26:26 fcos-next5 caddy[43845]: runtime.GOMAXPROCS=2
Oct 15 17:26:26 fcos-next5 caddy[43845]: runtime.Version=go1.23.2
Oct 15 17:26:26 fcos-next5 caddy[43845]: os.Getwd=/
Oct 15 17:26:26 fcos-next5 caddy[43845]: LANG=C.UTF-8
Oct 15 17:26:26 fcos-next5 caddy[43845]: PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin
Oct 15 17:26:26 fcos-next5 caddy[43845]: NOTIFY_SOCKET=/run/systemd/notify
Oct 15 17:26:26 fcos-next5 caddy[43845]: LISTEN_PID=43845
Oct 15 17:26:26 fcos-next5 caddy[43845]: LISTEN_FDS=2
Oct 15 17:26:26 fcos-next5 caddy[43845]: LISTEN_FDNAMES=caddy.socket:caddy.socket
Oct 15 17:26:26 fcos-next5 caddy[43845]: USER=caddy
Oct 15 17:26:26 fcos-next5 caddy[43845]: LOGNAME=caddy
Oct 15 17:26:26 fcos-next5 caddy[43845]: HOME=/var/lib/caddy
Oct 15 17:26:26 fcos-next5 caddy[43845]: INVOCATION_ID=3b389f56fa5645e19bcc47c90e4df4c0
Oct 15 17:26:26 fcos-next5 caddy[43845]: JOURNAL_STREAM=9:242885
Oct 15 17:26:26 fcos-next5 caddy[43845]: SYSTEMD_EXEC_PID=43845
Oct 15 17:26:26 fcos-next5 caddy[43845]: MEMORY_PRESSURE_WATCH=/sys/fs/cgroup/system.slice/caddy.service/memory.pressure
Oct 15 17:26:26 fcos-next5 caddy[43845]: MEMORY_PRESSURE_WRITE=c29tZSAyMDAwMDAgMjAwMDAwMAA=
Oct 15 17:26:26 fcos-next5 caddy[43845]: {"level":"info","ts":1729013186.1250536,"msg":"using config from file","file":"/etc/caddy/Caddyfile"}
Oct 15 17:26:26 fcos-next5 caddy[43845]: {"level":"info","ts":1729013186.133084,"msg":"adapted config to JSON","adapter":"caddyfile"}
Oct 15 17:26:26 fcos-next5 caddy[43845]: {"level":"info","ts":1729013186.1350574,"logger":"admin","msg":"admin endpoint started","address":"fd/4","enforce_origin":false,"origins":["","//127.0.0.1","//::1"]}
Oct 15 17:26:26 fcos-next5 caddy[43845]: {"level":"info","ts":1729013186.135701,"logger":"http.log","msg":"server running","name":"srv0","protocols":["h1","h2","h3"]}
Oct 15 17:26:26 fcos-next5 caddy[43845]: {"level":"info","ts":1729013186.1358926,"logger":"tls.cache.maintenance","msg":"started background certificate maintenance","cache":"0x400063cb00"}
Oct 15 17:26:26 fcos-next5 caddy[43845]: {"level":"info","ts":1729013186.1370368,"msg":"autosaved config (load with --resume flag)","file":"/var/lib/caddy/.config/caddy/autosave.json"}
Oct 15 17:26:26 fcos-next5 caddy[43845]: {"level":"info","ts":1729013186.137156,"msg":"serving initial configuration"}
Oct 15 17:26:26 fcos-next5 caddy[43845]: {"level":"info","ts":1729013186.1375623,"logger":"http.log.access","msg":"handled request","request":{"remote_ip":"127.0.0.1","remote_port":"55170","client_ip":"127.0.0.1","proto":"HTTP/1.1","method":"GET","host":"test.example.com:8077","uri":"/","headers":{"User-Agent":["curl/8.9.1"],"Accept":["*/*"]}},"bytes_read":0,"user_id":"","duration":0.000027332,"size":12,"status":200,"resp_headers":{"Server":["Caddy"],"Content-Type":["text/plain; charset=utf-8"]}}
Oct 15 17:26:26 fcos-next5 systemd[1]: Started caddy.service - Caddy.
░░ Subject: A start job for unit caddy.service has finished successfully
░░ Defined-By: systemd
░░ Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel
░░ 
░░ A start job for unit caddy.service has finished successfully.
░░ 
░░ The job identifier is 44912.
Oct 15 17:26:26 fcos-next5 caddy[43845]: {"level":"info","ts":1729013186.1550741,"logger":"tls","msg":"storage cleaning happened too recently; skipping for now","storage":"FileStorage:/var/lib/caddy/.local/share/caddy","instance":"896930ed-628f-4cbf-8f97-981466427717","try_again":1729099586.1550682,"try_again_in":86399.999999042}
Oct 15 17:26:26 fcos-next5 caddy[43845]: {"level":"info","ts":1729013186.1552908,"logger":"tls","msg":"finished cleaning storage units"}
Oct 15 17:26:39 fcos-next5 systemd[1]: Reloading caddy.service - Caddy...
░░ Subject: A reload job for unit caddy.service has begun execution
░░ Defined-By: systemd
░░ Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel
░░ 
░░ A reload job for unit caddy.service has begun execution.
░░ 
░░ The job identifier is 45045.
Oct 15 17:26:39 fcos-next5 caddy[43855]: {"level":"info","ts":1729013199.412728,"msg":"using config from file","file":"/etc/caddy/Caddyfile"}
Oct 15 17:26:39 fcos-next5 caddy[43855]: {"level":"info","ts":1729013199.4152887,"msg":"adapted config to JSON","adapter":"caddyfile"}
Oct 15 17:26:39 fcos-next5 caddy[43855]: Error: sending configuration to instance: performing request: Post "http://127.0.0.1/load": dial fd: unknown network fd
Oct 15 17:26:39 fcos-next5 systemd[1]: caddy.service: Control process exited, code=exited, status=1/FAILURE
░░ Subject: Unit process exited
░░ Defined-By: systemd
░░ Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel
░░ 
░░ An ExecReload= process belonging to unit caddy.service has exited.
░░ 
░░ The process' exit code is 'exited' and its exit status is 1.
Oct 15 17:26:39 fcos-next5 systemd[1]: Reload failed for caddy.service - Caddy.
░░ Subject: A reload job for unit caddy.service has finished
░░ Defined-By: systemd
░░ Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel
░░ 
░░ A reload job for unit caddy.service has finished.
░░ 
░░ The job identifier is 45045 and the job result is failed.

Describe the results you received

The command in Step 10 failed (sudo systemctl reload caddy.service)

In the log output there is the error message

 Error: sending configuration to instance: performing request: Post "http://127.0.0.1/load": dial fd: unknown network fd

Describe the results you expected

I would have expected the command in Step 10 to succeed.

Additional information

I tried the same reproducer but with /etc/systemd/system/caddy.socket having the following content

[Socket]
ListenStream=0.0.0.0:8077
ListenStream=%t/caddy.sock
SocketUser=caddy
SocketGroup=caddy
[Install]
WantedBy=sockets.target

The error was the same.

About the system

Fedora CoreOS 41.20241006.1.1

# rpm-ostree status
State: idle
AutomaticUpdatesDriver: Zincati
  DriverState: active; periodically polling for updates (last checked Tue 2024-10-15 17:10:11 UTC)
Deployments:
● fedora:fedora/aarch64/coreos/next
                  Version: 41.20241006.1.1 (2024-10-08T00:00:38Z)
               BaseCommit: 0dae4f8983c2c67147e4e44dcb9e0183ee8d1a301a2047fe30be33937727c7ed
             GPGSignature: Valid signature by 466CF2D8B60BC3057AA9453ED0622462E99D6AD1
      RemovedBasePackages: containerd 1.7.22-2.fc41 moby-engine 27.3.1-1.fc41 runc 2:1.1.12-4.fc41

  fedora:fedora/aarch64/coreos/next
                  Version: 40.20240906.1.0 (2024-09-10T00:50:59Z)
               BaseCommit: 0db203f6e5bdc3ec6ce57e200c84d4506e12e641c681f2212ef77e8019f1381b
             GPGSignature: Valid signature by 115DF9AEF857853EE8445D0A0727707EA15B79CC
      RemovedBasePackages: containerd 1.6.23-5.fc40 moby-engine 24.0.5-4.fc40 runc 2:1.1.12-3.fc40

Oct 15 '24 17:10 eriksjolund

So, this is a good question, how do we connect to "fd 4" from an external process?

Oct 18 '24 22:10 mholt

So, this is a good question, how do we connect to "fd 4" from an external process?

Is it possible to connect via /proc/$MAINPID/fd/4?

Nov 30 '24 23:11 septatrix

Otherwise it is also possible to alternatively implement Type=notify-reload in caddy.

Nov 30 '24 23:11 septatrix