caddy icon indicating copy to clipboard operation
caddy copied to clipboard

ACME broken for .internal adresses

Open Sani7 opened this issue 4 months ago • 39 comments

using caddy version v2.10.0 I'm using my own "lab.internal" dns names and have step-ca handing out certs for that domain, but all of a sudden I get errors like this:

{"level":"error","ts":1753810352.0772765,"logger":"tls.obtain","msg":"will retry","error":"[iot.lab.internal] Obtain: subject 'iot.lab.internal' does not qualify for a public certificate","attempt":5,"retrying_in":600,"elapsed":600.079823967,"max_duration":2592000}

Is there a way to configure caddy to use a certain acme_ca for DNS names that are not exposed to the internet?

global caddyfile

{
	# Our local ACME server
	acme_ca https://ca.lab.internal/acme/acme/directory
	# step-ca doens't support ocsp yet
	ocsp_stapling off
}

regular file

iot.lab.internal {
        reverse_proxy http://192.168.168.34:3000
}

Sani7 avatar Jul 29 '25 17:07 Sani7

Thanks, I can reproduce the bug; it's trying to use Let's Encrypt... will look into it.

mholt avatar Jul 29 '25 18:07 mholt

This should now be working as of https://github.com/caddyserver/caddy/commit/0badb071efc38bb9cc055076f0a48d1725fe8cc8 -- feel free to test it out!

mholt avatar Aug 04 '25 22:08 mholt

I upgraded to version 2.10.2 and still have the same issue

Sani7 avatar Sep 05 '25 19:09 Sani7

@Sani7 please show evidence. Show your config, your logs, etc.

francislavoie avatar Sep 05 '25 19:09 francislavoie

version string: v2.10.2 h1:g/gTYjGMD0dec+UgMw8SnfmJ3I9+M2TdvoRL/Ovu6U8= error string in systemd:

Sep 05 15:17:08 lab.internal caddy[1742800]: {"level":"error","ts":1757099828.2885928,"logger":"tls.obtain","msg":"will retry","error":"[iot.lab.internal] Obtain: subject 'iot.lab.internal' does not qualify for a public certificate","attempt":3,"retrying_in":120,"elapsed":180.019319989,"max_duration":2592000}

same config as before

Sani7 avatar Sep 05 '25 19:09 Sani7

I can't reproduce the bug anymore, so you'll have to help us reproduce it (or you can submit a patch).

Post your full logs as well. Sometimes people are using two versions of Caddy without realizing it.

Ideally, we need to be able to reproduce the bug in the most minimal way possible using the latest version of Caddy. This allows us to write regression tests to verify the fix is working. If we can't reproduce it, then you'll have to test our changes for us until it's fixed -- and then we can't add test cases, either.

I've attached a template below that will help make this easier and faster! This will require some effort on your part -- please understand that we will be dedicating time to fix the bug you are reporting if you can just help us understand it and reproduce it easily.

This template will ask for some information you've already provided; that's OK, just fill it out the best you can. :+1: I've also included some helpful tips below the template. Feel free to let me know if you have any questions!

Thank you again for your report, we look forward to resolving it!

Template

## 1. Environment

### 1a. Operating system and version

```
paste here
```


### 1b. Caddy version (run `caddy version` or paste commit SHA)

This should be the latest version of Caddy:

```
paste here
```


## 2. Description

### 2a. What happens (briefly explain what is wrong)




### 2b. Why it's a bug (if it's not obvious)




### 2c. Log output

```
paste terminal output or logs here
```



### 2d. Workaround(s)




### 2e. Relevant links




## 3. Tutorial (minimal steps to reproduce the bug)




Instructions -- please heed otherwise we cannot help you (help us help you!)

  1. Environment: Please fill out your OS and Caddy versions, even if you don't think they are relevant. (They are always relevant.) If you built Caddy from source, provide the commit SHA and specify your exact Go version.

  2. Description: Describe at a high level what the bug is. What happens? Why is it a bug? Not all bugs are obvious, so convince readers that it's actually a bug.

    • 2c) Log output: Paste terminal output and/or complete logs in a code block. DO NOT REDACT INFORMATION except for credentials. Please enable debug and access logs.
    • 2d) Workaround: What are you doing to work around the problem in the meantime? This can help others who encounter the same problem, until we implement a fix.
    • 2e) Relevant links: Please link to any related issues, pull requests, docs, and/or discussion. This can add crucial context to your report.
  3. Tutorial: What are the minimum required specific steps someone needs to take in order to experience the same bug? Your goal here is to make sure that anyone else can have the same experience with the bug as you do. You are writing a tutorial, so make sure to carry it out yourself before posting it. Please:

    • Start with an empty config. Add only the lines/parameters that are absolutely required to reproduce the bug.
    • Do not run Caddy inside containers.
    • Run Caddy manually in your terminal; do not use systemd or other init systems.
    • If making HTTP requests, avoid web browsers. Use a simpler HTTP client instead, like curl.
    • Do not redact any information from your config (except credentials). Domain names are public knowledge and often necessary for quick resolution of an issue!
    • Note that ignoring this advice may result in delays, or even in your issue being closed. 😞 Only actionable issues are kept open, and if there is not enough information or clarity to reproduce the bug, then the report is not actionable.

Example of a tutorial:

Create a config file:
{ ... }

Open terminal and run Caddy:

$ caddy ...

Make an HTTP request:

$ curl ...

Notice that the result is ___ but it should be ___.

mholt avatar Sep 05 '25 19:09 mholt

1. Environment

1a. Operating system and version

Fedora Linux 42 (Server Edition) aarch64 on raspberry pi 3b

1b. Caddy version (run caddy version or paste commit SHA)

This should be the latest version of Caddy:

v2.10.2 h1:g/gTYjGMD0dec+UgMw8SnfmJ3I9+M2TdvoRL/Ovu6U8=

2. Description

2a. What happens (briefly explain what is wrong)

Using step-ca as a local ACME server on a different device, configured as global acme-server, I get the error Obtain: subject 'iot.lab.internal' does not qualify for a public certificate

2b. Why it's a bug (if it's not obvious)

acme should work for all DNS addresses, including .internal, when using a custom acme server ( .internal DNS names are not accessible from outside the local network)

2c. Log output

Sep 05 15:47:56 lab.internal caddy[1744406]: {"level":"info","ts":1757101676.5152752,"logger":"tls.obtain","msg":"obtaining certificate","identifier":"iot.lab.internal"}
Sep 05 15:47:56 lab.internal caddy[1744406]: {"level":"error","ts":1757101676.5261343,"logger":"tls.obtain","msg":"will retry","error":"[iot.lab.internal] Obtain: subject 'iot.lab.internal' does not qualify for a public certificate","attempt":3,"retrying_in":120,"elapsed":180.019319989,"max_duration":2592000}

3. Tutorial (minimal steps to reproduce the bug)

Used this to set up my home lab back in the day: link to blog So custom DNS for lab.internal addresses that point to server with caddy as reverse proxy using a local acme-server to hand out certificates

Extra info

Last Known working version v2.8.4 h1:q3pe0wpBj1OcHFZ3n/1nl4V4bxBrYoSoab7rL9BMYNk= Definitely broken after version v2.9.0 h1:rteY8N18LsQn+2KVk6R10Vg/AlNsID1N/Ek9JLjm2yE=

Sani7 avatar Sep 05 '25 20:09 Sani7

Thanks. Can you ensure Caddy is running with --environ and then post the full log output?

mholt avatar Sep 05 '25 21:09 mholt

Good Morning @Sani7 @mholt, I have a similar issue but with a slightly different config.

Step-Ca is my acme server, too. But i want running dns-01 challenge as well as http-01 challenge. So i set up bind9 dns server with the appropriate config there.

The dns challenge works very well, i think this is something what you fixed above, right? The http challenge dont work. Its using the local acme issuer. All Domain names in dns server are pointing to caddy server.

Maybe it is by design? .internal domains arent issued against acme_ca in general? The docs hinting this if i understand on the right way. https://caddyserver.com/docs/automatic-https#hostname-requirements

in the Caddyfile below i want to challenge nc and wiki via dns-01 and test.example.internal via http-01. nc and wiki works well, test.example.internal do not. its issued by local. (i checked /var/lib/caddy/.local/share/caddy/certificates directory) Between caddy server, dns and step-ca are no firewall. so there should nothing be blocked.

But if i change the test.example.internal to test.example.com i can see in the logs that caddy tries to solve the challenge aganst acme_ca. It does not finished because i had no dns record for that, but this let me think that my thesis could be correct :-)

OS:

Debian Bookworm
Linux caddy 6.1.0-23-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.1.99-1 (2024-07-15) x86_64

Caddy Version with rf2136 dns plugin:

v2.10.2 h1:g/gTYjGMD0dec+UgMw8SnfmJ3I9+M2TdvoRL/Ovu6U8=

CaddyFile:

# global
{
    email [email protected]
    acme_ca https://ca.example.internal:8443/acme/acme/directory
    acme_ca_root /etc/ssl/certs/ca-certificates.crt   # ca root cert is bundled in systems trust store
}

# reusable snippet for dns-01 challenge. So we can use other challenges on other hosts when needed.
(acme-dns-challenge) {
    tls {
        dns rfc2136 {
            key_name "_acme"
            key_alg "hmac-sha512"
            key "MYSUPERSECRET"
            server "192.168.1.82:53"
        }
    }
}

(skip-insecure-tls) {
    transport http {
        tls
        tls_insecure_skip_verify
    }
}

nc.example.internal {
    import acme-dns-challenge
    redir /.well-known/carddav /remote.php/dav/ 301
    redir /.well-known/caldav /remote.php/dav/ 301

    reverse_proxy 192.168.1.3:443 {
        import skip-insecure-tls
    }

}

wiki.example.internal {
    import acme-dns-challenge
    reverse_proxy 192.168.1.4:443 {
        import skip-insecure-tls
    }
}

test.example.internal {
    reverse_proxy 192.168.1.4:80
}

looking forward for feedback :-)

Chris

ChrisChoke avatar Oct 10 '25 05:10 ChrisChoke

Hi @mholt Sorry for the long wait. My complete log output is this. Before was a grab from systemd-log I also updated to version v2.10.2 h1:g/gTYjGMD0dec+UgMw8SnfmJ3I9+M2TdvoRL/Ovu6U8= @ChrisChoke, I think we have the same issue

2025/10/10 11:02:20.877	INFO	maxprocs: Leaving GOMAXPROCS=4: CPU quota undefined
2025/10/10 11:02:20.878	INFO	GOMEMLIMIT is updated	{"package": "github.com/KimMachineGun/automemlimit/memlimit", "GOMEMLIMIT": 819250790, "previous": 9223372036854775807}
caddy.HomeDir=/root
caddy.AppDataDir=/root/.local/share/caddy
caddy.AppConfigDir=/root/.config/caddy
caddy.ConfigAutosavePath=/root/.config/caddy/autosave.json
caddy.Version=v2.10.2 h1:g/gTYjGMD0dec+UgMw8SnfmJ3I9+M2TdvoRL/Ovu6U8=
runtime.GOOS=linux
runtime.GOARCH=arm64
runtime.Compiler=gc
runtime.NumCPU=4
runtime.GOMAXPROCS=4
runtime.Version=go1.25.0
os.Getwd=/home/sanderspeetjens

HISTSIZE=1000
HOSTNAME=lab.internal
LANG=en_US.UTF-8
LS_COLORS=rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:mi=01;37;41:su=37;41:sg=30;43:ca=00:tw=30;42:ow=34;42:st=37;44:ex=01;32:*.7z=01;31:*.ace=01;31:*.alz=01;31:*.apk=01;31:*.arc=01;31:*.arj=01;31:*.bz=01;31:*.bz2=01;31:*.cab=01;31:*.cpio=01;31:*.crate=01;31:*.deb=01;31:*.drpm=01;31:*.dwm=01;31:*.dz=01;31:*.ear=01;31:*.egg=01;31:*.esd=01;31:*.gz=01;31:*.jar=01;31:*.lha=01;31:*.lrz=01;31:*.lz=01;31:*.lz4=01;31:*.lzh=01;31:*.lzma=01;31:*.lzo=01;31:*.pyz=01;31:*.rar=01;31:*.rpm=01;31:*.rz=01;31:*.sar=01;31:*.swm=01;31:*.t7z=01;31:*.tar=01;31:*.taz=01;31:*.tbz=01;31:*.tbz2=01;31:*.tgz=01;31:*.tlz=01;31:*.txz=01;31:*.tz=01;31:*.tzo=01;31:*.tzst=01;31:*.udeb=01;31:*.war=01;31:*.whl=01;31:*.wim=01;31:*.xz=01;31:*.z=01;31:*.zip=01;31:*.zoo=01;31:*.zst=01;31:*.avif=01;35:*.jpg=01;35:*.jpeg=01;35:*.mjpg=01;35:*.mjpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.svg=01;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01;35:*.m2v=01;35:*.mkv=01;35:*.webm=01;35:*.webp=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35:*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;35:*.rmvb=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.flv=01;35:*.gl=01;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.cgm=01;35:*.emf=01;35:*.ogv=01;35:*.ogx=01;35:*.aac=01;36:*.au=01;36:*.flac=01;36:*.m4a=01;36:*.mid=01;36:*.midi=01;36:*.mka=01;36:*.mp3=01;36:*.mpc=01;36:*.ogg=01;36:*.ra=01;36:*.wav=01;36:*.oga=01;36:*.opus=01;36:*.spx=01;36:*.xspf=01;36:*~=00;90:*#=00;90:*.bak=00;90:*.crdownload=00;90:*.dpkg-dist=00;90:*.dpkg-new=00;90:*.dpkg-old=00;90:*.dpkg-tmp=00;90:*.old=00;90:*.orig=00;90:*.part=00;90:*.rej=00;90:*.rpmnew=00;90:*.rpmorig=00;90:*.rpmsave=00;90:*.swp=00;90:*.tmp=00;90:*.ucf-dist=00;90:*.ucf-new=00;90:*.ucf-old=00;90:
TERM=xterm-256color
MAIL=/var/spool/mail/sanderspeetjens
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/var/lib/snapd/snap/bin
LOGNAME=root
USER=root
HOME=/root
SHELL=/bin/bash
SUDO_COMMAND=/usr/sbin/caddy run --environ --config /etc/caddy/Caddyfile
SUDO_USER=sanderspeetjens
SUDO_UID=1000
SUDO_GID=1000
SUDO_HOME=/home/sanderspeetjens
SUDO_TTY=/dev/pts/0
2025/10/10 11:02:20.880	INFO	using config from file	{"file": "/etc/caddy/Caddyfile"}
2025/10/10 11:02:20.896	INFO	adapted config to JSON	{"adapter": "caddyfile"}
2025/10/10 11:02:20.907	INFO	admin	admin endpoint started	{"address": "localhost:2019", "enforce_origin": false, "origins": ["//localhost:2019", "//[::1]:2019", "//127.0.0.1:2019"]}
2025/10/10 11:02:20.909	INFO	tls.cache.maintenance	started background certificate maintenance	{"cache": "0x40006a6700"}
2025/10/10 11:02:20.909	INFO	http.auto_https	server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS	{"server_name": "srv0", "https_port": 443}
2025/10/10 11:02:20.909	INFO	http.auto_https	enabling automatic HTTP->HTTPS redirects	{"server_name": "srv0"}
2025/10/10 11:02:20.910	WARN	http.auto_https	server is listening only on the HTTP port, so no automatic HTTPS will be applied to this server	{"server_name": "srv1", "http_port": 80}
2025/10/10 11:02:21.000	WARN	http	HTTP/2 skipped because it requires TLS	{"network": "tcp", "addr": ":80"}
2025/10/10 11:02:21.000	WARN	http	HTTP/3 skipped because it requires TLS	{"network": "tcp", "addr": ":80"}
2025/10/10 11:02:21.001	INFO	http.log	server running	{"name": "srv1", "protocols": ["h1", "h2", "h3"]}
2025/10/10 11:02:21.001	INFO	http	enabling HTTP/3 listener	{"addr": ":443"}
2025/10/10 11:02:21.003	INFO	http.log	server running	{"name": "srv0", "protocols": ["h1", "h2", "h3"]}
2025/10/10 11:02:21.003	INFO	http	enabling automatic TLS certificate management	{"domains": ["partdb.lab.internal", "iot.lab.internal", "ntfy.lab.internal", "transmission.lab.internal", "cockpit.lab.internal", "fritz.lab.internal", "gogs.lab.internal", "plex.lab.internal", "cockpit.prometheus.lab.internal"]}
2025/10/10 11:02:21.005	WARN	pki.ca.local	installing root certificate (you might be prompted for password)	{"path": "storage:pki/authorities/local/root.crt"}
2025/10/10 11:02:21.006	INFO	not NSS security databases found
2025/10/10 11:02:21.006	INFO	define JAVA_HOME environment variable to use the Java trust
2025/10/10 11:02:21.031	INFO	tls.obtain	acquiring lock	{"identifier": "cockpit.lab.internal"}
2025/10/10 11:02:21.064	INFO	tls.obtain	acquiring lock	{"identifier": "iot.lab.internal"}
2025/10/10 11:02:21.064	INFO	tls.obtain	acquiring lock	{"identifier": "partdb.lab.internal"}
2025/10/10 11:02:21.065	INFO	tls.obtain	acquiring lock	{"identifier": "transmission.lab.internal"}
2025/10/10 11:02:21.078	INFO	tls.obtain	acquiring lock	{"identifier": "fritz.lab.internal"}
2025/10/10 11:02:21.086	INFO	tls.obtain	acquiring lock	{"identifier": "plex.lab.internal"}
2025/10/10 11:02:21.094	INFO	tls.obtain	lock acquired	{"identifier": "cockpit.lab.internal"}
2025/10/10 11:02:21.095	INFO	tls.obtain	obtaining certificate	{"identifier": "cockpit.lab.internal"}
2025/10/10 11:02:21.099	ERROR	tls.obtain	will retry	{"error": "[cockpit.lab.internal] Obtain: subject 'cockpit.lab.internal' does not qualify for a public certificate", "attempt": 1, "retrying_in": 60, "elapsed": 0.004832166, "max_duration": 2592000}
2025/10/10 11:02:21.104	INFO	tls.obtain	lock acquired	{"identifier": "transmission.lab.internal"}
2025/10/10 11:02:21.105	INFO	tls.obtain	obtaining certificate	{"identifier": "transmission.lab.internal"}
2025/10/10 11:02:21.111	ERROR	tls.obtain	will retry	{"error": "[transmission.lab.internal] Obtain: subject 'transmission.lab.internal' does not qualify for a public certificate", "attempt": 1, "retrying_in": 60, "elapsed": 0.006701845, "max_duration": 2592000}
2025/10/10 11:02:21.120	INFO	tls.obtain	acquiring lock	{"identifier": "ntfy.lab.internal"}
2025/10/10 11:02:21.120	INFO	tls.obtain	acquiring lock	{"identifier": "cockpit.prometheus.lab.internal"}
2025/10/10 11:02:21.138	INFO	tls.obtain	acquiring lock	{"identifier": "gogs.lab.internal"}
2025/10/10 11:02:21.172	INFO	tls.obtain	lock acquired	{"identifier": "iot.lab.internal"}
2025/10/10 11:02:21.172	INFO	tls.obtain	lock acquired	{"identifier": "fritz.lab.internal"}
2025/10/10 11:02:21.173	INFO	tls.obtain	obtaining certificate	{"identifier": "iot.lab.internal"}
2025/10/10 11:02:21.173	INFO	tls.obtain	obtaining certificate	{"identifier": "fritz.lab.internal"}
2025/10/10 11:02:21.176	ERROR	tls.obtain	will retry	{"error": "[fritz.lab.internal] Obtain: subject 'fritz.lab.internal' does not qualify for a public certificate", "attempt": 1, "retrying_in": 60, "elapsed": 0.003183579, "max_duration": 2592000}
2025/10/10 11:02:21.176	ERROR	tls.obtain	will retry	{"error": "[iot.lab.internal] Obtain: subject 'iot.lab.internal' does not qualify for a public certificate", "attempt": 1, "retrying_in": 60, "elapsed": 0.003966128, "max_duration": 2592000}
2025/10/10 11:02:21.178	INFO	tls.obtain	lock acquired	{"identifier": "partdb.lab.internal"}
2025/10/10 11:02:21.178	INFO	tls.obtain	obtaining certificate	{"identifier": "partdb.lab.internal"}
2025/10/10 11:02:21.182	INFO	tls.obtain	lock acquired	{"identifier": "plex.lab.internal"}
2025/10/10 11:02:21.183	INFO	tls.obtain	lock acquired	{"identifier": "gogs.lab.internal"}
2025/10/10 11:02:21.184	INFO	tls.obtain	obtaining certificate	{"identifier": "plex.lab.internal"}
2025/10/10 11:02:21.183	INFO	tls.obtain	lock acquired	{"identifier": "ntfy.lab.internal"}
2025/10/10 11:02:21.185	INFO	tls.obtain	obtaining certificate	{"identifier": "ntfy.lab.internal"}
2025/10/10 11:02:21.183	INFO	tls.obtain	lock acquired	{"identifier": "cockpit.prometheus.lab.internal"}
2025/10/10 11:02:21.186	INFO	tls	cleaning storage unit	{"storage": "FileStorage:/root/.local/share/caddy"}
2025/10/10 11:02:21.186	INFO	tls.obtain	obtaining certificate	{"identifier": "cockpit.prometheus.lab.internal"}
2025/10/10 11:02:21.184	ERROR	tls.obtain	will retry	{"error": "[partdb.lab.internal] Obtain: subject 'partdb.lab.internal' does not qualify for a public certificate", "attempt": 1, "retrying_in": 60, "elapsed": 0.005583151, "max_duration": 2592000}
2025/10/10 11:02:21.189	ERROR	tls.obtain	will retry	{"error": "[plex.lab.internal] Obtain: subject 'plex.lab.internal' does not qualify for a public certificate", "attempt": 1, "retrying_in": 60, "elapsed": 0.004737426, "max_duration": 2592000}
2025/10/10 11:02:21.185	INFO	tls.obtain	obtaining certificate	{"identifier": "gogs.lab.internal"}
2025/10/10 11:02:21.192	ERROR	tls.obtain	will retry	{"error": "[ntfy.lab.internal] Obtain: subject 'ntfy.lab.internal' does not qualify for a public certificate", "attempt": 1, "retrying_in": 60, "elapsed": 0.00766908, "max_duration": 2592000}
2025/10/10 11:02:21.194	ERROR	tls.obtain	will retry	{"error": "[gogs.lab.internal] Obtain: subject 'gogs.lab.internal' does not qualify for a public certificate", "attempt": 1, "retrying_in": 60, "elapsed": 0.00912699, "max_duration": 2592000}
2025/10/10 11:02:21.194	ERROR	tls.obtain	will retry	{"error": "[cockpit.prometheus.lab.internal] Obtain: subject 'cockpit.prometheus.lab.internal' does not qualify for a public certificate", "attempt": 1, "retrying_in": 60, "elapsed": 0.008381681, "max_duration": 2592000}
2025/10/10 11:02:21.199	INFO	tls	finished cleaning storage units
2025/10/10 11:02:41.576	INFO	certificate installed properly in linux trusts
2025/10/10 11:02:41.580	INFO	autosaved config (load with --resume flag)	{"file": "/root/.config/caddy/autosave.json"}
2025/10/10 11:02:41.580	INFO	serving initial configuration
2025/10/10 11:03:21.101	INFO	tls.obtain	obtaining certificate	{"identifier": "cockpit.lab.internal"}
2025/10/10 11:03:21.105	ERROR	tls.obtain	will retry	{"error": "[cockpit.lab.internal] Obtain: subject 'cockpit.lab.internal' does not qualify for a public certificate", "attempt": 2, "retrying_in": 120, "elapsed": 60.010909638, "max_duration": 2592000}
2025/10/10 11:03:21.112	INFO	tls.obtain	obtaining certificate	{"identifier": "transmission.lab.internal"}
2025/10/10 11:03:21.117	ERROR	tls.obtain	will retry	{"error": "[transmission.lab.internal] Obtain: subject 'transmission.lab.internal' does not qualify for a public certificate", "attempt": 2, "retrying_in": 120, "elapsed": 60.013089733, "max_duration": 2592000}
2025/10/10 11:03:21.177	INFO	tls.obtain	obtaining certificate	{"identifier": "fritz.lab.internal"}
2025/10/10 11:03:21.178	INFO	tls.obtain	obtaining certificate	{"identifier": "iot.lab.internal"}
2025/10/10 11:03:21.181	ERROR	tls.obtain	will retry	{"error": "[fritz.lab.internal] Obtain: subject 'fritz.lab.internal' does not qualify for a public certificate", "attempt": 2, "retrying_in": 120, "elapsed": 60.008585534, "max_duration": 2592000}
2025/10/10 11:03:21.182	ERROR	tls.obtain	will retry	{"error": "[iot.lab.internal] Obtain: subject 'iot.lab.internal' does not qualify for a public certificate", "attempt": 2, "retrying_in": 120, "elapsed": 60.010138287, "max_duration": 2592000}
2025/10/10 11:03:21.191	INFO	tls.obtain	obtaining certificate	{"identifier": "plex.lab.internal"}
2025/10/10 11:03:21.191	INFO	tls.obtain	obtaining certificate	{"identifier": "partdb.lab.internal"}
2025/10/10 11:03:21.194	INFO	tls.obtain	obtaining certificate	{"identifier": "ntfy.lab.internal"}
2025/10/10 11:03:21.196	INFO	tls.obtain	obtaining certificate	{"identifier": "gogs.lab.internal"}
2025/10/10 11:03:21.196	ERROR	tls.obtain	will retry	{"error": "[plex.lab.internal] Obtain: subject 'plex.lab.internal' does not qualify for a public certificate", "attempt": 2, "retrying_in": 120, "elapsed": 60.01189932, "max_duration": 2592000}
2025/10/10 11:03:21.196	ERROR	tls.obtain	will retry	{"error": "[partdb.lab.internal] Obtain: subject 'partdb.lab.internal' does not qualify for a public certificate", "attempt": 2, "retrying_in": 120, "elapsed": 60.017742054, "max_duration": 2592000}
2025/10/10 11:03:21.196	INFO	tls.obtain	obtaining certificate	{"identifier": "cockpit.prometheus.lab.internal"}
2025/10/10 11:03:21.198	ERROR	tls.obtain	will retry	{"error": "[ntfy.lab.internal] Obtain: subject 'ntfy.lab.internal' does not qualify for a public certificate", "attempt": 2, "retrying_in": 120, "elapsed": 60.01355822, "max_duration": 2592000}
2025/10/10 11:03:21.201	ERROR	tls.obtain	will retry	{"error": "[gogs.lab.internal] Obtain: subject 'gogs.lab.internal' does not qualify for a public certificate", "attempt": 2, "retrying_in": 120, "elapsed": 60.016795705, "max_duration": 2592000}
2025/10/10 11:03:21.204	ERROR	tls.obtain	will retry	{"error": "[cockpit.prometheus.lab.internal] Obtain: subject 'cockpit.prometheus.lab.internal' does not qualify for a public certificate", "attempt": 2, "retrying_in": 120, "elapsed": 60.018552833, "max_duration": 2592000}

Sani7 avatar Oct 10 '25 11:10 Sani7

This is a log of version v2.8.4 h1:q3pe0wpBj1OcHFZ3n/1nl4V4bxBrYoSoab7rL9BMYNk= that works

caddy.HomeDir=/root
caddy.AppDataDir=/root/.local/share/caddy
caddy.AppConfigDir=/root/.config/caddy
caddy.ConfigAutosavePath=/root/.config/caddy/autosave.json
caddy.Version=v2.8.4 h1:q3pe0wpBj1OcHFZ3n/1nl4V4bxBrYoSoab7rL9BMYNk=
runtime.GOOS=linux
runtime.GOARCH=arm64
runtime.Compiler=gc
runtime.NumCPU=4
runtime.GOMAXPROCS=4
runtime.Version=go1.22.3
os.Getwd=/home/sanderspeetjens

HISTSIZE=1000
HOSTNAME=lab.internal
LANG=en_US.UTF-8
LS_COLORS=rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:mi=01;37;41:su=37;41:sg=30;43:ca=00:tw=30;42:ow=34;42:st=37;44:ex=01;32:*.7z=01;31:*.ace=01;31:*.alz=01;31:*.apk=01;31:*.arc=01;31:*.arj=01;31:*.bz=01;31:*.bz2=01;31:*.cab=01;31:*.cpio=01;31:*.crate=01;31:*.deb=01;31:*.drpm=01;31:*.dwm=01;31:*.dz=01;31:*.ear=01;31:*.egg=01;31:*.esd=01;31:*.gz=01;31:*.jar=01;31:*.lha=01;31:*.lrz=01;31:*.lz=01;31:*.lz4=01;31:*.lzh=01;31:*.lzma=01;31:*.lzo=01;31:*.pyz=01;31:*.rar=01;31:*.rpm=01;31:*.rz=01;31:*.sar=01;31:*.swm=01;31:*.t7z=01;31:*.tar=01;31:*.taz=01;31:*.tbz=01;31:*.tbz2=01;31:*.tgz=01;31:*.tlz=01;31:*.txz=01;31:*.tz=01;31:*.tzo=01;31:*.tzst=01;31:*.udeb=01;31:*.war=01;31:*.whl=01;31:*.wim=01;31:*.xz=01;31:*.z=01;31:*.zip=01;31:*.zoo=01;31:*.zst=01;31:*.avif=01;35:*.jpg=01;35:*.jpeg=01;35:*.mjpg=01;35:*.mjpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.svg=01;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01;35:*.m2v=01;35:*.mkv=01;35:*.webm=01;35:*.webp=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35:*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;35:*.rmvb=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.flv=01;35:*.gl=01;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.cgm=01;35:*.emf=01;35:*.ogv=01;35:*.ogx=01;35:*.aac=01;36:*.au=01;36:*.flac=01;36:*.m4a=01;36:*.mid=01;36:*.midi=01;36:*.mka=01;36:*.mp3=01;36:*.mpc=01;36:*.ogg=01;36:*.ra=01;36:*.wav=01;36:*.oga=01;36:*.opus=01;36:*.spx=01;36:*.xspf=01;36:*~=00;90:*#=00;90:*.bak=00;90:*.crdownload=00;90:*.dpkg-dist=00;90:*.dpkg-new=00;90:*.dpkg-old=00;90:*.dpkg-tmp=00;90:*.old=00;90:*.orig=00;90:*.part=00;90:*.rej=00;90:*.rpmnew=00;90:*.rpmorig=00;90:*.rpmsave=00;90:*.swp=00;90:*.tmp=00;90:*.ucf-dist=00;90:*.ucf-new=00;90:*.ucf-old=00;90:
TERM=xterm-256color
MAIL=/var/spool/mail/sanderspeetjens
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/var/lib/snapd/snap/bin
LOGNAME=root
USER=root
HOME=/root
SHELL=/bin/bash
SUDO_COMMAND=/usr/sbin/caddy run --environ --config /etc/caddy/Caddyfile
SUDO_USER=sanderspeetjens
SUDO_UID=1000
SUDO_GID=1000
SUDO_HOME=/home/sanderspeetjens
SUDO_TTY=/dev/pts/0
2025/10/10 11:15:04.127	INFO	using config from file	{"file": "/etc/caddy/Caddyfile"}
2025/10/10 11:15:04.168	INFO	adapted config to JSON	{"adapter": "caddyfile"}
2025/10/10 11:15:04.183	INFO	admin	admin endpoint started	{"address": "localhost:2019", "enforce_origin": false, "origins": ["//localhost:2019", "//[::1]:2019", "//127.0.0.1:2019"]}
2025/10/10 11:15:04.185	INFO	tls.cache.maintenance	started background certificate maintenance	{"cache": "0x4000898f80"}
2025/10/10 11:15:04.186	INFO	http.auto_https	server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS	{"server_name": "srv0", "https_port": 443}
2025/10/10 11:15:04.186	INFO	http.auto_https	enabling automatic HTTP->HTTPS redirects	{"server_name": "srv0"}
2025/10/10 11:15:04.186	WARN	http.auto_https	server is listening only on the HTTP port, so no automatic HTTPS will be applied to this server	{"server_name": "srv1", "http_port": 80}
2025/10/10 11:15:04.207	INFO	http	enabling HTTP/3 listener	{"addr": ":443"}
2025/10/10 11:15:04.208	INFO	http.log	server running	{"name": "srv0", "protocols": ["h1", "h2", "h3"]}
2025/10/10 11:15:04.209	INFO	http.log	server running	{"name": "srv1", "protocols": ["h1", "h2", "h3"]}
2025/10/10 11:15:04.209	INFO	http	enabling automatic TLS certificate management	{"domains": ["partdb.lab.internal", "transmission.lab.internal", "cockpit.lab.internal", "iot.lab.internal", "cockpit.prometheus.lab.internal", "fritz.lab.internal", "plex.lab.internal", "gogs.lab.internal", "ntfy.lab.internal"]}
2025/10/10 11:15:04.212	INFO	autosaved config (load with --resume flag)	{"file": "/root/.config/caddy/autosave.json"}
2025/10/10 11:15:04.212	INFO	serving initial configuration
2025/10/10 11:15:04.216	INFO	tls.obtain	acquiring lock	{"identifier": "partdb.lab.internal"}
2025/10/10 11:15:04.218	INFO	tls.obtain	acquiring lock	{"identifier": "cockpit.prometheus.lab.internal"}
2025/10/10 11:15:04.220	INFO	tls.obtain	acquiring lock	{"identifier": "cockpit.lab.internal"}
2025/10/10 11:15:04.219	INFO	tls.obtain	acquiring lock	{"identifier": "iot.lab.internal"}
2025/10/10 11:15:04.223	INFO	tls.obtain	acquiring lock	{"identifier": "plex.lab.internal"}
2025/10/10 11:15:04.224	INFO	tls.obtain	acquiring lock	{"identifier": "fritz.lab.internal"}
2025/10/10 11:15:04.224	INFO	tls.obtain	acquiring lock	{"identifier": "gogs.lab.internal"}
2025/10/10 11:15:04.227	INFO	tls.obtain	acquiring lock	{"identifier": "transmission.lab.internal"}
2025/10/10 11:15:04.228	INFO	tls.obtain	acquiring lock	{"identifier": "ntfy.lab.internal"}
2025/10/10 11:15:04.243	INFO	tls	storage cleaning happened too recently; skipping for now	{"storage": "FileStorage:/root/.local/share/caddy", "instance": "c6862f73-f372-4596-b5c1-79755dd9c8b6", "try_again": "2025/10/11 11:15:04.243", "try_again_in": 86399.999997031}
2025/10/10 11:15:04.243	INFO	tls	finished cleaning storage units
2025/10/10 11:15:04.256	INFO	tls.obtain	lock acquired	{"identifier": "partdb.lab.internal"}
2025/10/10 11:15:04.257	INFO	tls.obtain	obtaining certificate	{"identifier": "partdb.lab.internal"}
2025/10/10 11:15:04.261	INFO	tls.obtain	lock acquired	{"identifier": "cockpit.prometheus.lab.internal"}
2025/10/10 11:15:04.262	INFO	tls.obtain	obtaining certificate	{"identifier": "cockpit.prometheus.lab.internal"}
2025/10/10 11:15:04.274	INFO	tls.obtain	lock acquired	{"identifier": "ntfy.lab.internal"}
2025/10/10 11:15:04.274	INFO	tls.obtain	lock acquired	{"identifier": "plex.lab.internal"}
2025/10/10 11:15:04.274	INFO	tls.obtain	obtaining certificate	{"identifier": "ntfy.lab.internal"}
2025/10/10 11:15:04.275	INFO	tls.obtain	obtaining certificate	{"identifier": "plex.lab.internal"}
2025/10/10 11:15:04.274	INFO	tls.obtain	lock acquired	{"identifier": "iot.lab.internal"}
2025/10/10 11:15:04.276	INFO	tls.obtain	obtaining certificate	{"identifier": "iot.lab.internal"}
2025/10/10 11:15:04.274	INFO	tls.obtain	lock acquired	{"identifier": "gogs.lab.internal"}
2025/10/10 11:15:04.274	INFO	tls.obtain	lock acquired	{"identifier": "transmission.lab.internal"}
2025/10/10 11:15:04.278	INFO	tls.obtain	lock acquired	{"identifier": "fritz.lab.internal"}
2025/10/10 11:15:04.280	INFO	tls.obtain	obtaining certificate	{"identifier": "transmission.lab.internal"}
2025/10/10 11:15:04.281	INFO	tls.obtain	obtaining certificate	{"identifier": "fritz.lab.internal"}
2025/10/10 11:15:04.280	INFO	tls.obtain	obtaining certificate	{"identifier": "gogs.lab.internal"}
2025/10/10 11:15:04.274	INFO	tls.obtain	lock acquired	{"identifier": "cockpit.lab.internal"}
2025/10/10 11:15:04.284	INFO	tls.obtain	obtaining certificate	{"identifier": "cockpit.lab.internal"}
2025/10/10 11:15:04.372	INFO	http	waiting on internal rate limiter	{"identifiers": ["partdb.lab.internal"], "ca": "https://ca.lab.internal/acme/acme/directory", "account": ""}
2025/10/10 11:15:04.373	INFO	http	done waiting on internal rate limiter	{"identifiers": ["partdb.lab.internal"], "ca": "https://ca.lab.internal/acme/acme/directory", "account": ""}
2025/10/10 11:15:04.373	INFO	http	waiting on internal rate limiter	{"identifiers": ["cockpit.prometheus.lab.internal"], "ca": "https://ca.lab.internal/acme/acme/directory", "account": ""}
2025/10/10 11:15:04.374	INFO	http	waiting on internal rate limiter	{"identifiers": ["gogs.lab.internal"], "ca": "https://ca.lab.internal/acme/acme/directory", "account": ""}
2025/10/10 11:15:04.374	INFO	http	using ACME account	{"account_id": "https://ca.lab.internal/acme/acme/account/TVnzlEkW4G2KqG8nOSK0xThcFtIyqa1k", "account_contact": []}
2025/10/10 11:15:04.374	INFO	http	waiting on internal rate limiter	{"identifiers": ["transmission.lab.internal"], "ca": "https://ca.lab.internal/acme/acme/directory", "account": ""}
2025/10/10 11:15:04.375	INFO	http	done waiting on internal rate limiter	{"identifiers": ["cockpit.prometheus.lab.internal"], "ca": "https://ca.lab.internal/acme/acme/directory", "account": ""}
2025/10/10 11:15:04.375	INFO	http	done waiting on internal rate limiter	{"identifiers": ["gogs.lab.internal"], "ca": "https://ca.lab.internal/acme/acme/directory", "account": ""}
2025/10/10 11:15:04.375	INFO	http	done waiting on internal rate limiter	{"identifiers": ["transmission.lab.internal"], "ca": "https://ca.lab.internal/acme/acme/directory", "account": ""}
2025/10/10 11:15:04.378	INFO	http	using ACME account	{"account_id": "https://ca.lab.internal/acme/acme/account/MXsROLvWNIZhY8bTkhi3Tq6rFzWw5y9f", "account_contact": []}
2025/10/10 11:15:04.378	INFO	http	using ACME account	{"account_id": "https://ca.lab.internal/acme/acme/account/KedzfwCvoBRXLip9lb1Nx3Coey5Hfrz2", "account_contact": []}
2025/10/10 11:15:04.376	INFO	http	using ACME account	{"account_id": "https://ca.lab.internal/acme/acme/account/AnXUU8jDwJyN3bUnOI5cQFfje5MbvLvR", "account_contact": []}
2025/10/10 11:15:04.383	INFO	http	waiting on internal rate limiter	{"identifiers": ["plex.lab.internal"], "ca": "https://ca.lab.internal/acme/acme/directory", "account": ""}
2025/10/10 11:15:04.384	INFO	http	done waiting on internal rate limiter	{"identifiers": ["plex.lab.internal"], "ca": "https://ca.lab.internal/acme/acme/directory", "account": ""}
2025/10/10 11:15:04.384	INFO	http	using ACME account	{"account_id": "https://ca.lab.internal/acme/acme/account/RhkD7PAr5VwhY45sesydEeGlHV3vCZ0k", "account_contact": []}
2025/10/10 11:15:04.388	INFO	http	waiting on internal rate limiter	{"identifiers": ["ntfy.lab.internal"], "ca": "https://ca.lab.internal/acme/acme/directory", "account": ""}
2025/10/10 11:15:04.389	INFO	http	done waiting on internal rate limiter	{"identifiers": ["ntfy.lab.internal"], "ca": "https://ca.lab.internal/acme/acme/directory", "account": ""}
2025/10/10 11:15:04.389	INFO	http	using ACME account	{"account_id": "https://ca.lab.internal/acme/acme/account/GO74M5y5JJk4lMPbxTADnIxJk77cZmQp", "account_contact": []}
2025/10/10 11:15:04.389	INFO	http	waiting on internal rate limiter	{"identifiers": ["fritz.lab.internal"], "ca": "https://ca.lab.internal/acme/acme/directory", "account": ""}
2025/10/10 11:15:04.390	INFO	http	waiting on internal rate limiter	{"identifiers": ["iot.lab.internal"], "ca": "https://ca.lab.internal/acme/acme/directory", "account": ""}
2025/10/10 11:15:04.388	INFO	http	waiting on internal rate limiter	{"identifiers": ["cockpit.lab.internal"], "ca": "https://ca.lab.internal/acme/acme/directory", "account": ""}
2025/10/10 11:15:04.391	INFO	http	done waiting on internal rate limiter	{"identifiers": ["fritz.lab.internal"], "ca": "https://ca.lab.internal/acme/acme/directory", "account": ""}
2025/10/10 11:15:04.394	INFO	http	using ACME account	{"account_id": "https://ca.lab.internal/acme/acme/account/vL17teKbQFgzyjp0f8MPNznevFynF8Ez", "account_contact": []}
2025/10/10 11:15:04.392	INFO	http	done waiting on internal rate limiter	{"identifiers": ["iot.lab.internal"], "ca": "https://ca.lab.internal/acme/acme/directory", "account": ""}
2025/10/10 11:15:04.397	INFO	http	using ACME account	{"account_id": "https://ca.lab.internal/acme/acme/account/OK0GrLITerSfSoLb6hD0bZL1H4q2IMov", "account_contact": []}
2025/10/10 11:15:04.393	INFO	http	done waiting on internal rate limiter	{"identifiers": ["cockpit.lab.internal"], "ca": "https://ca.lab.internal/acme/acme/directory", "account": ""}
2025/10/10 11:15:04.398	INFO	http	using ACME account	{"account_id": "https://ca.lab.internal/acme/acme/account/K3gxk0l88ZiMefi2GoOQpvn1TCRGaZlq", "account_contact": []}
2025/10/10 11:15:04.433	INFO	http.acme_client	trying to solve challenge	{"identifier": "partdb.lab.internal", "challenge_type": "tls-alpn-01", "ca": "https://ca.lab.internal/acme/acme/directory"}
2025/10/10 11:15:04.438	INFO	http.acme_client	trying to solve challenge	{"identifier": "transmission.lab.internal", "challenge_type": "tls-alpn-01", "ca": "https://ca.lab.internal/acme/acme/directory"}
2025/10/10 11:15:04.444	INFO	http.acme_client	trying to solve challenge	{"identifier": "cockpit.prometheus.lab.internal", "challenge_type": "tls-alpn-01", "ca": "https://ca.lab.internal/acme/acme/directory"}
2025/10/10 11:15:04.449	INFO	http.acme_client	trying to solve challenge	{"identifier": "ntfy.lab.internal", "challenge_type": "tls-alpn-01", "ca": "https://ca.lab.internal/acme/acme/directory"}
2025/10/10 11:15:04.455	INFO	http.acme_client	trying to solve challenge	{"identifier": "gogs.lab.internal", "challenge_type": "tls-alpn-01", "ca": "https://ca.lab.internal/acme/acme/directory"}
2025/10/10 11:15:04.456	INFO	http.acme_client	trying to solve challenge	{"identifier": "plex.lab.internal", "challenge_type": "tls-alpn-01", "ca": "https://ca.lab.internal/acme/acme/directory"}
2025/10/10 11:15:04.459	INFO	http.acme_client	trying to solve challenge	{"identifier": "fritz.lab.internal", "challenge_type": "tls-alpn-01", "ca": "https://ca.lab.internal/acme/acme/directory"}
2025/10/10 11:15:04.465	INFO	http.acme_client	trying to solve challenge	{"identifier": "cockpit.lab.internal", "challenge_type": "tls-alpn-01", "ca": "https://ca.lab.internal/acme/acme/directory"}
2025/10/10 11:15:04.468	INFO	http.acme_client	trying to solve challenge	{"identifier": "iot.lab.internal", "challenge_type": "tls-alpn-01", "ca": "https://ca.lab.internal/acme/acme/directory"}
2025/10/10 11:15:04.469	INFO	tls	served key authentication certificate	{"server_name": "cockpit.prometheus.lab.internal", "challenge": "tls-alpn-01", "remote": "192.168.168.34:50240", "distributed": false}
2025/10/10 11:15:04.474	INFO	tls	served key authentication certificate	{"server_name": "partdb.lab.internal", "challenge": "tls-alpn-01", "remote": "192.168.168.34:50224", "distributed": false}
2025/10/10 11:15:04.478	INFO	tls	served key authentication certificate	{"server_name": "gogs.lab.internal", "challenge": "tls-alpn-01", "remote": "192.168.168.34:50248", "distributed": false}
2025/10/10 11:15:04.478	INFO	tls	served key authentication certificate	{"server_name": "transmission.lab.internal", "challenge": "tls-alpn-01", "remote": "192.168.168.34:50236", "distributed": false}
2025/10/10 11:15:04.485	INFO	tls	served key authentication certificate	{"server_name": "ntfy.lab.internal", "challenge": "tls-alpn-01", "remote": "192.168.168.34:50244", "distributed": false}
2025/10/10 11:15:04.491	INFO	tls	served key authentication certificate	{"server_name": "fritz.lab.internal", "challenge": "tls-alpn-01", "remote": "192.168.168.34:50262", "distributed": false}
2025/10/10 11:15:04.499	INFO	tls	served key authentication certificate	{"server_name": "cockpit.lab.internal", "challenge": "tls-alpn-01", "remote": "192.168.168.34:50276", "distributed": false}
2025/10/10 11:15:04.499	INFO	tls	served key authentication certificate	{"server_name": "plex.lab.internal", "challenge": "tls-alpn-01", "remote": "192.168.168.34:50288", "distributed": false}
2025/10/10 11:15:04.506	INFO	tls	served key authentication certificate	{"server_name": "iot.lab.internal", "challenge": "tls-alpn-01", "remote": "192.168.168.34:50302", "distributed": false}
2025/10/10 11:15:04.739	INFO	http.acme_client	authorization finalized	{"identifier": "cockpit.prometheus.lab.internal", "authz_status": "valid"}
2025/10/10 11:15:04.739	INFO	http.acme_client	validations succeeded; finalizing order	{"order": "https://ca.lab.internal/acme/acme/order/30vjjbEVzq0S79Rnzuwk4oexYuiKFkiM"}
2025/10/10 11:15:04.751	INFO	http.acme_client	authorization finalized	{"identifier": "partdb.lab.internal", "authz_status": "valid"}
2025/10/10 11:15:04.751	INFO	http.acme_client	validations succeeded; finalizing order	{"order": "https://ca.lab.internal/acme/acme/order/G5bv9gVKMXCbXITikhtPU4s0uEhlmh9V"}
2025/10/10 11:15:04.756	INFO	http.acme_client	authorization finalized	{"identifier": "gogs.lab.internal", "authz_status": "valid"}
2025/10/10 11:15:04.756	INFO	http.acme_client	validations succeeded; finalizing order	{"order": "https://ca.lab.internal/acme/acme/order/Vkdp7NoZU6LjKjZl6NKFQ8bEamOPsNvk"}
2025/10/10 11:15:04.771	INFO	http.acme_client	authorization finalized	{"identifier": "ntfy.lab.internal", "authz_status": "valid"}
2025/10/10 11:15:04.771	INFO	http.acme_client	validations succeeded; finalizing order	{"order": "https://ca.lab.internal/acme/acme/order/T4SvTdYVdq03PiSHVrOVtO1bdLX9SG7S"}
2025/10/10 11:15:04.774	INFO	http.acme_client	authorization finalized	{"identifier": "cockpit.lab.internal", "authz_status": "valid"}
2025/10/10 11:15:04.774	INFO	http.acme_client	validations succeeded; finalizing order	{"order": "https://ca.lab.internal/acme/acme/order/TTLtyl4JkBhvAXc9GKiBrEkkQoShwqgC"}
2025/10/10 11:15:04.777	INFO	http.acme_client	authorization finalized	{"identifier": "transmission.lab.internal", "authz_status": "valid"}
2025/10/10 11:15:04.777	INFO	http.acme_client	validations succeeded; finalizing order	{"order": "https://ca.lab.internal/acme/acme/order/AxNu0yImeI3BbL52mJFDR8SWOjXjJ8uj"}
2025/10/10 11:15:04.778	INFO	http.acme_client	authorization finalized	{"identifier": "plex.lab.internal", "authz_status": "valid"}
2025/10/10 11:15:04.778	INFO	http.acme_client	validations succeeded; finalizing order	{"order": "https://ca.lab.internal/acme/acme/order/ckyAPGT4ycguRZOHOWzB8googQ6dBy0W"}
2025/10/10 11:15:04.779	INFO	http.acme_client	authorization finalized	{"identifier": "fritz.lab.internal", "authz_status": "valid"}
2025/10/10 11:15:04.779	INFO	http.acme_client	validations succeeded; finalizing order	{"order": "https://ca.lab.internal/acme/acme/order/97Tro67DU64BmXbFO51nFRNID9AMqTIp"}
2025/10/10 11:15:04.786	INFO	http.acme_client	authorization finalized	{"identifier": "iot.lab.internal", "authz_status": "valid"}
2025/10/10 11:15:04.786	INFO	http.acme_client	validations succeeded; finalizing order	{"order": "https://ca.lab.internal/acme/acme/order/OIWycJZGvoXtV0mmo7NExIgSJwVfQK6F"}
2025/10/10 11:15:04.930	INFO	http.acme_client	successfully downloaded available certificate chains	{"count": 1, "first_url": "https://ca.lab.internal/acme/acme/certificate/IJGFVwYlksxQ6kjs6GouSX5QCT5zZelf"}
2025/10/10 11:15:04.934	INFO	tls.obtain	certificate obtained successfully	{"identifier": "cockpit.prometheus.lab.internal", "issuer": "ca.lab.internal-acme-acme-directory"}
2025/10/10 11:15:04.935	INFO	tls.obtain	releasing lock	{"identifier": "cockpit.prometheus.lab.internal"}
2025/10/10 11:15:05.076	INFO	http.acme_client	successfully downloaded available certificate chains	{"count": 1, "first_url": "https://ca.lab.internal/acme/acme/certificate/Fel637XrCUC71lBgCmgpXukjt1Fx3SDG"}
2025/10/10 11:15:05.078	INFO	tls.obtain	certificate obtained successfully	{"identifier": "partdb.lab.internal", "issuer": "ca.lab.internal-acme-acme-directory"}
2025/10/10 11:15:05.079	INFO	tls.obtain	releasing lock	{"identifier": "partdb.lab.internal"}
2025/10/10 11:15:05.220	INFO	http.acme_client	successfully downloaded available certificate chains	{"count": 1, "first_url": "https://ca.lab.internal/acme/acme/certificate/uHkH2RLDFeFVOAhPy8p8Mj8KbfFtRxn7"}
2025/10/10 11:15:05.222	INFO	tls.obtain	certificate obtained successfully	{"identifier": "gogs.lab.internal", "issuer": "ca.lab.internal-acme-acme-directory"}
2025/10/10 11:15:05.223	INFO	tls.obtain	releasing lock	{"identifier": "gogs.lab.internal"}
2025/10/10 11:15:05.365	INFO	http.acme_client	successfully downloaded available certificate chains	{"count": 1, "first_url": "https://ca.lab.internal/acme/acme/certificate/TqwkRPMJG4A5tbNHZppIu936gvyQXNWR"}
2025/10/10 11:15:05.368	INFO	tls.obtain	certificate obtained successfully	{"identifier": "cockpit.lab.internal", "issuer": "ca.lab.internal-acme-acme-directory"}
2025/10/10 11:15:05.369	INFO	tls.obtain	releasing lock	{"identifier": "cockpit.lab.internal"}
2025/10/10 11:15:05.534	INFO	http.acme_client	successfully downloaded available certificate chains	{"count": 1, "first_url": "https://ca.lab.internal/acme/acme/certificate/NNn5X29sPHJ7vjtHHQ5kjf3dJNpqRhJP"}
2025/10/10 11:15:05.536	INFO	tls.obtain	certificate obtained successfully	{"identifier": "ntfy.lab.internal", "issuer": "ca.lab.internal-acme-acme-directory"}
2025/10/10 11:15:05.537	INFO	tls.obtain	releasing lock	{"identifier": "ntfy.lab.internal"}
2025/10/10 11:15:05.677	INFO	http.acme_client	successfully downloaded available certificate chains	{"count": 1, "first_url": "https://ca.lab.internal/acme/acme/certificate/5HUmDKSbNBoQWEn8oGPuFXnVMHyKIPpu"}
2025/10/10 11:15:05.681	INFO	tls.obtain	certificate obtained successfully	{"identifier": "plex.lab.internal", "issuer": "ca.lab.internal-acme-acme-directory"}
2025/10/10 11:15:05.682	INFO	tls.obtain	releasing lock	{"identifier": "plex.lab.internal"}
2025/10/10 11:15:05.820	INFO	http.acme_client	successfully downloaded available certificate chains	{"count": 1, "first_url": "https://ca.lab.internal/acme/acme/certificate/jzSxIVUocWREfr3GNAsVV7LNc0gCdOQA"}
2025/10/10 11:15:05.823	INFO	tls.obtain	certificate obtained successfully	{"identifier": "transmission.lab.internal", "issuer": "ca.lab.internal-acme-acme-directory"}
2025/10/10 11:15:05.824	INFO	tls.obtain	releasing lock	{"identifier": "transmission.lab.internal"}
2025/10/10 11:15:05.964	INFO	http.acme_client	successfully downloaded available certificate chains	{"count": 1, "first_url": "https://ca.lab.internal/acme/acme/certificate/5Ffe4Wb9YgkJqkB8luF3mRTexaF3oMc0"}
2025/10/10 11:15:05.967	INFO	tls.obtain	certificate obtained successfully	{"identifier": "fritz.lab.internal", "issuer": "ca.lab.internal-acme-acme-directory"}
2025/10/10 11:15:05.968	INFO	tls.obtain	releasing lock	{"identifier": "fritz.lab.internal"}
2025/10/10 11:15:06.107	INFO	http.acme_client	successfully downloaded available certificate chains	{"count": 1, "first_url": "https://ca.lab.internal/acme/acme/certificate/OLlieHgVTzfh7H0wlVcLSMJRSfQZZa35"}
2025/10/10 11:15:06.110	INFO	tls.obtain	certificate obtained successfully	{"identifier": "iot.lab.internal", "issuer": "ca.lab.internal-acme-acme-directory"}
2025/10/10 11:15:06.111	INFO	tls.obtain	releasing lock	{"identifier": "iot.lab.internal"}

Sani7 avatar Oct 10 '25 11:10 Sani7

Thank you for the update -- can you please provide unredacted configs? (except credentials)

It matters for troubleshooting this. Redactions almost always interfere with troubleshooting efforts and waste everyone's time.

mholt avatar Oct 14 '25 16:10 mholt

This is my config: (cat /etc/caddy/Caddyfile /etc/caddy/Caddyfile.d/*)

# The Caddyfile is an easy way to configure your Caddy web server.
#
# https://caddyserver.com/docs/caddyfile

#Global options
{
	# Our local ACME server
	acme_ca https://ca.lab.internal/acme/acme/directory
	# step-ca doens't support ocsp yet
	ocsp_stapling off
}

# As an alternative to editing the above site block, you can add your own site
# block files in the Caddyfile.d directory, and they will be included as long
# as they use the .caddyfile extension.
import Caddyfile.d/*.caddyfile
cockpit.lab.internal {
        reverse_proxy https://127.0.0.1:9090 {
	        transport http {
        	    tls_insecure_skip_verify
        }
    }
}
cockpit.prometheus.lab.internal {
        reverse_proxy https://192.168.168.34:9090 {
        transport http {
            tls_insecure_skip_verify
        }
    }
}
fritz.lab.internal {
        reverse_proxy https://192.168.168.1 {
	        transport http {
        	    tls_insecure_skip_verify
	        }
		header_up Host {upstream_hostport}
    }
}
gogs.lab.internal {
        reverse_proxy http://192.168.168.34:10880
}
iot.lab.internal {
        reverse_proxy http://192.168.168.34:3000
	handle_path  /iot_api/* {
		reverse_proxy http://192.168.168.34:8080
	}
}
ntfy.lab.internal {
   reverse_proxy 192.168.168.34:8081

    # Redirect HTTP to HTTPS, but only for GET topic addresses, since we want
    # it to work with curl without the annoying https:// prefix
    @httpget {
        protocol http
        method GET
        path_regexp ^/([-_a-z0-9]{0,64}$|docs/|static/)
    }
    redir @httpget https://{host}{uri}
}

partdb.lab.internal {
   reverse_proxy 192.168.168.34:8082
}
plex.lab.internal {
        reverse_proxy http://ds.lab.internal:32400 
}

transmission.lab.internal {
   reverse_proxy 192.168.168.34:9091
}

Sani7 avatar Oct 14 '25 16:10 Sani7

Thank you! Currently investigating.

mholt avatar Oct 14 '25 21:10 mholt

Seems that auto HTTPS is removing the explicitly configured issuer. Still investigating.

mholt avatar Oct 17 '25 05:10 mholt

@Sani7 @ChrisChoke Would you be able to try the patch in https://github.com/caddyserver/caddy/pull/7324 ? CI should have build artifacts momentarily, or you can build from that branch using xcaddy or manually with git or whatever.

mholt avatar Oct 29 '25 21:10 mholt

Can test at friday at earliest. I am on a short trip. But I will do this on Friday.

ChrisChoke avatar Oct 30 '25 06:10 ChrisChoke

Works for me using version v2.10.3-0.20251029221500-d0a7295066f5 h1:Uq1yJyINBeZ7YZ3VUs2fp7GUzIonsh5XbR2G+Qr78UU=

Sani7 avatar Oct 30 '25 18:10 Sani7

Great to hear! Thank you @Sani7 . Will await @ChrisChoke 's report soon and then merge :)

mholt avatar Oct 30 '25 19:10 mholt

caddy infos:

compiled via xcaddy build internal-tls --with github.com/caddy-dns/rfc2136

Okt 31 12:19:27 caddy caddy[27359]: caddy.HomeDir=/var/lib/caddy
Okt 31 12:19:27 caddy caddy[27359]: caddy.AppDataDir=/var/lib/caddy/.local/share/caddy
Okt 31 12:19:27 caddy caddy[27359]: caddy.AppConfigDir=/var/lib/caddy/.config/caddy
Okt 31 12:19:27 caddy caddy[27359]: caddy.ConfigAutosavePath=/var/lib/caddy/.config/caddy/autosave.json
Okt 31 12:19:27 caddy caddy[27359]: caddy.Version=v2.10.3-0.20251029221500-d0a7295066f5 h1:Uq1yJyINBeZ7YZ3VUs2fp7GUzIonsh5XbR2G+Qr78UU=
Okt 31 12:19:27 caddy caddy[27359]: runtime.GOOS=linux
Okt 31 12:19:27 caddy caddy[27359]: runtime.GOARCH=amd64
Okt 31 12:19:27 caddy caddy[27359]: runtime.Compiler=gc
Okt 31 12:19:27 caddy caddy[27359]: runtime.NumCPU=2
Okt 31 12:19:27 caddy caddy[27359]: runtime.GOMAXPROCS=2
Okt 31 12:19:27 caddy caddy[27359]: runtime.Version=go1.25.3
Okt 31 12:19:27 caddy caddy[27359]: os.Getwd=/
Okt 31 12:19:27 caddy caddy[27359]: LANG=de_DE.UTF-8
Okt 31 12:19:27 caddy caddy[27359]: PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
Okt 31 12:19:27 caddy caddy[27359]: NOTIFY_SOCKET=/run/systemd/notify
Okt 31 12:19:27 caddy caddy[27359]: HOME=/var/lib/caddy
Okt 31 12:19:27 caddy caddy[27359]: LOGNAME=caddy
Okt 31 12:19:27 caddy caddy[27359]: USER=caddy
Okt 31 12:19:27 caddy caddy[27359]: INVOCATION_ID=12461ba8422a4a9197892972476ad5d0
Okt 31 12:19:27 caddy caddy[27359]: JOURNAL_STREAM=8:467238
Okt 31 12:19:27 caddy caddy[27359]: SYSTEMD_EXEC_PID=27359
logger":"tls.obtain","msg":"certificate obtained successfully","identifier":"test.example.internal","issuer":"local"}
logger":"tls.obtain","msg":"releasing lock","identifier":"test.example.internal"}
logger":"tls","msg":"stapling OCSP","identifiers":["test.example.internal"]}

For me it does not work at the moment. Sorry. I tested a binary of caddy without dns plugin as well. but it doesnt change anything. It picks the local issuer instead acme_ca for http-01 challenge. Maybe i make something wrong? When it works for @Sani7 maybe i am the fault at the moment 😆

Chris

ChrisChoke avatar Oct 31 '25 11:10 ChrisChoke

@ChrisChoke It looks like it does work:

"identifier":"test.example.internal","issuer":"local"

mholt avatar Oct 31 '25 14:10 mholt

But this certificate is located at /var/lib/caddy/.local/share/caddy/certificates/local instead under the stepca.example.internal directory. And the certificate was not issued by step ca. It is issued by caddys internal pki.

i expect that it should looks like this.

"identifier":"test.example.internal","issuer":"ca.example.internal:8443-acme-acme-directory"

than the certificate will issued by my internal step-ca.

ChrisChoke avatar Oct 31 '25 14:10 ChrisChoke

Okay... weird. So what is the output of caddy adapt for your config now?

(Please double- and triple-check that you're running the correct binary not only when getting the environment, but also when actually running the adapt command or the server).

mholt avatar Oct 31 '25 15:10 mholt

i will check that at monday. i will share the information early the morning. Sorry, but this weekend i cant get all the information.

@Sani7 can you share your log? which binary you used? from CI artifacts or self built?

ChrisChoke avatar Oct 31 '25 16:10 ChrisChoke

I used xcaddy as well using this command: xcaddy build d0a7295066f58aaf884b1de70d6481fd2bf483ce --output caddy

Sani7 avatar Oct 31 '25 16:10 Sani7

This is my log output, same issue here. Did not triple check

caddy.HomeDir=/home/sanderspeetjens
caddy.AppDataDir=/home/sanderspeetjens/.local/share/caddy
caddy.AppConfigDir=/home/sanderspeetjens/.config/caddy
caddy.ConfigAutosavePath=/home/sanderspeetjens/.config/caddy/autosave.json
caddy.Version=v2.10.3-0.20251029221500-d0a7295066f5 h1:Uq1yJyINBeZ7YZ3VUs2fp7GUzIonsh5XbR2G+Qr78UU=
runtime.GOOS=linux
runtime.GOARCH=arm64
runtime.Compiler=gc
runtime.NumCPU=4
runtime.GOMAXPROCS=4
runtime.Version=go1.25.3
os.Getwd=/home/sanderspeetjens

SHELL=/bin/bash
HISTCONTROL=ignoredups
HISTSIZE=1000
HOSTNAME=lab.internal
GPG_TTY=/dev/pts/0
EDITOR=/usr/bin/vim
PWD=/home/sanderspeetjens
LOGNAME=sanderspeetjens
XDG_SESSION_TYPE=tty
MOTD_SHOWN=pam
HOME=/home/sanderspeetjens
LANG=en_US.UTF-8
LS_COLORS=rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:mi=01;37;41:su=37;41:sg=30;43:ca=00:tw=30;42:ow=34;42:st=37;44:ex=01;32:*.7z=01;31:*.ace=01;31:*.alz=01;31:*.apk=01;31:*.arc=01;31:*.arj=01;31:*.bz=01;31:*.bz2=01;31:*.cab=01;31:*.cpio=01;31:*.crate=01;31:*.deb=01;31:*.drpm=01;31:*.dwm=01;31:*.dz=01;31:*.ear=01;31:*.egg=01;31:*.esd=01;31:*.gz=01;31:*.jar=01;31:*.lha=01;31:*.lrz=01;31:*.lz=01;31:*.lz4=01;31:*.lzh=01;31:*.lzma=01;31:*.lzo=01;31:*.pyz=01;31:*.rar=01;31:*.rpm=01;31:*.rz=01;31:*.sar=01;31:*.swm=01;31:*.t7z=01;31:*.tar=01;31:*.taz=01;31:*.tbz=01;31:*.tbz2=01;31:*.tgz=01;31:*.tlz=01;31:*.txz=01;31:*.tz=01;31:*.tzo=01;31:*.tzst=01;31:*.udeb=01;31:*.war=01;31:*.whl=01;31:*.wim=01;31:*.xz=01;31:*.z=01;31:*.zip=01;31:*.zoo=01;31:*.zst=01;31:*.avif=01;35:*.jpg=01;35:*.jpeg=01;35:*.mjpg=01;35:*.mjpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.svg=01;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01;35:*.m2v=01;35:*.mkv=01;35:*.webm=01;35:*.webp=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35:*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;35:*.rmvb=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.flv=01;35:*.gl=01;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.cgm=01;35:*.emf=01;35:*.ogv=01;35:*.ogx=01;35:*.aac=01;36:*.au=01;36:*.flac=01;36:*.m4a=01;36:*.mid=01;36:*.midi=01;36:*.mka=01;36:*.mp3=01;36:*.mpc=01;36:*.ogg=01;36:*.ra=01;36:*.wav=01;36:*.oga=01;36:*.opus=01;36:*.spx=01;36:*.xspf=01;36:*~=00;90:*#=00;90:*.bak=00;90:*.crdownload=00;90:*.dpkg-dist=00;90:*.dpkg-new=00;90:*.dpkg-old=00;90:*.dpkg-tmp=00;90:*.old=00;90:*.orig=00;90:*.part=00;90:*.rej=00;90:*.rpmnew=00;90:*.rpmorig=00;90:*.rpmsave=00;90:*.swp=00;90:*.tmp=00;90:*.ucf-dist=00;90:*.ucf-new=00;90:*.ucf-old=00;90:
SSH_CONNECTION=192.168.168.29 43672 192.168.168.35 22
XDG_SESSION_CLASS=user
SELINUX_ROLE_REQUESTED=
TERM=xterm-256color
LESSOPEN=||/usr/bin/lesspipe.sh %s
USER=sanderspeetjens
SELINUX_USE_CURRENT_RANGE=
SHLVL=1
XDG_SESSION_ID=43
XDG_RUNTIME_DIR=/run/user/1000
SSH_CLIENT=192.168.168.29 43672 22
DEBUGINFOD_URLS=https://debuginfod.fedoraproject.org/ 
DEBUGINFOD_IMA_CERT_PATH=/etc/keys/ima:
PATH=/home/sanderspeetjens/.local/bin:/home/sanderspeetjens/bin:/usr/local/bin:/usr/bin
SELINUX_LEVEL_REQUESTED=
DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/1000/bus
MAIL=/var/spool/mail/sanderspeetjens
SSH_TTY=/dev/pts/0
_=/usr/bin/caddy
GOTRACEBACK=none
2025/10/31 16:55:27.928	INFO	maxprocs: Leaving GOMAXPROCS=4: CPU quota undefined
2025/10/31 16:55:27.928	INFO	GOMEMLIMIT is updated	{"GOMEMLIMIT": 819250790, "previous": 9223372036854775807}
2025/10/31 16:55:27.928	INFO	using config from file	{"file": "/etc/caddy/Caddyfile"}
2025/10/31 16:55:27.928	INFO	adapted config to JSON	{"adapter": "caddyfile"}
2025/10/31 16:55:27.943	INFO	admin	admin endpoint started	{"address": "localhost:2019", "enforce_origin": false, "origins": ["//localhost:2019", "//[::1]:2019", "//127.0.0.1:2019"]}
2025/10/31 16:55:27.944	INFO	tls.cache.maintenance	started background certificate maintenance	{"cache": "0x4000132e80"}
2025/10/31 16:55:27.945	INFO	http.auto_https	server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS	{"server_name": "srv0", "https_port": 443}
2025/10/31 16:55:27.945	INFO	http.auto_https	enabling automatic HTTP->HTTPS redirects	{"server_name": "srv0"}
2025/10/31 16:55:27.945	WARN	http.auto_https	server is listening only on the HTTP port, so no automatic HTTPS will be applied to this server	{"server_name": "srv1", "http_port": 80}
2025/10/31 16:55:28.054	INFO	http	enabling HTTP/3 listener	{"addr": ":443"}
2025/10/31 16:55:28.055	INFO	failed to sufficiently increase receive buffer size (was: 208 kiB, wanted: 7168 kiB, got: 416 kiB). See https://github.com/quic-go/quic-go/wiki/UDP-Buffer-Sizes for details.
2025/10/31 16:55:28.056	INFO	http.log	server running	{"name": "srv0", "protocols": ["h1", "h2", "h3"]}
2025/10/31 16:55:28.057	WARN	http	HTTP/2 skipped because it requires TLS	{"network": "tcp", "addr": ":80"}
2025/10/31 16:55:28.057	WARN	http	HTTP/3 skipped because it requires TLS	{"network": "tcp", "addr": ":80"}
2025/10/31 16:55:28.057	INFO	http.log	server running	{"name": "srv1", "protocols": ["h1", "h2", "h3"]}
2025/10/31 16:55:28.057	INFO	http	enabling automatic TLS certificate management	{"domains": ["plex.lab.internal", "iot.lab.internal", "transmission.lab.internal", "fritz.lab.internal", "gogs.lab.internal", "partdb.lab.internal", "cockpit.prometheus.lab.internal", "cockpit.lab.internal", "ntfy.lab.internal"]}
2025/10/31 16:55:28.061	WARN	pki.ca.local	installing root certificate (you might be prompted for password)	{"path": "storage:pki/authorities/local/root.crt"}
2025/10/31 16:55:28.063	INFO	not NSS security databases found
2025/10/31 16:55:28.063	INFO	define JAVA_HOME environment variable to use the Java trust
2025/10/31 16:55:28.071	INFO	tls.obtain	acquiring lock	{"identifier": "iot.lab.internal"}
2025/10/31 16:55:28.084	INFO	tls.obtain	acquiring lock	{"identifier": "gogs.lab.internal"}
2025/10/31 16:55:28.090	INFO	tls.obtain	acquiring lock	{"identifier": "transmission.lab.internal"}
2025/10/31 16:55:28.205	INFO	tls.obtain	acquiring lock	{"identifier": "plex.lab.internal"}
2025/10/31 16:55:28.209	INFO	tls.obtain	lock acquired	{"identifier": "iot.lab.internal"}
2025/10/31 16:55:28.210	INFO	tls.obtain	acquiring lock	{"identifier": "cockpit.lab.internal"}
2025/10/31 16:55:28.210	INFO	tls.obtain	obtaining certificate	{"identifier": "iot.lab.internal"}
2025/10/31 16:55:28.209	INFO	tls.obtain	lock acquired	{"identifier": "transmission.lab.internal"}
2025/10/31 16:55:28.213	INFO	tls.obtain	obtaining certificate	{"identifier": "transmission.lab.internal"}
2025/10/31 16:55:28.209	INFO	tls.obtain	lock acquired	{"identifier": "gogs.lab.internal"}
2025/10/31 16:55:28.215	INFO	tls.obtain	acquiring lock	{"identifier": "cockpit.prometheus.lab.internal"}
2025/10/31 16:55:28.215	INFO	tls.obtain	acquiring lock	{"identifier": "partdb.lab.internal"}
2025/10/31 16:55:28.219	INFO	tls.obtain	obtaining certificate	{"identifier": "gogs.lab.internal"}
2025/10/31 16:55:28.213	INFO	tls.obtain	acquiring lock	{"identifier": "fritz.lab.internal"}
2025/10/31 16:55:28.250	INFO	tls.obtain	lock acquired	{"identifier": "plex.lab.internal"}
2025/10/31 16:55:28.250	INFO	tls.obtain	lock acquired	{"identifier": "fritz.lab.internal"}
2025/10/31 16:55:28.250	INFO	tls.obtain	lock acquired	{"identifier": "partdb.lab.internal"}
2025/10/31 16:55:28.251	INFO	tls	cleaning storage unit	{"storage": "FileStorage:/home/sanderspeetjens/.local/share/caddy"}
2025/10/31 16:55:28.252	INFO	tls.obtain	acquiring lock	{"identifier": "ntfy.lab.internal"}
2025/10/31 16:55:28.252	INFO	tls.obtain	obtaining certificate	{"identifier": "fritz.lab.internal"}
2025/10/31 16:55:28.252	INFO	tls.obtain	obtaining certificate	{"identifier": "partdb.lab.internal"}
2025/10/31 16:55:28.251	INFO	tls.obtain	obtaining certificate	{"identifier": "plex.lab.internal"}
2025/10/31 16:55:28.251	INFO	tls.obtain	lock acquired	{"identifier": "cockpit.prometheus.lab.internal"}
2025/10/31 16:55:28.257	INFO	tls.obtain	obtaining certificate	{"identifier": "cockpit.prometheus.lab.internal"}
2025/10/31 16:55:28.287	INFO	tls.obtain	lock acquired	{"identifier": "cockpit.lab.internal"}
2025/10/31 16:55:28.329	INFO	tls.obtain	obtaining certificate	{"identifier": "cockpit.lab.internal"}
2025/10/31 16:55:28.370	INFO	tls.obtain	certificate obtained successfully	{"identifier": "transmission.lab.internal", "issuer": "local"}
2025/10/31 16:55:28.371	INFO	tls.obtain	releasing lock	{"identifier": "transmission.lab.internal"}
2025/10/31 16:55:28.378	WARN	tls	stapling OCSP	{"identifiers": ["transmission.lab.internal"]}
2025/10/31 16:55:28.384	INFO	tls.obtain	lock acquired	{"identifier": "ntfy.lab.internal"}
2025/10/31 16:55:28.386	INFO	tls.obtain	obtaining certificate	{"identifier": "ntfy.lab.internal"}
2025/10/31 16:55:28.394	INFO	tls.obtain	certificate obtained successfully	{"identifier": "cockpit.prometheus.lab.internal", "issuer": "local"}
2025/10/31 16:55:28.398	INFO	tls.obtain	releasing lock	{"identifier": "cockpit.prometheus.lab.internal"}
2025/10/31 16:55:28.400	INFO	tls.obtain	certificate obtained successfully	{"identifier": "fritz.lab.internal", "issuer": "local"}
2025/10/31 16:55:28.401	INFO	tls.obtain	releasing lock	{"identifier": "fritz.lab.internal"}
2025/10/31 16:55:28.406	WARN	tls	stapling OCSP	{"identifiers": ["cockpit.prometheus.lab.internal"]}
2025/10/31 16:55:28.407	WARN	tls	stapling OCSP	{"identifiers": ["fritz.lab.internal"]}
2025/10/31 16:55:28.415	INFO	tls.obtain	certificate obtained successfully	{"identifier": "cockpit.lab.internal", "issuer": "local"}
2025/10/31 16:55:28.416	INFO	tls.obtain	releasing lock	{"identifier": "cockpit.lab.internal"}
2025/10/31 16:55:28.420	WARN	tls	stapling OCSP	{"identifiers": ["cockpit.lab.internal"]}
2025/10/31 16:55:28.428	INFO	tls.obtain	certificate obtained successfully	{"identifier": "gogs.lab.internal", "issuer": "local"}
2025/10/31 16:55:28.430	INFO	tls.obtain	releasing lock	{"identifier": "gogs.lab.internal"}
2025/10/31 16:55:28.434	WARN	tls	stapling OCSP	{"identifiers": ["gogs.lab.internal"]}
2025/10/31 16:55:28.471	INFO	tls.obtain	certificate obtained successfully	{"identifier": "ntfy.lab.internal", "issuer": "local"}
2025/10/31 16:55:28.472	INFO	tls	finished cleaning storage units
2025/10/31 16:55:28.472	INFO	tls.obtain	releasing lock	{"identifier": "ntfy.lab.internal"}
2025/10/31 16:55:28.476	WARN	tls	stapling OCSP	{"identifiers": ["ntfy.lab.internal"]}
2025/10/31 16:55:28.479	INFO	tls.obtain	certificate obtained successfully	{"identifier": "partdb.lab.internal", "issuer": "local"}
2025/10/31 16:55:28.480	INFO	tls.obtain	releasing lock	{"identifier": "partdb.lab.internal"}
2025/10/31 16:55:28.484	WARN	tls	stapling OCSP	{"identifiers": ["partdb.lab.internal"]}
2025/10/31 16:55:28.511	INFO	tls.obtain	certificate obtained successfully	{"identifier": "iot.lab.internal", "issuer": "local"}
2025/10/31 16:55:28.511	INFO	tls.obtain	releasing lock	{"identifier": "iot.lab.internal"}
2025/10/31 16:55:28.516	WARN	tls	stapling OCSP	{"identifiers": ["iot.lab.internal"]}
2025/10/31 16:55:28.523	INFO	tls.obtain	certificate obtained successfully	{"identifier": "plex.lab.internal", "issuer": "local"}
2025/10/31 16:55:28.523	INFO	tls.obtain	releasing lock	{"identifier": "plex.lab.internal"}
2025/10/31 16:55:28.527	WARN	tls	stapling OCSP	{"identifiers": ["plex.lab.internal"]}

Sani7 avatar Oct 31 '25 16:10 Sani7

are you sure that youre certificates are issued from your internal step-ca server? @Sani7

2025/10/31 16:55:28.523 INFO tls.obtain certificate obtained successfully {"identifier": "plex.lab.internal", "issuer": "local"} 2025/10/31 16:55:28.523 INFO tls.obtain releasing lock {"identifier": "plex.lab.internal"} 2025/10/31 16:55:28.527 WARN tls stapling OCSP {"identifiers": ["plex.lab.internal"]}

that looks similar to mine. but when i analyse the certificate, its issued by caddys internal pki instead from my step-ca pki from acme_ca directive.

your caddy 2.8.4 logs have your step-ca as issuer

2025/10/10 11:15:05.681 INFO tls.obtain certificate obtained successfully {"identifier": "plex.lab.internal", "issuer": "ca.lab.internal-acme-acme-directory"} 2025/10/10 11:15:05.682 INFO tls.obtain releasing lock {"identifier": "plex.lab.internal"}

ChrisChoke avatar Oct 31 '25 17:10 ChrisChoke

Maybe the question is, does acme_ca apply to internal-only names?

For people with mixed public and private domains in their config, they might expect acme_ca to only apply to their public domains, since they want their private ones to be issued with Caddy's CA. At least, I think that's typically what I see.

Maybe a separate directive is needed, like internal_acme_ca or something.

mholt avatar Oct 31 '25 17:10 mholt

@ChrisChoke It first loaded certificates that where expired from step-ca and now it is generating local certs

Sani7 avatar Oct 31 '25 18:10 Sani7

Maybe the question is, does acme_ca apply to internal-only names?

For people with mixed public and private domains in their config, they might expect acme_ca to only apply to their public domains, since they want their private ones to be issued with Caddy's CA. At least, I think that's typically what I see.

Maybe a separate directive is needed, like internal_acme_ca or something.

yeah that i was not thinking about. but the way how its currently work is something what i not understand. The domains which i issued with dns challenge respect the acme_ca directive and the one without the dns challenge snippet does not. they are issued by local pki. its a bit inconsitent, or? i thought the global acme_ca directive count overall for all configured domains how long i configure in the domain context an other acme server. regardless of wether for internal or public websites. the logs on 2.8.4 show me that it was different than now how it works.

@Sani7 is that how it works after the patch something what you expected? i think the behavior on 2.8.4 and now are different, or not?

ChrisChoke avatar Oct 31 '25 20:10 ChrisChoke