Caddy does not trust its own Local CA - x509

[tctlrd]

Issue Details

Running the latest caddy in a container. Caddy does not trust its own local CA. Even though it is successfully installing the local CA's root into the system trust store.

I am pointing acme_ca at a local openbao acme server which is serving with caddy's local CA signed cert (acme_ca will not accept a non-tls hostname endpoint: #1592)

{"level":"info","ts":1763811706.332481,"msg":"certificate installed properly in linux trusts"}
...
ERROR tls.obtain could not get certificate from issuer ...with server: provisioning client: performing request: Get \"...pki_int/acme/directory\": tls: failed to verify certificate: x509: certificate signed by unknown authority"}

This acme_ca_root config works after a caddy restart (it does not yet exist at first launch): acme_ca_root /data/caddy/pki/authorities/local/root.crt

Thank you for your attention to this issue!

Assistance Disclosure

AI not used

If AI was used, describe the extent to which it was used.

No response

[mohammed90]

Ideally, we need to be able to reproduce the bug in the most minimal way possible. This allows us to write regression tests to verify the fix is working. If we can't reproduce it, then you'll have to test our changes for us until it's fixed -- and then we can't add test cases, either.

I've attached a template below that will help make this easier and faster! This will require some effort on your part -- please understand that we will be dedicating time to fix the bug you are reporting if you can just help us understand it and reproduce it easily.

This template will ask for some information you've already provided; that's OK, just fill it out the best you can. :+1: I've also included some helpful tips below the template. Feel free to let me know if you have any questions!

Thank you again for your report, we look forward to resolving it!

Template

## 1. Environment

### 1a. Operating system and version

```
paste here
```


### 1b. Caddy version (run `caddy version` or paste commit SHA)

```
paste here
```


### 1c. Go version (if building Caddy from source; run `go version`)

```
paste here
```


## 2. Description

### 2a. What happens (briefly explain what is wrong)




### 2b. Why it's a bug (if it's not obvious)




### 2c. Log output

```
paste terminal output or logs here
```



### 2d. Workaround(s)




### 2e. Relevant links




## 3. Tutorial (minimal steps to reproduce the bug)

Helpful tips

1. Environment: Please fill out your OS and Caddy versions, even if you don't think they are relevant. (They are always relevant.) If you built Caddy from source, provide the commit SHA and specify your exact Go version.

2. Description: Describe at a high level what the bug is. What happens? Why is it a bug? Not all bugs are obvious, so convince readers that it's actually a bug.

   • 2c) Log output: Paste terminal output and/or complete logs in a code block. DO NOT REDACT INFORMATION except for credentials.
   • 2d) Workaround: What are you doing to work around the problem in the meantime? This can help others who encounter the same problem, until we implement a fix.
   • 2e) Relevant links: Please link to any related issues, pull requests, docs, and/or discussion. This can add crucial context to your report.

3. Tutorial: What are the minimum required specific steps someone needs to take in order to experience the same bug? Your goal here is to make sure that anyone else can have the same experience with the bug as you do. You are writing a tutorial, so make sure to carry it out yourself before posting it. Please:

   • Start with an empty config. Add only the lines/parameters that are absolutely required to reproduce the bug.
   • Do not run Caddy inside containers.
   • Run Caddy manually in your terminal; do not use systemd or other init systems.
   • If making HTTP requests, avoid web browsers. Use a simpler HTTP client instead, like curl.
   • Do not redact any information from your config (except credentials). Domain names are public knowledge and often necessary for quick resolution of an issue!
   • Note that ignoring this advice may result in delays, or even in your issue being closed. 😞 Only actionable issues are kept open, and if there is not enough information or clarity to reproduce the bug, then the report is not actionable.

Example of a tutorial:

Create a config file:

{ ... }

Open terminal and run Caddy:

$ caddy ...

Make an HTTP request:

$ curl ...

Notice that the result is ___ but it should be ___.

[tctlrd]

thanks @mohammed90

3. Tutorial (minimal steps to reproduce the bug)

Create first Caddyfile: ./client

{
    local_certs
    on_demand_tls {
        ask http://localhost:5555/
    }
}

http://localhost:5555 {
	respond 200
}

https://client {
    tls {
        on_demand
        issuer acme https://server/acme/local/directory
    }
    root * /usr/share/caddy
    file_server
}

https://server {
    root * /usr/share/caddy
    file_server
}

Start the first container

mkdir client_data

podman run -d --name client \
  --network debug \
  -v "$(pwd)/client:/etc/caddy/Caddyfile:z" \
  -v "$(pwd)/client_data:/data:z" \
  -p 4443:443 \
  docker.io/caddy

Create second Caddyfile: ./server

{
    debug
}

https://server {
    tls /client/caddy/certificates/local/server/server.crt /client/caddy/certificates/local/server/server.key
    acme_server
}

Start the second container.

mkdir server_data

podman run -d --name server \
  --network debug \
  -v "$(pwd)/server:/etc/caddy/Caddyfile:z" \
  -v "$(pwd)/server_data:/data:z" \
  -v "$(pwd)/client_data:/client:z" \
  docker.io/caddy

Add 127.0.0.1 client to /etc/hosts file.

Send query to first container: curl https://client:4443

Check container logs: podman logs client

{"level":"info","ts":1763832993.1185694,"msg":"using config from file","file":"/etc/caddy/Caddyfile"}
{"level":"info","ts":1763832993.1207433,"msg":"adapted config to JSON","adapter":"caddyfile"}
{"level":"warn","ts":1763832993.1207626,"msg":"Caddyfile input is not formatted; run 'caddy fmt --overwrite' to fix inconsistencies","adapter":"caddyfile","file":"/etc/caddy/Caddyfile","line":2}
{"level":"info","ts":1763832993.1233463,"logger":"admin","msg":"admin endpoint started","address":"localhost:2019","enforce_origin":false,"origins":["//[::1]:2019","//127.0.0.1:2019","//localhost:2019"]}
{"level":"info","ts":1763832993.124353,"logger":"tls.cache.maintenance","msg":"started background certificate maintenance","cache":"0xc000523d80"}
{"level":"info","ts":1763832993.1424704,"logger":"http.auto_https","msg":"server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS","server_name":"srv0","https_port":443}
{"level":"info","ts":1763832993.1425338,"logger":"http.auto_https","msg":"enabling automatic HTTP->HTTPS redirects","server_name":"srv0"}
{"level":"info","ts":1763832993.143412,"logger":"http","msg":"enabling HTTP/3 listener","addr":":443"}
{"level":"info","ts":1763832993.1437027,"msg":"failed to sufficiently increase receive buffer size (was: 208 kiB, wanted: 7168 kiB, got: 416 kiB). See https://github.com/quic-go/quic-go/wiki/UDP-Buffer-Sizes for details."}
{"level":"info","ts":1763832993.143878,"logger":"http.log","msg":"server running","name":"srv0","protocols":["h1","h2","h3"]}
{"level":"warn","ts":1763832993.144031,"logger":"http","msg":"HTTP/2 skipped because it requires TLS","network":"tcp","addr":":5555"}
{"level":"warn","ts":1763832993.1440434,"logger":"http","msg":"HTTP/3 skipped because it requires TLS","network":"tcp","addr":":5555"}
{"level":"info","ts":1763832993.1440506,"logger":"http.log","msg":"server running","name":"srv1","protocols":["h1","h2","h3"]}
{"level":"warn","ts":1763832993.14415,"logger":"http","msg":"HTTP/2 skipped because it requires TLS","network":"tcp","addr":":80"}
{"level":"warn","ts":1763832993.1441598,"logger":"http","msg":"HTTP/3 skipped because it requires TLS","network":"tcp","addr":":80"}
{"level":"info","ts":1763832993.1441672,"logger":"http.log","msg":"server running","name":"remaining_auto_https_redirects","protocols":["h1","h2","h3"]}
{"level":"info","ts":1763832993.144178,"logger":"http","msg":"enabling automatic TLS certificate management","domains":["client","server"]}
{"level":"warn","ts":1763832993.1443417,"logger":"pki.ca.local","msg":"installing root certificate (you might be prompted for password)","path":"storage:pki/authorities/local/root.crt"}
{"level":"info","ts":1763832993.1449661,"msg":"warning: \"certutil\" is not available, install \"certutil\" with \"apt install libnss3-tools\" or \"yum install nss-tools\" and try again"}
{"level":"info","ts":1763832993.1449852,"msg":"define JAVA_HOME environment variable to use the Java trust"}
{"level":"info","ts":1763832993.1461627,"logger":"tls","msg":"cleaning storage unit","storage":"FileStorage:/data/caddy"}
{"level":"info","ts":1763832993.146519,"logger":"tls.obtain","msg":"acquiring lock","identifier":"server"}
{"level":"info","ts":1763832993.1474688,"logger":"tls","msg":"finished cleaning storage units"}
{"level":"info","ts":1763832993.1477306,"logger":"tls.obtain","msg":"lock acquired","identifier":"server"}
{"level":"info","ts":1763832993.147893,"logger":"tls.obtain","msg":"obtaining certificate","identifier":"server"}
{"level":"info","ts":1763832993.155135,"logger":"tls.obtain","msg":"certificate obtained successfully","identifier":"server","issuer":"local"}
{"level":"info","ts":1763832993.1553261,"logger":"tls.obtain","msg":"releasing lock","identifier":"server"}
{"level":"warn","ts":1763832993.156047,"logger":"tls","msg":"stapling OCSP","identifiers":["server"]}
{"level":"info","ts":1763832993.2070396,"msg":"certificate installed properly in linux trusts"}
{"level":"info","ts":1763832993.2076292,"msg":"autosaved config (load with --resume flag)","file":"/config/caddy/autosave.json"}
{"level":"info","ts":1763832993.2076485,"msg":"serving initial configuration"}
{"level":"info","ts":1763833010.4826975,"logger":"tls.on_demand","msg":"obtaining new certificate","remote_ip":"10.89.0.1","remote_port":"35030","server_name":"client"}
{"level":"info","ts":1763833010.4905872,"logger":"tls.obtain","msg":"acquiring lock","identifier":"client"}
{"level":"info","ts":1763833010.4915922,"logger":"tls.obtain","msg":"lock acquired","identifier":"client"}
{"level":"info","ts":1763833010.4917595,"logger":"tls.obtain","msg":"obtaining certificate","identifier":"client"}
{"level":"info","ts":1763833010.4928184,"logger":"tls.issuance.acme","msg":"creating new account because no account for configured email is known to us","email":"","ca":"https://server/acme/local/directory","error":"open /data/caddy/acme/server-acme-local-directory/users/default/default.json: no such file or directory"}
{"level":"info","ts":1763833010.4929159,"logger":"tls.issuance.acme","msg":"ACME account has empty status; registering account with ACME server","contact":[],"location":""}
{"level":"info","ts":1763833010.4939184,"logger":"tls.issuance.acme","msg":"creating new account because no account for configured email is known to us","email":"","ca":"https://server/acme/local/directory","error":"open /data/caddy/acme/server-acme-local-directory/users/default/default.json: no such file or directory"}
{"level":"warn","ts":1763833010.5002823,"msg":"HTTP request failed; retrying","url":"https://server/acme/local/directory","error":"performing request: Get \"https://server/acme/local/directory\": tls: failed to verify certificate: x509: certificate signed by unknown authority"}
{"level":"warn","ts":1763833010.7548432,"msg":"HTTP request failed; retrying","url":"https://server/acme/local/directory","error":"performing request: Get \"https://server/acme/local/directory\": tls: failed to verify certificate: x509: certificate signed by unknown authority"}
{"level":"warn","ts":1763833011.009524,"msg":"HTTP request failed; retrying","url":"https://server/acme/local/directory","error":"performing request: Get \"https://server/acme/local/directory\": tls: failed to verify certificate: x509: certificate signed by unknown authority"}
{"level":"error","ts":1763833011.0097425,"logger":"tls.obtain","msg":"could not get certificate from issuer","identifier":"client","issuer":"server-acme-local-directory","error":"registering account [] with server: provisioning client: performing request: Get \"https://server/acme/local/directory\": tls: failed to verify certificate: x509: certificate signed by unknown authority"}
{"level":"error","ts":1763833011.0097966,"logger":"tls.obtain","msg":"will retry","error":"[client] Obtain: registering account [] with server: provisioning client: performing request: Get \"https://server/acme/local/directory\": tls: failed to verify certificate: x509: certificate signed by unknown authority","attempt":1,"retrying_in":60,"elapsed":0.518183028,"max_duration":2592000}

check second container logs: podman logs server

{"level":"debug","ts":1763831940.6168392,"logger":"events","msg":"event","name":"tls_get_certificate","id":"b7801690-ec5e-41d8-9ecc-9e82ea7039b0","origin":"tls","data":{"client_hello":{"CipherSuites":[49195,49199,49196,49200,52393,52392,49161,49171,49162,49172,4865,4866,4867],"ServerName":"server","SupportedCurves":[4588,29,23,24,25],"SupportedPoints":"AA==","SignatureSchemes":[2052,1027,2055,2053,2054,1025,1281,1537,1283,1539],"SupportedProtos":["h2","http/1.1"],"SupportedVersions":[772,771],"RemoteAddr":{"IP":"10.89.0.7","Port":59698,"Zone":""},"LocalAddr":{"IP":"10.89.0.8","Port":443,"Zone":""}}}}
{"level":"debug","ts":1763831940.6169198,"logger":"tls.handshake","msg":"choosing certificate","identifier":"server","num_choices":1}
{"level":"debug","ts":1763831940.6169329,"logger":"tls.handshake","msg":"custom certificate selection results","identifier":"server","subjects":["server"],"managed":false,"issuer_key":"","hash":"1b22c11cb59348e27d61ab9931c542c3a034d0ac0e6a376fd9737a68fea207ee"}
{"level":"debug","ts":1763831940.6169462,"logger":"tls.handshake","msg":"matched certificate in cache","remote_ip":"10.89.0.7","remote_port":"59698","subjects":["server"],"managed":false,"expiration":1763875109,"hash":"1b22c11cb59348e27d61ab9931c542c3a034d0ac0e6a376fd9737a68fea207ee"}
{"level":"debug","ts":1763831940.618525,"logger":"http.stdlib","msg":"http: TLS handshake error from 10.89.0.7:59698: remote error: tls: bad certificate"}

[mholt]

I'm pretty sure in containers the trust store installation can't succeed, right?

[francislavoie]

It can succeed when the container runs as root (default in Docker) but they're using podman here and trying to run it as non-root.

[tctlrd]

@mholt @francislavoie - I was running as root in podman and the trust store install succeeds as seen above.

{"level":"info","ts":1763832993.2070396,"msg":"certificate installed properly in linux trusts"}

I jumped into the container and /etc/ssl/certs/ca-certificates.crt included the caddy root CA cert.

My best guess is that caddy grabs the trust store into memory before installing it's own root CA and thus it does not trust its own CA, though it was properly installed.

[mholt]

Are you sure the cert is installed before the client(s) making the connection are started/running at all?

[tctlrd]

@mholt - Are you asking if the root CA cert is "installed in linux trusts" before the caddy process starts running in that same container? In my tutorial, this is the first container: "client". If so the log indicates that caddy initiates the install of its root CA after generating it... so the answer is no.

[mholt]

I am not sure if Caddy needs a restart if it is to use the system's updated trust store -- it depends on if the Go standard lib loads it at program start, or just as needed... I am not sure how those underlying APIs work. It could be worth a shot restarting after generating and see if that works?

[hslatman]

The trusted roots will generally be initialized once, and subsequent calls to SystemCertsPool may not return the updated roots: https://cs.opensource.google/go/go/+/refs/tags/go1.25.4:src/crypto/x509/cert_pool.go;l=106-116. The actual logic depends on the platform. I believe on Linux it will read from several directories, and do that only once; Windows and Darwin have different behavior with the platform APIs doing the verification with (I believe) the most recent version of their trust stores.

It might be an option to always include Caddy's internal CA as a trusted root on top of the system roots. This is what we do in step-ca too, so that the CA can perform outgoing mTLS requests to servers using a cert from the CA itself.

[tctlrd]

It might be an option to always include Caddy's internal CA as a trusted root on top of the system roots. This is what we do in step-ca too, so that the CA can perform outgoing mTLS requests to servers using a cert from the CA itself.

This seems like a great idea.

[tctlrd]

I am not sure if Caddy needs a restart if it is to use the system's updated trust store -- it depends on if the Go standard lib loads it at program start, or just as needed... I am not sure how those underlying APIs work. It could be worth a shot restarting after generating and see if that works?

Do you have a recommendation on how to have caddy automatically restart in the container after generating?

[tctlrd]

@mohammed90 :mailbox_with_no_mail: do we still need more info or can you remove the label?

[francislavoie]

Looks like there's an open issue in Go for the ability to reload the root cert store https://github.com/golang/go/issues/41888

[francislavoie]

So I think you can fix this by configuring the trusted_roots option in the ACME issuer, see https://caddyserver.com/docs/caddyfile/directives/tls#acme.

Unfortunately this isn't dynamic right now, it just takes a path to a PEM file. So you would use the full path to the storage: /var/lib/caddy/.local/share/caddy/pki/authorities/local/root.crt

We could add support for trust_pool modules, but we would need certmagic to take a callback for the trust chain (in which case the user could configure a merged system + pki_root pool maybe)... but that would still require the user to have the foresight to configure that.

Ultimately I think we should just resign to the fact this is a known issue, and wait until Go adds a way to reload system trust and we can trigger that when we install a root cert. For now, just restarting Caddy should be fine, and it should only ever be a one-time thing when the problem is noticed, cause the root cert has a very long lifetime.

[tctlrd]

@francislavoie thanks for digging into this. I like the plan.

I had tried the trusted_roots route originally, but it makes caddy fail to start when the root cert does not yet exist; so this requires having caddy generate its root cert and then changing the config. This is not a great option for container deployment and IAC...

In the meantime what do you recommend for automated caddy restart after the CA trust install when running in a container?

Or are you thinking there should be persistent storage mounted at /etc/ssl/certs/?