Initial cert request fails when fallback certs are present

[jdeepwell]

trafficstars

We are in the transition phase from certbot, so currently all our virtual Apache hosts have valid Let's Encrypt certs generated by certbot.

Procedure:

• We activate mod_md (by adding the domain via MDomain ... but leave the certbot certs in as fallback.
• After mod_md has done the staging we remove the fallback certs and restart the server.

This worked fine for the first servers...

But now we continuously receive a "Received certificate which is not self-signed" error from Let's Encrypt:

[Wed Apr 02 15:16:36.998152 2025] [md:error] [pid 22744:tid 131228598535872] ACME server authz: challenge 'invalid' for mdcworx.mdc-berlin.de at https://acme-v02.api.letsencrypt.org/acme/authz/.../.... Exact response was: {"identifier":{"type":"dns","value":"mdcworx.mdc-berlin.de"},"status":"invalid","expires":"2025-04-09T13:16:35Z","challenges":[{"type":"tls-alpn-01","url":"https://acme-v02.api.letsencrypt.org/acme/chall/.../.../...","status":"invalid","validated":"2025-04-02T13:16:35Z","error":{"type":"urn:ietf:params:acme:error:unauthorized","detail":"Incorrect validation certificate for tls-alpn-01 challenge. Requested mdcworx.mdc-berlin.de from 77.244.241.77:443. Received certificate which is not self-signed.","status":403},"token":"...","validationRecord":[{"hostname":"mdcworx.mdc-berlin.de","port":"443","addressesResolved":["77.244.241.77"],"addressUsed":"77.244.241.77"}]}]}

I'm puzzled by the error message. I was under the impression that mod_md would use its self-signed staging certs when receiving the tls-alpn-01 challenge and only use the fallback certs (the ones still configured with SSLCertificateFile, etc.) for "normal" requests.

I'm not sure what has changed (we compared the hack out of the config for the servers that worked that way and the ones that didn't - with no result).

My question: Should this work the way I described above? And if so, any idea for a reason for the error message we are receiving from Let's Encypt now?

When we remove the old certbot certs immediately the whole process works fine – but then the servers are unreachable during staging....

---

Config details:

Ubuntu 24.04 LTS, Apache 2.4.52 mod_mod as bundled with Ubuntu/Apache

In main Apache config:

MDContactEmail [email protected]
MDCertificateAgreement accepted
Protocols http/1.1 acme-tls/1
MDRenewMode auto
MDRequireHttps permanent

In VirtualHost configs:

MDomain xxx.xxx.xxx

<VirtualHost *:443>
	 ServerName xxx.xxx.xxx
	 DocumentRoot /var/www/...

	 SSLEngine on
         # remark below after mod_md staging
	 SSLCertificateFile /etc/letsencrypt/live/wd22.artware.eu/cert.pem
	 SSLCertificateKeyFile /etc/letsencrypt/live/wd22.artware.eu/privkey.pem
	 SSLCertificateChainFile /etc/letsencrypt/live/wd22.artware.eu/chain.pem
...

[icing]

The complication with how the TLS-ALPN-01 challenge works in Apache is that the "challenge" cert is added to the SSL* context of OpenSSL and does not replace the already existing ones. Normally, if a site has only RSA certificates and you do not use MDPrivateKeys, the challenge cert will also be RSA and is selected by OpenSSL during the handshake with Let's Encrypt.

If a site uses other cert types, e.g. EC private keys, things get more complicated. There was a recent issue #377 where MDPrivateKeys was used and the challenge cert was not selected, leading to failures.

Without knowing more details about the site that fails and the config/certs in play, it is hard to say.