acme-dns
acme-dns copied to clipboard
joohoi
ACME-DNS-API not pulling a certificate for itself
Hi I have been working on setting up a acme-dns and have ran into an issue where the web API is not pulling it's own let's encrypt cert. I feel like I am missing something simple but I am to far in to see what is behind me.
Any help would be appreciated
halibut.juggedfish.com will be playing the part of my public dns record and 8.8.8.8 will be playing the part of my public ip address
ERROR
root@halibut ~]# systemctl status acme-dns -l
? acme-dns.service - Limited DNS server with RESTful HTTP API to handle ACME DNS challenges easily and securely
Loaded: loaded (/etc/systemd/system/acme-dns.service; enabled; vendor preset: disabled)
Active: active (running) since Thu 2022-12-22 09:49:24 EST; 46min ago
Main PID: 737 (acme-dns)
CGroup: /system.slice/acme-dns.service
+-737 /usr/local/bin/acme-dns
Dec 22 09:49:24 halibut.juggedfish.com systemd[1]: Started Limited DNS server with RESTful HTTP API to handle ACME DNS challenges easily and securely.
Dec 22 09:49:24 halibut.juggedfish.com acme-dns[737]: time="2022-12-22T09:49:24-05:00" level=info msg="Using config file" file=/etc/acme-dns/config.cfg
Dec 22 09:49:48 halibut.juggedfish.com acme-dns[737]: time="2022-12-22T09:49:48-05:00" level=info msg="http: TLS handshake error from 10.248.2.1:55191: no certificate available for 'halibut.juggedfish.com'"
Dec 22 09:49:48 halibut.juggedfish.com acme-dns[737]: time="2022-12-22T09:49:48-05:00" level=info msg="http: TLS handshake error from 10.248.2.1:55192: no certificate available for 'halibut.juggedfish.com'"
Dec 22 09:49:48 halibut.juggedfish.com acme-dns[737]: time="2022-12-22T09:49:48-05:00" level=info msg="http: TLS handshake error from 10.248.2.1:55193: no certificate available for 'halibut.juggedfish.com'"
Dec 22 09:49:48 halibut.juggedfish.com acme-dns[737]: time="2022-12-22T09:49:48-05:00" level=info msg="http: TLS handshake error from 10.248.2.1:55194: no certificate available for 'halibut.juggedfish.com'"
CONFIG
# ================= DO NOT MODIFY THIS FILE =================
#
# Manual changes will be lost when this file is regenerated.
#
# Please read the developer's guide, which is available
# at NethServer official site: https://www.nethserver.org
#
#
[general]
# dns interface
listen = "10.110.144.53:1053"
# protocol, "udp", "udp4", "udp6" or "tcp", "tcp4", "tcp6"
protocol = "both"
# domain name to serve the requests off of
domain = "halibut.juggedfish.com"
# zone name server
nsname = "ns1.halibut.juggedfish.com"
# admin email address, where @ is substituted with .
nsadmin = "[email protected]"
# predefined records served in addition to the TXT
records = [
# default A
"halibut.juggedfish.com. A 8.8.8.8",
# A
"ns1.halibut.juggedfish.com. A 8.8.8.8",
"ns2.halibut.juggedfish.com. A 8.8.8.8",
# NS
"halibut.juggedfish.com. NS ns1.halibut.juggedfish.com.",
"halibut.juggedfish.com. NS ns2.halibut.juggedfish.com.",
]
# debug messages from CORS etc
debug = true
[database]
# Database engine to use, sqlite3 or postgres
engine = "sqlite3"
connection = "/etc/acme-dns/acme-dns.db"
[api]
# listen ip eg. 127.0.0.1
ip = "10.110.144.53"
# disable registration endpoint
disable_registration = false
# listen port, eg. 443 for default HTTPS
port = "443"
# possible values: "letsencrypt", "letsencryptstaging", "cert", "none"
tls = "letsencrypt"
# only used if tls = "letsencrypt"
acme_cache_dir = "/etc/acme-dns/api-certs"
# CORS AllowOrigins, wildcards can be used
corsorigins = [
"*"
]
# use HTTP header to get the client ip
use_header = false
# header name to pull the ip address / list of ip addresses from
header_name = "X-Forwarded-For"
[logconfig]
# logging level: "error", "warning", "info" or "debug"
loglevel = "warning"
# possible values: stdout, TODO file & integrations
logtype = "stdout"
# file path for logfile TODO
# logfile = "./acme-dns.log"
# format, either "json" or "text"
logformat = "text"
PUBLIC DNS
A
ns1.halibut
8.8.8.8
15 minutes
A
ns2.halibut
8.8.8.8
15 minutes
CNAME
_acme-challenge.halibut
halibut.juggedfish.com
15 minutes
NS
halibut
ns2.halibut.juggedfish.com
15 minutes
NS
halibut
ns1.halibut.juggedfish.com
15 minutes
Are port 53 DNS queries against your instance working? You are listening on 1053 but will need to port forward this externally from 53 for normal DNS queries to work (you may already be doing that). I'd imagine if that doesn't work then it also won't be able to use itself to complete a DNS challenge for it's own cert. There was/used to be an http-01 challenge mode but I don't know the config to use that instead of DNS validation.
Hi @protogenxl, I ran into a similar issue.
In my case, I was running the acme-dns.service as a non-root user, and the user did not have write permission in his home directory. By default, the service uses WorkingDirectory=~.
Are you running the service as root?
EDIT
Does the user runner the service has write permissions in /etc/acme-dns/api-certs?
@webprofusion-chrisc yes the DNS forward on my firewall seems to working correctly
2022-12-27 08:11:12 Allow 8.0.38.4 8.8.8.8 dns/udp 53597 53 6-WAN 1-Trusted ProxyAllow: DNS question match (LetsEncryptDNS-proxy-00) DNS-Incoming.1 proc_id="dns-proxy" rc="590" msg_id="1DFF-000E" proxy_act="DNS-Incoming.1" rule_name="*" query_type="A" question="nS2.halibut.juggedfish.com" | Traffic
2022-12-27 08:11:12 Allow 8.0.38.4 8.8.8.8 dns/udp 53597 53 6-WAN 1-Trusted Allowed 82 55 (LetsEncryptDNS-proxy-00) proc_id="firewall" rc="100" msg_id="3000-0148" dst_ip_nat="10.110.144.53" dst_port_nat="1053" | Traffic
@novakele I am running the service as acme-dns and permissions appear to be correct
[root@halibut ~]# cd /etc/acme-dns/api-certs
[root@halibut api-certs]# ls -lah
total 0
drwx------ 4 992 acme-dns 100 Dec 27 08:03 .
drwxrwxr-x 3 root acme-dns 60 Dec 21 06:17 ..
drwx------ 3 992 acme-dns 60 Dec 20 23:53 acme
drwx------ 2 992 acme-dns 142 Dec 27 08:03 locks
-rw------- 1 992 acme-dns 0 Dec 21 06:17 rw_test_288443979776093768
-rw------- 1 992 acme-dns 0 Dec 21 06:17 rw_test_9142020181166123590
It is strange that the owner uid (992) does not resolve to the user acme-dns.
Could you provide the output of id acme-dns? Should the uid of acme-dns be anything else than 992, that is your problem.
Here are the permissions for my instance:
root@lighthouse:~# tree -pufidg /var/lib/acme-dns/
[drwxr-xr-x acme-dns acme-dns] /var/lib/acme-dns
[drwx------ acme-dns acme-dns] /var/lib/acme-dns/api-certs
[drwx------ acme-dns acme-dns] /var/lib/acme-dns/api-certs/acme
[drwx------ acme-dns acme-dns] /var/lib/acme-dns/api-certs/acme/acme-v02.api.letsencrypt.org-directory
[drwx------ acme-dns acme-dns] /var/lib/acme-dns/api-certs/acme/acme-v02.api.letsencrypt.org-directory/users
[drwx------ acme-dns acme-dns] /var/lib/acme-dns/api-certs/acme/acme-v02.api.letsencrypt.org-directory/users/<EMAIL>
[drwx------ acme-dns acme-dns] /var/lib/acme-dns/api-certs/certificates
[drwx------ acme-dns acme-dns] /var/lib/acme-dns/api-certs/certificates/acme-v02.api.letsencrypt.org-directory
[drwx------ acme-dns acme-dns] /var/lib/acme-dns/api-certs/certificates/acme-v02.api.letsencrypt.org-directory/<DOMAIN>
[drwx------ acme-dns acme-dns] /var/lib/acme-dns/api-certs/locks
[drwx------ acme-dns acme-dns] /var/lib/acme-dns/api-certs/ocsp
Also, here is the output of the same commands you ran (I use /var/lib/acme-dns instead of /etc/acme-dns for the home directory):
root@lighthouse:~# cd /var/lib/acme-dns/api-certs/
root@lighthouse:/var/lib/acme-dns/api-certs# ls -lah
total 24K
drwx------ 6 acme-dns acme-dns 4.0K Dec 23 22:19 .
drwxr-xr-x 3 acme-dns acme-dns 4.0K Dec 23 23:31 ..
drwx------ 3 acme-dns acme-dns 4.0K Dec 23 22:19 acme
drwx------ 3 acme-dns acme-dns 4.0K Dec 23 22:19 certificates
drwx------ 2 acme-dns acme-dns 4.0K Dec 23 22:19 locks
drwx------ 2 acme-dns acme-dns 4.0K Dec 23 22:19 ocsp
See #315, I encountered similar problems. I am using the Dockerfile to run acme-dns. The v0.8 release works just fine. In my opinion this is not an environment problem, but a problem of the software itself.
On my profile I've got an improved Dockerfile based on the v0.8 release.
I'm seeing similar behavior to what has been reported here and in #315 on a new installation in FreeBSD 13.1-RELEASE-p5 and the current upstream compiled and packaged acme-dns-1.0_3,1 (installed today).
- DNS queries from outside the perimeter, directed at the public DNS name of the delegation, are answered as expected (both expecting a result as well as expecting NXDOMAIN) Edit: SOA record also confirmed served to outside request
- Trying to connect to the HTTP API using curl results in a TLS handshake error
- There are no suggestions in the logs of any issues
- There do not appear to be any permission problems on the file system (other than acme-dns configured by the port author to be running as root)
time="2023-02-01T19:13:53-08:00" level=info msg="Using config file" file=/usr/local/etc/acme-dns/config.cfg
time="2023-02-01T19:13:53-08:00" level=info msg="Connected to database"
time="2023-02-01T19:13:53-08:00" level=debug msg="Adding new record to domain" domain=<DELEGATED_NS_NAME>. recordtype=A
time="2023-02-01T19:13:53-08:00" level=debug msg="Adding new record to domain" domain=<DELEGATED_NS_NAME>. recordtype=NS
time="2023-02-01T19:13:53-08:00" level=debug msg="Adding new record to domain" domain=<DELEGATED_NS_NAME>. recordtype=SOA
time="2023-02-01T19:13:53-08:00" level=info msg="Listening HTTPS" domain=<DELEGATED_NS_NAME> host="0.0.0.0:443"
time="2023-02-01T19:13:53-08:00" level=info msg="Listening DNS" addr= proto=tcp
time="2023-02-01T19:13:53-08:00" level=info msg="Listening DNS" addr= proto=udp
time="2023-02-01T19:14:36-08:00" level=debug msg="Answering question for domain" domain=should.fail.<MY_DOMAIN>. qtype=A rcode=NXDOMAIN
time="2023-02-01T19:15:11-08:00" level=info msg="http: TLS handshake error from <TEST_HOST_IP>:57801: no certificate available for '<DELEGATED_NS_NAME>'"
On a subsequent restart, I additionally get
time="2023-02-01T19:29:33-08:00" level=info msg="2023/02/01 19:29:33 [INFO][FileStorage:/var/db/acme-dns/api-certs] Lock for 'issue_cert_<DELEGATED_NS_NAME>' is stale (created: 2023-02-01 19:13:53.163243238 -0800 PST, last update: 2023-02-01 19:29:23.133675565 -0800 PST); removing then retrying: /var/db/acme-dns/api-certs/locks/issue_cert_<DELEGATED_NS_NAME>.lock"
sudo find var/db/acme-dns/api-certs/ -type d -exec ls -ld {} \;
drwx------ 4 root wheel 4 Feb 1 19:13 var/db/acme-dns/api-certs/
drwx------ 2 root wheel 3 Feb 1 19:13 var/db/acme-dns/api-certs/locks
drwx------ 4 root wheel 4 Feb 1 19:15 var/db/acme-dns/api-certs/acme
drwx------ 3 root wheel 3 Feb 1 19:13 var/db/acme-dns/api-certs/acme/acme-v02.api.letsencrypt.org-directory
drwx------ 3 root wheel 3 Feb 1 19:13 var/db/acme-dns/api-certs/acme/acme-v02.api.letsencrypt.org-directory/users
drwx------ 2 root wheel 4 Feb 1 19:13 var/db/acme-dns/api-certs/acme/acme-v02.api.letsencrypt.org-directory/users/<NOTIFICATON_EMAIL>
drwx------ 3 root wheel 3 Feb 1 19:15 var/db/acme-dns/api-certs/acme/acme-staging-v02.api.letsencrypt.org-directory
drwx------ 3 root wheel 3 Feb 1 19:15 var/db/acme-dns/api-certs/acme/acme-staging-v02.api.letsencrypt.org-directory/users
drwx------ 2 root wheel 4 Feb 1 19:15 var/db/acme-dns/api-certs/acme/acme-staging-v02.api.letsencrypt.org-directory/users/<NOTIFICATON_EMAIL>
$ egrep -v '^#' usr/local/etc/acme-dns/config.cfg
[general]
protocol = "both"
domain = "<DELEGATED_NS_NAME>"
nsname = "<DELEGATED_NS_NAME>"
nsadmin = "<[email protected]>"
records = [
# domain pointing to the public IP of your acme-dns server
"<DELEGATED_NS_NAME>. A <DELEGATED_NS_PUBLIC_IP>",
# specify that auth.example.org will resolve any *.auth.example.org records
"<DELEGATED_NS_NAME>. NS <DELEGATED_NS_NAME>.",
]
debug = true
[database]
engine = "sqlite3"
connection = "/var/db/acme-dns/acme-dns.db"
[api]
ip = "0.0.0.0"
disable_registration = false
port = "443"
tls = "letsencrypt"
acme_cache_dir = "/var/db/acme-dns/api-certs"
notification_email = "<NOTIFICATON_EMAIL>"
corsorigins = [
"*"
]
use_header = false
header_name = "X-Forwarded-For"
[logconfig]
loglevel = "debug"
logtype = "stdout"
logformat = "text"
Confirmed still an issue in FreeBSD 13.2 and package acme-dns-1.0_12,1
No changes in behavior identified
Might be related to this https://github.com/joohoi/acme-dns/issues/339#issuecomment-1935928848
I did try adding the CAA record as described on #339 with no change in behavior
Removing the entire /var/db/acme-dns/api-certs directory and allowing it to be recreated also did not change the behavior.
It appears that the query is coming in for the TXT record. The following appears to repeat periodically.
time="2024-02-11T17:41:11-08:00" level=debug msg="Answering question for domain" domain=<DELEGATED_NS_NAME>. qtype=AAAA rcode=NOERROR
time="2024-02-11T17:41:11-08:00" level=debug msg="Answering question for domain" domain=_Acme-cHALlENgE.<DELEGATED_NS_NAME>. qtype=TXT rcode=NOERROR
time="2024-02-11T17:41:11-08:00" level=debug msg="Answering question for domain" domain=<DELEGATED_NS_NAME>. qtype=AAAA rcode=NOERROR
time="2024-02-11T17:41:11-08:00" level=debug msg="Answering question for domain" domain=_AcMe-chaLLEnGe.<DELEGATED_NS_NAME>. qtype=TXT rcode=NOERROR
time="2024-02-11T17:41:11-08:00" level=debug msg="Answering question for domain" domain=<DELEGATED_NS_NAME>. qtype=AAAA rcode=NOERROR
time="2024-02-11T17:41:11-08:00" level=debug msg="Answering question for domain" domain=<f4...UUID...b1>.<DELEGATED_NS_NAME>. qtype=TXT rcode=NOERROR
time="2024-02-11T17:41:11-08:00" level=debug msg="Answering question for domain" domain=<DELEGATED_NS_NAME>. qtype=AAAA rcode=NOERROR
time="2024-02-11T17:41:11-08:00" level=debug msg="Answering question for domain" domain=_aCME-CHALlENgE.<DELEGATED_NS_NAME>. qtype=TXT rcode=NOERROR
time="2024-02-11T17:41:11-08:00" level=debug msg="Answering question for domain" domain=<DELEGATED_NS_NAME>. qtype=AAAA rcode=NOERROR
time="2024-02-11T17:41:11-08:00" level=debug msg="Answering question for domain" domain=<f4...UUID...b1>.<DELEGATED_NS_NAME>. qtype=TXT rcode=NOERROR
time="2024-02-11T17:41:11-08:00" level=debug msg="Answering question for domain" domain=<DELEGATED_NS_NAME>. qtype=AAAA rcode=NOERROR
time="2024-02-11T17:41:11-08:00" level=debug msg="Answering question for domain" domain=<f4...UUID...b1>.<DELEGATED_NS_NAME>. qtype=TXT rcode=NOERROR
The requests for the TXT record come in from a variety of IP addresses, including one that reverse-resolves to outbound1g.letsencrypt.org and another to outbound2e.letsencrypt.org
There is no certificate present anywhere under /var/db/acme-dns/api-certs and openssl s_client confirms that there is no certificate available.
Edit: For clarity, the instance works through the API to have other clients renew certificates. It just isn't able to take care of its own.