[Bug]: No certificates issued anymore

[ottobaer]

Steps to Reproduce

Hi,

After replacing dnsmasq with bind I can't get any certificates anymore it seems.

Your Environment

• OS - ubuntu 24.04 (arm64) / armbian
• step-ca Version - Smallstep CA/0.26.1 (linux/arm64

Expected Behavior

Expecting to get certificates either via tls-alpn or http challenge.

Actual Behavior

I'm trying to get a certificate with certbot ( I tried with lego and caddy, but I get the same error).

[pandabaer.lan.ursidae.space] failed to initiate challenge: Post "https://pki.lan.ursidae.space:10443/acme/acme/challenge/RDsb1u3j8ewE09bfRmZTzX6JYfvvhMOi/YRan9DzGPuRZ5vQNSnt043hMrz4c5gRh": local error: tls: bad record MAC

On the step-ca side I get this:

May 26 10:29:43 sunbear.lan.ursidae.space step-ca[4382]: time="2024-05-26T10:29:43+02:00" level=info duration=16.508265ms duration-ns=16508265 fields.time="2024-05-26T10:29:43+02:00" method=POST name=ca nonce=QWhoUWViUWRLdElIcDRLbURMNkJIY09XTE9aOWdmUDc path=/acme/acme/authz/RDsb1u3j8ewE09bfRmZTzX6JYfvvhMOi protocol=HTTP/1.1 referer= remote-address=192.168.1.25 request-id=1cff4c8e-54a2-4e70-b6c8-653e0c3f006a response="{\"identifier\":{\"type\":\"dns\",\"value\":\"pandabaer.lan.ursidae.space\"},\"status\":\"pending\",\"challenges\":[{\"type\":\"dns-01\",\"status\":\"pending\",\"token\":\"b76rsopbfenJfiaPF5ecEbEucZYxnIqp\",\"url\":\"https://pki.lan.ursidae.space:10443/acme/acme/challenge/RDsb1u3j8ewE09bfRmZTzX6JYfvvhMOi/VYrG3FFNPaTMS4c6ohZblULShCnsTqNo\"},{\"type\":\"http-01\",\"status\":\"pending\",\"token\":\"b76rsopbfenJfiaPF5ecEbEucZYxnIqp\",\"url\":\"https://pki.lan.ursidae.space:10443/acme/acme/challenge/RDsb1u3j8ewE09bfRmZTzX6JYfvvhMOi/2EkOBiZw7xtYX9vTcWjVmeCzP4IsPBjB\"},{\"type\":\"tls-alpn-01\",\"status\":\"pending\",\"token\":\"b76rsopbfenJfiaPF5ecEbEucZYxnIqp\",\"url\":\"https://pki.lan.ursidae.space:10443/acme/acme/challenge/RDsb1u3j8ewE09bfRmZTzX6JYfvvhMOi/YRan9DzGPuRZ5vQNSnt043hMrz4c5gRh\",\"error\":{\"type\":\"urn:ietf:params:acme:error:connection\",\"detail\":\"The server could not connect to validation target\"}}],\"wildcard\":false,\"expires\":\"2024-05-27T08:29:23Z\"}" size=906 status=200 user-agent="lego-cli/v4.16.1 xenolf-acme/4.16.1 (release; linux; amd64)" user-id=
May 26 10:29:43 sunbear.lan.ursidae.space step-ca[4382]: time="2024-05-26T10:29:43+02:00" level=warning duration=18.269363ms duration-ns=18269363 error="expected POST-as-GET" fields.time="2024-05-26T10:29:43+02:00" method=POST name=ca nonce=eTAzNENhYVNCcVFkeUl1WnUzWnlBQXNVcEROT2hVMlk path=/acme/acme/authz/RDsb1u3j8ewE09bfRmZTzX6JYfvvhMOi protocol=HTTP/1.1 referer= remote-address=192.168.1.25 request-id=0780c212-6d3f-4f1f-889a-fe002506cfc4 response="{\"type\":\"urn:ietf:params:acme:error:malformed\",\"detail\":\"The request message was malformed\"}" size=93 status=400 user-agent="lego-cli/v4.16.1 xenolf-acme/4.16.1 (release; linux; amd64)" user-id=

Interestingly I also get the same reponse when I'm trying to get a certificate on the same machine step-ca runs on

step ca certificate --provisioner acme sunbear.lan.ursidae.space a.crt a.key --ca-url https://pki.lan.ursidae.space:10443/acme/acme/directory --root /etc/caddy/certs/root_ca.crt 
✔ Provisioner: acme (ACME)
Using Standalone Mode HTTP challenge to validate sunbear.lan.ursidae.space . Error!

error validating ACME Challenge at https://pki.lan.ursidae.space:10443/acme/acme/challenge/yqKImXsh79qtWgI3rZJiFnnlpCcYDUmu/9hTWH0bWKJnKwJfYHteKSEPxH1SjMFxh: client POST https://pki.lan.ursidae.space:10443/acme/acme/new-order failed: Post "https://pki.lan.ursidae.space:10443/acme/acme/challenge/yqKImXsh79qtWgI3rZJiFnnlpCcYDUmu/9hTWH0bWKJnKwJfYHteKSEPxH1SjMFxh": stream error: stream ID 17; INTERNAL_ERROR; received from peer

In the logs I get this

May 26 10:46:25 sunbear.lan.ursidae.space step-ca[4382]: time="2024-05-26T10:46:25+02:00" level=info duration=20.02632722s duration-ns=20026327220 fields.time="2024-05-26T10:46:05+02:00" method=POST name=ca nonce=NjVRMjV0TFlVNVRrazNkTlRtTUc4a1JFZ0RwWDRQUzg path=/acme/acme/challenge/yqKImXsh79qtWgI3rZJiFnnlpCcYDUmu/9hTWH0bWKJnKwJfYHteKSEPxH1SjMFxh protocol=HTTP/2.0 referer= remote-address=192.168.1.2 request-id=7ffe45e8-3e9b-4ad2-a1d3-ef946907ea0a response="{\"type\":\"http-01\",\"status\":\"pending\",\"token\":\"RmTig8o5sekhGW3p31wBAAgHU3sXebCn\",\"url\":\"https://pki.lan.ursidae.space:10443/acme/acme/challenge/yqKImXsh79qtWgI3rZJiFnnlpCcYDUmu/9hTWH0bWKJnKwJfYHteKSEPxH1SjMFxh\",\"error\":{\"type\":\"urn:ietf:params:acme:error:connection\",\"detail\":\"The server could not connect to validation target\"}}" size=329 status=200 user-agent="Smallstep CLI/0.26.1 (linux/arm64)" user-id=

This is definitely not a connection problem, this was on the same host also there are no firewalls.

I can connect to the acme server with a browser over https without a problem (this was on the same PC I tried getting the certbot certificate above).

I also tried getting a certificate on another host with a JWT token, this works without a problem.

Additional Context

Here is the DNS information, it looks ok for me.

Here is for the host requesting the certificate

forward pandabaer.lan.ursidae.space

; <<>> DiG 9.18.27 <<>> pandabaer.lan.ursidae.space
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 17475
;; flags: qr aa rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
; COOKIE: 09925ab558533157010000006652f7fce742106bf041e0a2 (good)
;; QUESTION SECTION:
;pandabaer.lan.ursidae.space.   IN      A

;; ANSWER SECTION:
pandabaer.lan.ursidae.space. 1200 IN    A       192.168.1.25

;; Query time: 0 msec
;; SERVER: 192.168.1.2#53(192.168.1.2) (UDP)
;; WHEN: Sun May 26 10:51:08 CEST 2024
;; MSG SIZE  rcvd: 100

reverse 192.168.1.25

; <<>> DiG 9.18.27 <<>> -x 192.168.1.2
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 34539
;; flags: qr aa rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
; COOKIE: 1634039cb73a255a010000006652f816952391d1bfeb8ebf (good)
;; QUESTION SECTION:
;2.1.168.192.in-addr.arpa.      IN      PTR

;; ANSWER SECTION:
2.1.168.192.in-addr.arpa. 1200  IN      PTR     sunbear.lan.ursidae.space.

;; Query time: 0 msec
;; SERVER: 192.168.1.2#53(192.168.1.2) (UDP)
;; WHEN: Sun May 26 10:51:34 CEST 2024
;; MSG SIZE  rcvd: 120

Here for the host step-ca runs on

forward pki.lan.ursidae.space

; <<>> DiG 9.18.27 <<>> pki.lan.ursidae.space
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 50928
;; flags: qr aa rd ra; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
; COOKIE: ffba18ce4970085a010000006652f859d7c015453c7041be (good)
;; QUESTION SECTION:
;pki.lan.ursidae.space.         IN      A

;; ANSWER SECTION:
pki.lan.ursidae.space.  86400   IN      CNAME   sunbear.lan.ursidae.space.
sunbear.lan.ursidae.space. 1200 IN      A       192.168.1.2

;; Query time: 0 msec
;; SERVER: 192.168.1.2#53(192.168.1.2) (UDP)
;; WHEN: Sun May 26 10:52:41 CEST 2024
;; MSG SIZE  rcvd: 116

reverse 192.168.1.2

; <<>> DiG 9.18.27 <<>> -x 192.168.1.2
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 33060
;; flags: qr aa rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
; COOKIE: 425ebda4d519d378010000006652f871cc652545e67aaeb6 (good)
;; QUESTION SECTION:
;2.1.168.192.in-addr.arpa.      IN      PTR

;; ANSWER SECTION:
2.1.168.192.in-addr.arpa. 1200  IN      PTR     sunbear.lan.ursidae.space.

;; Query time: 0 msec
;; SERVER: 192.168.1.2#53(192.168.1.2) (UDP)
;; WHEN: Sun May 26 10:53:05 CEST 2024
;; MSG SIZE  rcvd: 120