Problem with domain validation - DNS-01, opnsense with bind

[wacki4]

Hi! I'am trying to validate with DNS-01 my subdomain using opnsense acme plugin, and bind.

My DNS works without a problem - it is avaiable from outside, and returns correct IP addresses for entrances which i made.

But i cannot generate certificate cause of error: "invalid domain".

I found out that it is returned by acme.sh/dnsapi/dns_opnsense.sh .

Running /api/bind/domain/get i got in return:

{ "domain": { "domains": { "domain": { "19b51657-5206-4a1f-a0a6-98b7eab8eb51": { "enabled": "1", "type": { "master": { "value": "master", "selected": 1 }, "slave": { "value": "slave", "selected": 0 } }, "masterip": { "": { "value": "", "selected": 1 } }, "allownotifyslave": { "": { "value": "", "selected": 1 } }, "domainname": "test.pl", "allowtransfer": { "": { "value": "none", "selected": 0 }, "7ff76356-1da2-45bd-a733-52fe959f2518": { "value": "all", "selected": 1 } }, "allowquery": { "": { "value": "none", "selected": 0 }, "7ff76356-1da2-45bd-a733-52fe959f2518": { "value": "all", "selected": 1 } }, "serial": "2110031633", "ttl": "86400", "refresh": "21600", "retry": "3600", "expire": "3542400", "negative": "3600", "mailadmin": "admin.test.pl", "dnsserver": "ns.test.pl" } } } } }

And this is my log where problem occurs:

2021-10-03T16:44:56	acme.sh[91474]	] Please check log file for more details: /var/log/acme.sh.log
2021-10-03T16:44:56	acme.sh[7671]	] _on_issue_err
2021-10-03T16:44:56	acme.sh[44902]	] Error add txt for domain:_acme-challenge.aaa.test.pl
2021-10-03T16:44:56	acme.sh[90579]	] invalid domain
2021-10-03T16:44:56	acme.sh[49488]	] h='pl'
2021-10-03T16:44:56	acme.sh[79755]	] h='test.pl'
2021-10-03T16:44:56	acme.sh[5610]	] h='aaa.test.pl'
2021-10-03T16:44:55	acme.sh[90452]	] _hcode='0'
2021-10-03T16:44:55	acme.sh[45503]	] ret='0'
2021-10-03T16:44:55	acme.sh[48565]	] _CURL='curl --silent --dump-header /var/etc/acme-client/home/http.header -L --trace-ascii /tmp/tmp.xdI3cqrJ --insecure '
2021-10-03T16:44:55	acme.sh[74843]	] displayError='1'
2021-10-03T16:44:55	acme.sh[4499]	] timeout=
2021-10-03T16:44:55	acme.sh[33181]	] url='https://[CUT OFF FOR SECURITY]/api/bind/domain/get'
2021-10-03T16:44:55	acme.sh[36307]	] GET
2021-10-03T16:44:55	acme.sh[56945]	] Retrying GET
2021-10-03T16:44:55	acme.sh[30350]	] Detect root zone

Plugin in opnsense is running on version 3.2

[blablup]

I'll look into it. Did you format the json string? Because for me, it looks like there are more white spaces then I get. Maybe I just have to relax the regex

[wacki4]

I didn't format it additionaly. It is copy/paste from log. I was going to look into it also on weekend, will You look sooner?

[blablup]

I also see that you ask acme to create a acme dns entry for aaa.test.pl but the configured domain in opnsense looks like it is only test.pl. Did you create a dedicated (sub)zone aaa.test.pl?

No promise to look into it sooner

[wacki4]

Yes, subzone is aaa.test.pl. I will leave here some info if i will look into it on weekend.

[superwinni2]

Is there anything new to this? Or anyting (simple) what I can test?

[blablup]

Do a test run with the MR #3764 . I only have subdomains and need domain-alias mode on my opnsense boxes. I just registered a new domain for testing, but am not sure If I can do the test today.

[superwinni2]

Sorry but that's too much for me... I don't know how...

[superwinni2]

Something new to this problem?

[blablup]

because the mr is merged it should be fixed. I worked on some other fixes to also fix the CI check, but ran into problems with solaris check. I hadn't time to look into that. But this can be closed in my opinion.

[superwinni2]

I updated OPNssense to 21.1.1 today... It still doesn't work...

My log shows the following: `<14>1 2022-02-16T15:29:23+01:00 OPNsense1.funny.intern acme.sh 12524 - [meta sequenceId="1"] [Wed Feb 16 15:29:23 CET 2022] Using CA: https://acme-v02.api.letsencrypt.org/directory <14>1 2022-02-16T15:29:23+01:00 OPNsense1.funny.intern acme.sh 33255 - [meta sequenceId="2"] [Wed Feb 16 15:29:23 CET 2022] Multi domain='DNS:domain.link,DNS:.domain.link' <14>1 2022-02-16T15:29:23+01:00 OPNsense1.funny.intern acme.sh 39663 - [meta sequenceId="3"] [Wed Feb 16 15:29:23 CET 2022] Getting domain auth token for each domain <14>1 2022-02-16T15:29:27+01:00 OPNsense1.funny.intern acme.sh 66984 - [meta sequenceId="4"] [Wed Feb 16 15:29:27 CET 2022] Getting webroot for domain='domain.link' <14>1 2022-02-16T15:29:27+01:00 OPNsense1.funny.intern acme.sh 78711 - [meta sequenceId="5"] [Wed Feb 16 15:29:27 CET 2022] Getting webroot for domain='.domain.link' <14>1 2022-02-16T15:29:27+01:00 OPNsense1.funny.intern acme.sh 98156 - [meta sequenceId="6"] [Wed Feb 16 15:29:27 CET 2022] Adding txt value: obnjbs2EuzFKecNvu84pdUzgqvEoHfBC-L73jKnPzks for domain: _acme-challenge.home.domain.de <14>1 2022-02-16T15:29:27+01:00 OPNsense1.funny.intern acme.sh 9634 - [meta sequenceId="7"] [Wed Feb 16 15:29:27 CET 2022] Adding record _acme-challenge.home.domain.de with challenge: obnjbs2EuzFKecNvu84pdUzgqvEoHfBC-L73jKnPzks <11>1 2022-02-16T15:29:27+01:00 OPNsense1.funny.intern acme.sh 21862 - [meta sequenceId="8"] [Wed Feb 16 15:29:27 CET 2022] invalid domain <11>1 2022-02-16T15:29:27+01:00 OPNsense1.funny.intern acme.sh 23626 - [meta sequenceId="9"] [Wed Feb 16 15:29:27 CET 2022] Error add txt for domain:_acme-challenge.home.domain.de <11>1 2022-02-16T15:29:27+01:00 OPNsense1.funny.intern acme.sh 25134 - [meta sequenceId="10"] [Wed Feb 16 15:29:27 CET 2022] Please add '--debug' or '--log' to check more details. <11>1 2022-02-16T15:29:27+01:00 OPNsense1.funny.intern acme.sh 26884 - [meta sequenceId="11"] [Wed Feb 16 15:29:27 CET 2022] See: https://github.com/acmesh-official/acme.sh/wiki/How-to-debug-acme.sh

Sorry for dumb question.. How do I know if the MR is inserted?

[richard-9000]

Debugging this myself, looks like the opnsense bind api has changed again. Seems to be a pretty fragile 270 character shell/grep regex to blame due to the new section in the domains response:

"transferkeyalgo":{"":{"value":"none","selected":true} "hmac-sha512":{"value":"HMAC-SHA512","selected":0} "hmac-sha384":{"value":"HMAC-SHA384","selected":0} "hmac-sha256":{"value":"HMAC-SHA256","selected":0} "hmac-sha224":{"value":"HMAC-SHA224","selected":0} "hmac-sha1":{"value":"HMAC-SHA1","selected":0} "hmac-md5":{"value":"HMAC-MD5","selected":0}

I'll get a hack working for it, but I wonder if there's any guidance on better json parsing in acme? A standard json parser wouldn't have these problems.

[richard-9000]

https://github.com/acmesh-official/acme.sh/pull/3948

Working againt os-bind 1.19

This will break again without a proper fix / json parser.

[SBado]

Fix using jq:

_get_root() {
  domain=$1
  i=2
  p=1
  if _opns_rest "GET" "/domain/get"; then
    _domain_response="$response"
  else
    return 1
  fi

  while true; do
    h=$(printf "%s" "$domain" | cut -d . -f $i-100)
    if [ -z "$h" ]; then
      #not valid
      return 1
    fi
    _debug h "$h"
    id=$(echo $response | jq -r '.domain.domains.domain | to_entries[] | .key')
    d=$(echo $response | jq --arg i "$id" -r '.domain.domains.domain[$i].domainname')

    if [ -n "$id" ] && [ $d == $h ]; then
      _debug id "$id"
      _host=$(printf "%s" "$domain" | cut -d . -f 1-$p)
      _domain="${h}"
      _domainid="${id}"
      return 0
    fi
    p=$i
    i=$(_math $i + 1)
  done
  _debug "$domain not found"

  return 1
}

I'm testing it right now, maybe I will submit a PR later.

[koelle25]

Thanks @SBado, I had to modify your jq alternative a bit, because it failed when there are multiple domains in the $domain_response ($id then had multiple IDs in it and that was when your second jq failed). This stackoverflow post helped me with this and now it seems to work like a charm:

_get_root() {
  domain=$1
  i=2
  p=1
  if _opns_rest "GET" "/domain/get"; then
    _domain_response="$response"
  else
    return 1
  fi

  while true; do
    h=$(printf "%s" "$domain" | cut -d . -f $i-100)
    if [ -z "$h" ]; then
      #not valid
      return 1
    fi
    _debug h "$h"
    id=$(echo $response | jq --arg h "$h" -r '.domain.domains.domain | with_entries(select(.value.domainname == $h)) | to_entries[] | .key')

    if [ -n "$id" ]; then
      _debug id "$id"
      _host=$(printf "%s" "$domain" | cut -d . -f 1-$p)
      _domain="${h}"
      _domainid="${id}"
      return 0
    fi
    p=$i
    i=$(_math $i + 1)
  done
  _debug "$domain not found"

  return 1
}

Important note for OPNsense users: You'll have to install the jq package first. Login via SSH then run:

sudo pkg install jq

EDIT 2022-06-24: I have slightly updated the jq string above (replaced .value.domainname|match($h) with just .value.domainname == $h).

Reason: The old version actually returned multiple IDs, if you have defined sub-subdomain zones in your BIND server (e.g. a zone for "example.com" and a zone for "sub.example.com").

Due to this, domain validation fails, despite the _get_root function apparently returning a success status (see log below).

/var/log/acme.sh.log

[Fri Jun 24 14:30:58 CEST 2022] data='{"record":{"enabled":"1","domain":"3137c4c2-4604-4657-8be7-9b1406b0b530
8bcc1f9e-e785-4109-b329-85907137fc6f
b4a85424-0b8d-45f8-b23e-b127801d7984
5cfdbbaf-97fd-46e3-a3e5-07ddb6e18641","name":"_acme-challenge","type":"TXT","value":"***REDACTED***"}}'
[Fri Jun 24 14:30:58 CEST 2022] POST
[Fri Jun 24 14:30:58 CEST 2022] _post_url='https://***REDACTED***:***REDACTED***@***REDACTED***:8443/api/bind/record/addRecord'
[Fri Jun 24 14:30:58 CEST 2022] _CURL='curl --silent --dump-header /var/etc/acme-client/home/http.header  -L  --insecure  '
[Fri Jun 24 14:30:58 CEST 2022] _ret='0'
[Fri Jun 24 14:30:58 CEST 2022] data='{}'
[Fri Jun 24 14:30:58 CEST 2022] POST
[Fri Jun 24 14:30:58 CEST 2022] _post_url='https://***REDACTED***:***REDACTED***@***REDACTED***:8443/api/bind/service/reconfigure'
[Fri Jun 24 14:30:58 CEST 2022] _CURL='curl --silent --dump-header /var/etc/acme-client/home/http.header  -L  --insecure  '
[Fri Jun 24 14:30:59 CEST 2022] _ret='0'
[Fri Jun 24 14:30:59 CEST 2022] Record created
[Fri Jun 24 14:30:59 CEST 2022] The txt record is added: Success.
[Fri Jun 24 14:30:59 CEST 2022] Sleep 30 seconds for the txt records to take effect

[Fri Jun 24 14:31:29 CEST 2022] ok, let's start to verify
[Fri Jun 24 14:31:29 CEST 2022] Verifying: ***REDACTED***
[Fri Jun 24 14:31:29 CEST 2022] d='***REDACTED***'
[Fri Jun 24 14:31:29 CEST 2022] keyauthorization='***REDACTED***'
[Fri Jun 24 14:31:29 CEST 2022] uri='https://acme-v02.api.letsencrypt.org/acme/chall-v3/***REDACTED***'
[Fri Jun 24 14:31:29 CEST 2022] _currentRoot='dns_opnsense'
[Fri Jun 24 14:31:29 CEST 2022] url='https://acme-v02.api.letsencrypt.org/acme/chall-v3/***REDACTED***'
[Fri Jun 24 14:31:29 CEST 2022] payload='{}'
[Fri Jun 24 14:31:29 CEST 2022] POST
[Fri Jun 24 14:31:29 CEST 2022] _post_url='https://acme-v02.api.letsencrypt.org/acme/chall-v3/***REDACTED***'
[Fri Jun 24 14:31:29 CEST 2022] _CURL='curl --silent --dump-header /var/etc/acme-client/home/http.header  -L '
[Fri Jun 24 14:31:30 CEST 2022] _ret='0'
[Fri Jun 24 14:31:30 CEST 2022] code='200'
[Fri Jun 24 14:31:30 CEST 2022] trigger validation code: 200
[Fri Jun 24 14:31:30 CEST 2022] Pending, The CA is processing your order, please just wait. (1/30)
[Fri Jun 24 14:31:30 CEST 2022] sleep 2 secs to verify again
[Fri Jun 24 14:31:32 CEST 2022] checking
[Fri Jun 24 14:31:32 CEST 2022] url='https://acme-v02.api.letsencrypt.org/acme/chall-v3/***REDACTED***'
[Fri Jun 24 14:31:32 CEST 2022] payload
[Fri Jun 24 14:31:32 CEST 2022] POST
[Fri Jun 24 14:31:32 CEST 2022] _post_url='https://acme-v02.api.letsencrypt.org/acme/chall-v3/***REDACTED***'
[Fri Jun 24 14:31:32 CEST 2022] _CURL='curl --silent --dump-header /var/etc/acme-client/home/http.header  -L '
[Fri Jun 24 14:31:32 CEST 2022] _ret='0'
[Fri Jun 24 14:31:32 CEST 2022] code='200'
[Fri Jun 24 14:31:32 CEST 2022] ***REDACTED***:Verify error:DNS problem: NXDOMAIN looking up TXT for _acme-challenge.***REDACTED*** - check that a DNS record exists for this domain

[superwinni2]

I've edited by hand and installed the jq package to test it too. Worked for me. I've tried it with DNS:domain.de,DNS:*.domain.de Certificates with Automatic DNS Alias Mode wich are going to be redirected to my local bind server at home.

[koelle25]

I have slightly updated the jq string in my comment above to mitigate an unlikely but possible error (see edited comment for details).

[SBado]

I completely forgot about this issue (and my fix)! I updated OPNsense the other day and now here I am again. Thanks @koelle25 for your improvements, it's probably time to open that PR.

[blablup]

Sry for late response. I already had a better regex, but I had Problems with the test workflow on some Platforms. I now found a regex which works on all Platforms. @SBado I agree that jq would be nice and easy, but that would add a dependence to acme.sh on all Platforms because this plugin is able to run on remote hosts as well. Also If you still want to use jq you should check that the domain is managed on that host (type:master) and is enabled.

[koelle25]

As the PR got merged this issue can be closed now I think. @wacki4 can you close this issue/mark it as resolved?

[wacki4]

Still Opnsense has old acme client - new change cannot connect to current version on Bind server. I will check everything when there would be avaiable correct client.

[koelle25]

The fix has been integrated into the new 3.0.5 release. Currently OPNsense only delivers version 3.0.4_2. You/we have to wait until the new release arrives in their update repos.

[wacki4]

I have seen it - now manually edited files by jq extension delivered here. When there would be avaiable new version, i will update opnsense and check if everything works ok.

[wacki4]

Confirmed - all works great on 22.7.11_1.