security/acme-client: uploads outdated certificates to truenas and proxmox

[Agrigor]

Important notices Before you add a new report, we ask you kindly to acknowledge the following:

• [x] I have read the contributing guide lines at https://github.com/opnsense/plugins/blob/master/CONTRIBUTING.md
• [x] I have searched the existing issues, open and closed, and I'm convinced that mine is new.
• [x] The title contains the plugin to which this issue belongs

Describe the bug I am using the latest opnsense (OPNsense 25.1.4_1-amd64) with acme-plugin (v4.9) and a working acme infrastructure for multiple domains and Let's encrypt. One of the domains (opensb.de) is only for internal purposes and includes several automations which essentially upload the certificate via SFTP and custom remote commands via SSH. These are all working! In addition the automations for uploading the certificate to truenas and proxmox are in use, connect and upload the certificate succesfully, but unfortunately the wrong or just an old one since I've created the certificate in 2024 initally. It seems to upload the initial one. If look at the mdates in the specific directory where the certs which are uploaded are located they all have the date of 2024 (see screenshot below)

To Reproduce Steps to reproduce the behavior:

1. Go to Services > ACME Client > Certificates > run automations or renew certificate
2. See error in syslog (AcmeClient: AcmeClient: The shell command returned exit code '0' ...)
3. See valid from and valid until date, which is a time period back in 2024

Expected behavior Uploaded current valid / renewed certificate

Relevant log files

<15>1 2025-03-26T22:57:37+01:00 opnsense.opensb.de acme.sh 4181 - [meta sequenceId="92"] [Wed Mar 26 22:57:37 CET 2025] Using server: https://acme-v02.api.letsencrypt.org/directory <15>1 2025-03-26T22:57:37+01:00 opnsense.opensb.de acme.sh 9345 - [meta sequenceId="93"] [Wed Mar 26 22:57:37 CET 2025] Running cmd: deploy <15>1 2025-03-26T22:57:37+01:00 opnsense.opensb.de acme.sh 13189 - [meta sequenceId="94"] [Wed Mar 26 22:57:37 CET 2025] Using config home: /var/etc/acme-client/home <15>1 2025-03-26T22:57:37+01:00 opnsense.opensb.de acme.sh 17644 - [meta sequenceId="95"] [Wed Mar 26 22:57:37 CET 2025] ACME_DIRECTORY='https://acme-v02.api.letsencrypt.org/directory' <15>1 2025-03-26T22:57:37+01:00 opnsense.opensb.de acme.sh 26740 - [meta sequenceId="96"] [Wed Mar 26 22:57:37 CET 2025] DOMAIN_PATH='/var/etc/acme-client/cert-home/6624efc6c904d7.93985714/opensb.de_ecc' <15>1 2025-03-26T22:57:37+01:00 opnsense.opensb.de acme.sh 36218 - [meta sequenceId="97"] [Wed Mar 26 22:57:37 CET 2025] _deployApi='/usr/local/share/examples/acme.sh/deploy/proxmoxve.sh' <15>1 2025-03-26T22:57:37+01:00 opnsense.opensb.de acme.sh 39132 - [meta sequenceId="98"] [Wed Mar 26 22:57:37 CET 2025] _cdomain='opensb.de' <15>1 2025-03-26T22:57:37+01:00 opnsense.opensb.de acme.sh 43639 - [meta sequenceId="99"] [Wed Mar 26 22:57:37 CET 2025] _ccert='/var/etc/acme-client/cert-home/6624efc6c904d7.93985714/opensb.de_ecc/opensb.de.cer' <15>1 2025-03-26T22:57:37+01:00 opnsense.opensb.de acme.sh 46358 - [meta sequenceId="100"] [Wed Mar 26 22:57:37 CET 2025] _cca='/var/etc/acme-client/cert-home/6624efc6c904d7.93985714/opensb.de_ecc/ca.cer' <15>1 2025-03-26T22:57:37+01:00 opnsense.opensb.de acme.sh 51357 - [meta sequenceId="101"] [Wed Mar 26 22:57:37 CET 2025] _cfullchain='/var/etc/acme-client/cert-home/6624efc6c904d7.93985714/opensb.de_ecc/fullchain.cer' <15>1 2025-03-26T22:57:37+01:00 opnsense.opensb.de acme.sh 82068 - [meta sequenceId="102"] [Wed Mar 26 22:57:37 CET 2025] TARGET_URL='https://proxmox.opensb.de:8006/api2/json/nodes/proxmox/certificates/custom' <14>1 2025-03-26T22:57:37+01:00 opnsense.opensb.de acme.sh 25634 - [meta sequenceId="103"] [Wed Mar 26 22:57:37 CET 2025] Push certificates to server <15>1 2025-03-26T22:57:37+01:00 opnsense.opensb.de acme.sh 28439 - [meta sequenceId="104"] [Wed Mar 26 22:57:37 CET 2025] POST <15>1 2025-03-26T22:57:37+01:00 opnsense.opensb.de acme.sh 31859 - [meta sequenceId="105"] [Wed Mar 26 22:57:37 CET 2025] _post_url='https://proxmox.opensb.de:8006/api2/json/nodes/proxmox/certificates/custom' <15>1 2025-03-26T22:57:37+01:00 opnsense.opensb.de acme.sh 42943 - [meta sequenceId="106"] [Wed Mar 26 22:57:37 CET 2025] _CURL='curl --silent --dump-header /var/etc/acme-client/home/http.header -L --trace-ascii /tmp/tmp.hqlRFG7ElZ -g --insecure ' <15>1 2025-03-26T22:57:37+01:00 opnsense.opensb.de acme.sh 46999 - [meta sequenceId="107"] [Wed Mar 26 22:57:37 CET 2025] _ret='0' <14>1 2025-03-26T22:57:37+01:00 opnsense.opensb.de acme.sh 51483 - [meta sequenceId="108"] [Wed Mar 26 22:57:37 CET 2025] Success

Screenshots

Additional context You can see at the screenshot, the config was updated today, but not the certificates.

Environment OPNsense 25.1.4_1-amd64 as Proxmox VM

[Agrigor]

Has anyone looked into this problem? :/

[lwwilliam]

I’m experiencing the same issue on OPNsense. When I check the certificate files, the certs folder (e.g. /var/etc/acme-client/certs//cert.pem) contains the renewed and valid certificate. However, the cert-home folder (e.g. /var/etc/acme-client/cert-home//) still holds the expired certificate.

I found this problem when i found out that my uploaded synology cert is expired

[fraenki]

Please try to renew the certificate and then provide the following log files:

• Services: ACME Client: Log Files -> System Log (or /var/log/system/latest.log)
• Services: ACME Client: Log Files -> ACME Log (or /var/log/acmeclient/latest.log)

Both log files are required.

[Agrigor]

Hi Fraenki, thanks for caring, see files attached. KR

ps.: Interesting to see, that there are no additional logs inside the acmeclient-latest.log after renewing cert o0

acmeclient-latest.log system-latest.log

[Agrigor]

Okay, the symlink of the latest file is pointing to an old logfile, so I attached the most current one.

acmeclient_20250617.log

[fraenki]

I can't find any issues. From the perspective of Acme Client, all commands ran without any error:

AcmeClient: successfully issued/renewed certificate: REDACTED
AcmeClient: running automation (acme.sh): Upload to TrueNAS
AcmeClient: The shell command returned exit code '0': '/usr/local/sbin/acme.sh --deploy (...) --deploy-hook truenas  (...)
AcmeClient: running automation (acme.sh): Upload to Proxmox
AcmeClient: The shell command returned exit code '0': '/usr/local/sbin/acme.sh --deploy (...) --deploy-hook proxmoxve (...)

And the acme.sh log also confirms that the Proxmox deploy hook ran successfully:

# Log excerpt for Proxmox deploy hook

Running cmd: deploy
...
Push certificates to server
...
_post_url='https://proxmox.REDACTED:8006/api2/json/nodes/proxmox
...
Success

However, it seems to fails when trying to remove the old certificate from TrueNAS, but it still reports overall success:

# Log excerpt for TrueNAS deploy hook

Running cmd: deploy
...
Detected TrueNAS system os: SCALE
Detected TrueNAS system version: 24.10
Getting current active certificate from TrueNAS
...
Active_UI_Certificate_ID='20
...
Uploading new certificate to TrueNAS
...
Current activate certificate ID: 26
...
Deleting old certificate
...
Here is the curl dump log:
== Info: URL rejected: Malformed input to a URL function
...
Success

This issue is unrelated to Acme Client in OPNsense, please check the upstream acme.sh project for further information: https://github.com/acmesh-official/acme.sh/issues/6343

If look at the mdates in the specific directory where the certs which are uploaded are located they all have the date of 2024 (see screenshot below)

Judging from the most recent logs, cert renewal was successful. The updated cert should be available in both locations, /var/etc/acme-client/certs/6624efc6c904d7.93985714/cert.pem (used by Acme Client) as well as /var/etc/acme-client/cert-home/6624efc6c904d7.93985714/REDACTED_ecc/REDACTED.cer (used by acme.sh deploy hooks).

[Agrigor]

Hi, regarding the removing of old certs, I will look afterwards and thanks for your effort!

But regarding the outdated certs, the problem is still existing:

Is there any log, which can give us a hint, why they are not overwritten? (it's actually weird, the config was changed, but not all the other files)

[fraenki]

Hm. Maybe acme.sh is misconfigured and uses the wrong CERT_HOME. Please run the following command:

grep -R CERT_HOME /var/etc/acme-client/accounts

And also check the version of the Acme Client config:

grep 'AcmeClient version' /conf/config.xml

[Agrigor]

Hey there, the outputs are:

[lwwilliam]

Hi, I’d like to add that I’m seeing what seems to be the same underlying issue, but with the synology_dsm deploy hook instead of Proxmox.

My problem in detail:

When I renew or create a new certificate in OPNsense ACME, it correctly creates or updates:

• /var/etc/acme-client/home/_ecc/
• /var/etc/acme-client/certs/<cert_id>/
• /var/etc/acme-client/keys/<cert_id>/

But it does not create or update:

/var/etc/acme-client/cert-home/<cert_id>/ That folder either remains stale with an old expired certificate, or doesn’t exist at all for new certificates. What happens next:

The deploy command run by the plugin still uses the cert-home path (which is wrong or missing):

/usr/local/sbin/acme.sh --deploy --syslog 6 --log-level 1 --server 'letsencrypt'
--home '/var/etc/acme-client/home'
--cert-home '/var/etc/acme-client/cert-home/<cert_id>'
--certpath '/var/etc/acme-client/certs/<cert_id>/cert.pem'
--keypath '/var/etc/acme-client/keys/<cert_id>/private.key'
--capath '/var/etc/acme-client/certs/<cert_id>/chain.pem'
--fullchainpath '/var/etc/acme-client/certs/<cert_id>/fullchain.pem'
--domain 'domain.com' --ecc --deploy-hook synology_dsm This means the deploy hook (in my case synology_dsm) ends up uploading the wrong/expired certificate or fails because the cert-home folder doesn’t even exist.

[fraenki]

@Agrigor, thanks for providing the necessary information. So this confirms that for unknown reason your acme.sh account config is invalid. The next version of Acme Client will automatically verify and fix all account configs.

The log will also contain one of these two messages: AcmeClient: fixing invalid account config (CERT_HOME) AcmeClient: account config is valid (CERT_HOME)

If you don't want to wait for the next release, you may try to apply this fix manually: opnsense-patch -c plugins 4215dae9d51a59c10dac39dd144998161b0091e9 Afterwards renew the certificate, this will trigger the verification of the account config.

[Agrigor]

Hey Fraenki, after a long holiday season and newly release of OpnSense 25.7, I finally did the update and it worked all out flawlessly and without any additional action necessary by me (except the renewing trigger^^). Thanks a lot and kind regards :)