cat: /etc/ssl/certs/2e5ac55d.0: No such file or directory

[anastasios-salis]

Certbot-zimbra, will not renew certificate and will give me the message: No such file or directory

below is the screen output.

root@mail:~# certbot_zimbra.sh -n -H mail.howtosurvey.gr -e mx.sodanet.gr certbot-zimbra v0.7.12 - https://github.com/YetOpen/certbot-zimbra Checking for dependencies... Detected Zimbra 8.8.15 on UBUNTU18_64 Using domain mail.howtosurvey.gr (as certificate DN) Got 1 domains to use as certificate SANs: mx.sodanet.gr Checking zimbra-proxy is running and enabled Detecting port from zimbraMailProxyPort Checking if process is listening on port 80 with name "nginx" user "zimbra" Making a backup of nginx templates in /opt/zimbra/conf/nginx/templates.20210925_102900 Patching nginx templates... Success. Nginx includes already patched, skipping zmproxy restart. Detecting certbot version... Detected certbot 0.31.0 Running /usr/bin/certbot certonly --webroot -w /opt/zimbra/data/nginx/html --cert-name mail.howtosurvey.gr -d mail.howtosurvey.gr -d mx.sodanet.gr Saving debug log to /var/log/letsencrypt/letsencrypt.log Plugins selected: Authenticator webroot, Installer None Cert is due for renewal, auto-renewing... Renewing an existing certificate Performing the following challenges: http-01 challenge for mail.howtosurvey.gr http-01 challenge for mx.sodanet.gr Using the webroot path /opt/zimbra/data/nginx/html for all unmatched domains. Waiting for verification... Cleaning up challenges

IMPORTANT NOTES:

• Congratulations! Your certificate and chain have been saved at: /etc/letsencrypt/live/mail.howtosurvey.gr/fullchain.pem Your key file has been saved at: /etc/letsencrypt/live/mail.howtosurvey.gr/privkey.pem Your cert will expire on 2021-12-24. To obtain a new or tweaked version of this certificate in the future, simply run certbot again. To non-interactively renew all of your certificates, run "certbot renew"

• If you like Certbot, please consider supporting our work by:

  Donating to ISRG / Let's Encrypt: https://letsencrypt.org/donate Donating to EFF: https://eff.org/donate-le

Preparing certificates for deployment. cat: /etc/ssl/certs/2e5ac55d.0: No such file or directory

An error seems to have occurred. Please read the output above for clues and try to rectify the situation. If you believe this is an error with the script, please file an issue at https://github.com/YetOpen/certbot-zimbra.

Please help me.

[jjakob]

Do you have ca-certificates installed and the latest version? Paste the output of "apt-cache policy ca-certificates"

On September 25, 2021 1:21:06 PM GMT+02:00, anastasios-salis @.***> wrote:

Certbot-zimbra, will not renew certificate and will give me the message: No such file or directory

below is the screen output.

@.***:~# certbot_zimbra.sh -n -H mail.howtosurvey.gr -e mx.sodanet.gr

certbot-zimbra v0.7.12 - https://github.com/YetOpen/certbot-zimbra Checking for dependencies... Detected Zimbra 8.8.15 on UBUNTU18_64 Using domain mail.howtosurvey.gr (as certificate DN) Got 1 domains to use as certificate SANs: mx.sodanet.gr Checking zimbra-proxy is running and enabled Detecting port from zimbraMailProxyPort Checking if process is listening on port 80 with name "nginx" user "zimbra" Making a backup of nginx templates in /opt/zimbra/conf/nginx/templates.20210925_102900 Patching nginx templates... Success. Nginx includes already patched, skipping zmproxy restart. Detecting certbot version... Detected certbot 0.31.0 Running /usr/bin/certbot certonly --webroot -w /opt/zimbra/data/nginx/html --cert-name mail.howtosurvey.gr -d mail.howtosurvey.gr -d mx.sodanet.gr Saving debug log to /var/log/letsencrypt/letsencrypt.log Plugins selected: Authenticator webroot, Installer None Cert is due for renewal, auto-renewing... Renewing an existing certificate Performing the following challenges: http-01 challenge for mail.howtosurvey.gr http-01 challenge for mx.sodanet.gr Using the webroot path /opt/zimbra/data/nginx/html for all unmatched domains. Waiting for verification... Cleaning up challenges

IMPORTANT NOTES:

• Congratulations! Your certificate and chain have been saved at: /etc/letsencrypt/live/mail.howtosurvey.gr/fullchain.pem Your key file has been saved at: /etc/letsencrypt/live/mail.howtosurvey.gr/privkey.pem Your cert will expire on 2021-12-24. To obtain a new or tweaked version of this certificate in the future, simply run certbot again. To non-interactively renew all of your certificates, run "certbot renew"

• If you like Certbot, please consider supporting our work by:

  Donating to ISRG / Let's Encrypt: https://letsencrypt.org/donate Donating to EFF: https://eff.org/donate-le

Preparing certificates for deployment. cat: /etc/ssl/certs/2e5ac55d.0: No such file or directory

An error seems to have occurred. Please read the output above for clues and try to rectify the situation. If you believe this is an error with the script, please file an issue at https://github.com/YetOpen/certbot-zimbra.

Please help me.

[aramilialon]

Hi! Glad to see that I'm not the one stupid but seems to be a real problem.... Same problem here, I have already checked the ca-certificate and is installed:

root@uhura:/etc/ssl/certs# apt-cache policy ca-certificates ca-certificates: Installed: 20210119~18.04.2 Candidate: 20210119~18.04.2 Version table: *** 20210119~18.04.2 500 500 http://de.archive.ubuntu.com/ubuntu bionic-updates/main amd64 Packages 500 http://de.archive.ubuntu.com/ubuntu bionic-updates/main i386 Packages 500 http://security.ubuntu.com/ubuntu bionic-security/main amd64 Packages 500 http://security.ubuntu.com/ubuntu bionic-security/main i386 Packages 100 /var/lib/dpkg/status 20180409 500 500 http://de.archive.ubuntu.com/ubuntu bionic/main amd64 Packages 500 http://de.archive.ubuntu.com/ubuntu bionic/main i386 Packages root@uhura:/etc/ssl/certs# update-ca-certificates -f Clearing symlinks in /etc/ssl/certs... done. Updating certificates in /etc/ssl/certs... 128 added, 0 removed; done. Running hooks in /etc/ca-certificates/update.d... done. root@uhura:/etc/ssl/certs# certbot_zimbra.sh -d -e SAN1.example.com SAN2.example.com certbot-zimbra v0.7.12 - https://github.com/YetOpen/certbot-zimbra Checking for dependencies... Detected Zimbra 8.8.15 on UBUNTU18_64 Using zmhostname to detect domain. Using domain mail.example.com (as certificate DN) Got 9 domains to use as certificate SANs: SAN1.example.com SAN2.example.com Preparing certificates for deployment. cat: /etc/ssl/certs/2e5ac55d.0: No such file or directory

An error seems to have occurred. Please read the output above for clues and try to rectify the situation. If you believe this is an error with the script, please file an issue at https://github.com/YetOpen/certbot-zimbra. root@uhura:/etc/ssl/certs

If I create a symbolic link named 2e5ac55d.0 to the ISRG_Root_X1.pem, instead....

Testing with zmcertmgr. ** Verifying '/run/certbot-zimbra/certs-2zFVpIGL/cert.pem' against '/run/certbot-zimbra/certs-2zFVpIGL/privkey.pem' Certificate '/run/certbot-zimbra/certs-2zFVpIGL/cert.pem' and private key '/run/certbot-zimbra/certs-2zFVpIGL/privkey.pem' match. ** Verifying '/run/certbot-zimbra/certs-2zFVpIGL/cert.pem' against '/run/certbot-zimbra/certs-2zFVpIGL/zimbra_chain.pem' ERROR: Unable to validate certificate chain: C = US, O = Internet Security Research Group, CN = ISRG Root X1 error 2 at 2 depth lookup: unable to get issuer certificate error /run/certbot-zimbra/certs-2zFVpIGL/cert.pem: verification failed

An error seems to have occurred. Please read the output above for clues and try to rectify the situation. If you believe this is an error with the script, please file an issue at https://github.com/YetOpen/certbot-zimbra.

[anastasios-salis]

Below is the output:

root@mail:~# apt-cache policy ca-certificates ca-certificates: Installed: 20210119~18.04.2 Candidate: 20210119~18.04.2 Version table: *** 20210119~18.04.2 500 500 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 Packages 500 http://archive.ubuntu.com/ubuntu bionic-security/main amd64 Packages 100 /var/lib/dpkg/status 20180409 500 500 http://archive.ubuntu.com/ubuntu bionic/main amd64 Packages root@mail:~#

[ways]

A workaround for the missing cert is listed in https://blog.zimbra.com/2021/09/zimbra-skilz-how-to-use-zimbra-with-lets-encrypt-certificates/ (but doesn't automatically solve the problem here).

[kribor]

A workaround for the missing cert is listed in https://blog.zimbra.com/2021/09/zimbra-skilz-how-to-use-zimbra-with-lets-encrypt-certificates/ (but doesn't automatically solve the problem here).

Thanks to this I was able to work around the issue and deploy the cert but I didn't find it obvious which steps I needed to follow so FYI this are the steps that (finally) worked for me.

• apt upgrade (upgrade ubuntu & zimbra packages, not sure if this makes any difference)
• Use latest certbot_zimbra (v0.7.12)
• Manually request new cert with x1 chain from letsencrypt: certbot --force-renewal --preferred-chain "ISRG Root X1" renew
• /usr/local/bin/certbot_zimbra.sh -d finally manages to deploy the cert!

[jjakob]

I think it's caused by Letsencrypt "DST Root CA X3" expiring. This file used to exist, but it has been removed after the root expired. If your cert still uses it (it shouldn't - see below) this error will be hit.

$ ls -la /etc/ssl/certs/2e5ac55d.0
lrwxrwxrwx 1 root root 18 Feb 24  2021 /etc/ssl/certs/2e5ac55d.0 -> DST_Root_CA_X3.pem
$ apt-cache policy ca-certificates
ca-certificates:
  Installed: 20210119~18.04.1
  Candidate: 20210119~18.04.1
$ apt-get update
$ apt-get install ca-certificates
$ apt-cache policy ca-certificates
ca-certificates:
  Installed: 20210119~18.04.2
  Candidate: 20210119~18.04.2
$ ls -la /etc/ssl/certs/2e5ac55d.0
ls: cannot access '/etc/ssl/certs/2e5ac55d.0': No such file or directory

There is a thread on this in the Letsencrypt forum: https://community.letsencrypt.org/t/help-thread-for-dst-root-ca-x3-expiration-september-2021/149190

Theoretically, if your system is up-to-date (has the X1 root in /etc/ssl) certbot should automatically issue a new cert with the X1 root when it automatically renews with no manual intervention. It's possible that if you somehow told certbot to not use the X1 root, perhaps some time ago by using the X3 as preferred-chain, that it won't automatically start using the X1 root and will keep using the X3 which has now expired. So a manual forced renew with the X1 chain should fix that issue. You'd need to also remove the X3 preferred chain option if you added it to the cronjob or in certbot_zinbra.sh. I'm not sure if it also gets saved into the certificate's preferences in /etc/letsencrypt, maybe it does and this is why you need to manually renew with the X1 chain.

The gist of it is, that it's not a certbot-zimbra issue, but a certbot issue.

[aramilialon]

The gist of it is, that it's not a certbot-zimbra issue, but a certbot issue.

Ok that the problem is in certbot-side, but looking for the parameters there is the -L that might be a good idea to be used with the --preferred-chain "ISRG Root X1" parameter. Does the -L option allow to use "inner parameters" when used to pass parameters?

[diypbx]

* certbot --force-renewal --preferred-chain "ISRG Root X1" renew

Is this the what I need to run to get the cert renewed as no expert here as I get this

Certbot can obtain and install HTTPS/TLS/SSL certificates. By default, it will attempt to use a webserver both for obtaining and installing the certificate. certbot: error: unrecognized arguments: --preferred-chain ISRG Root X1

Thanks

[jjakob]

As @kribor described, my recommendation would be:

if you get an error cat: /etc/ssl/certs/2e5ac55d.0: No such file or directory OR Can't find "DSTRootCAX3" OR Unable to validate certificate chain: O = Digital Signature Trust Co., CN = DST Root CA X3 you do the following:

• make sure you have latest ca-certificates (Debian/Ubuntu) or pki-base (RHEL/CentOS) package (do a apt-get dist-upgrade/upgrade/install ca-certificates or equivalent yum command)
• try forcing a renewal with certbot --force-renewal --preferred-chain "ISRG Root X1" renew (quotes around "ISRG Root X1" are important)
• if successful, run /usr/local/bin/certbot_zimbra.sh -d to deploy the new cert

It may be possible to use certbot_zimbra.sh -L '--preferred-chain "ISRG Root X1"' new to do the same, but you'll have to properly quote the parameter (single quotes should do, if that's not enough maybe try backslash escaping the double quotes inside). I've never tried to use it.

[diypbx]

Thanks

I am running Ubuntu 16.04 which is up to date.

if I run cerbot_zimbra.sh with the preferred chain I get

Certbot can obtain and install HTTPS/TLS/SSL certificates. By default, it will attempt to use a webserver both for obtaining and installing the certificate. certbot: error: unrecognized arguments: Root X1" Error: /usr/local/bin/certbot-auto exit status 2. Cannot proceed, exiting

If I try just using certbot with the preferred chain I get Certbot can obtain and install HTTPS/TLS/SSL certificates. By default, it will attempt to use a webserver both for obtaining and installing the certificate. certbot: error: unrecognized arguments: --preferred-chain ISRG Root X1

[jjakob]

With certbot_zimbra, try this:

certbot_zimbra.sh -L '--preferred-chain \"ISRG Root X1\"' new

With plain certbot, looks like you're missing the quotes, are you typing it exactly like this? --preferred-chain "ISRG Root X1"

If you omit the quotes, it'll think the parameter of --preferred-chain is only ISRG, and the rest (Root X1) are unrecognized options. Or are you using something that could be "eating" the quotes, e.g. passing the command through some other script?

[diypbx]

None of the commands worked for some reason and did exactly as typed

Ok so I can see the cert is actually in /etc/ssl/certs so have run dpkg-reconfigure ca-certificates and removed the old x3 cert from the list

Ran /usr/local/src/certbot-zimbra/certbot_zimbra.sh -p -H domainname Which seemed to run ok if I run the ssl checker I can see the cert is now due to expire on 2nd jan so that must have renewed ok

I now can't deploy it as if I run command to deploy it /usr/local/src/certbot-zimbra/certbot_zimbra.sh -d -H domainname

I get Preparing certificates for deployment. cat: /etc/ssl/certs/2e5ac55d.0: No such file or directory

[diypbx]

After running dpkg-reconfigure ca-certificates and deselecting the X3 cert I defiantly have the ISRG_Root_X1.pem in /etc/ssl

Ok so I have deleted all of the domain stuff in archive, live and renew in lets encrypt

I have run as from scratch ./certbot_zimbra.sh -n -c

I still get the same error Preparing certificates for deployment. cat: /etc/ssl/certs/2e5ac55d.0: No such file or directory

When I try anything with --preferred-chain it does not work and wondering if this is more to do with the version of certbot as it's 16.04

[kribor]

When I try anything with --preferred-chain it does not work and wondering if this is more to do with the version of certbot as it's 16.04

Most likely it's related to certbot being old as the preferred-chain option with X1 is a reasonably recent addition. Maybe you can install as a snap? That's what zimbra recommends: https://blog.zimbra.com/2021/09/zimbra-skilz-how-to-use-zimbra-with-lets-encrypt-certificates/. The error you're seeing is what happens when ca-certificates is updated (x3 cert removed) but the most recent cert from letsencrypt is issued using x3.

[jjakob]

@diypbx It's not a good idea to manually touch files in /etc/letsencrypt, you should always use certbot, or you risk making bigger problems. Looking at https://community.letsencrypt.org/t/preferred-chain-not-taking-effect/161568/11 they use that option exactly like that (maybe try with single quotes, but someone there used double quotes and it worked for them). Someone also mentioned you need certbot at least version 1.12. Did you use certbot-auto or the Ubuntu package? I know certbot-auto is deprecated and no longer updates, but I'm still using it on my server and it still renews fine. If you need to update certbot on 16.04, remove all packages and certbot-auto (search for the procedure), and install it using the python virtualenv method (even though they say it's not for production use, I've had success with it so far, making a venv in /opt/certbot_venv and putting a symlink in /usr/local/bin/certbot to /opt/venv/bin/certbot). If your python is too old, usually you can find a PPA with newer versions.

[diypbx]

I know what you mean about not deleting but is was starting to get to a last resort. It's becoming a big issue as I have 5 servers with load on people on them all. Checking my version of certbot it's certbot 0.27.0

What I don't understand is if the X3 no longer exists and the new ISRG_Root_X1 is in there why it's failing to deploy as should it not be using this now. Is there anyway to check.

Thanks for all your help

[kribor]

@diypbx you might try certbot --force-renewal renew, i.e. without the preferred-chain argument as I suppose cert issuing happens server-side and now that X3 is expired I would be surprised if it will send that back even to an old certbot client.

[diypbx]

@kribor Thanks I will give that a go. Already used my 5 up so will have to wait a bit.

[diypbx]

Thinking about this. If I have deleted the letsencrypt folder data in Live, archive and renew then this is a full new request so should be using the x1 to renew?

[jjakob]

I don't know how Letsencrypt/certbot work in such detail. The script uses certbot certonly with the right arguments (--webroot) to get a new cert, then that gets saved into a config file in /etc/letsencrypt by certbot. Perhaps it also saves the preferred chain (CA) so even if you delete the certs, it'll get the same one. If you delete the whole /etc/letsencrypt folder, you'll also lose your account info and letsencrypt may not like that (there are DOS limits, but you may be fine with a low request rate...) I'd definitely update certbot with the virtualenv method, maybe you need a newer one to use the newer CA. I don't know. If you have more problems try asking on the letsencrypt forum.

[anastasios-salis]

sudo snap install --classic certbot mv /usr/bin/certbot /usr/bin/certbot-old sudo ln -s /snap/bin/certbot /usr/bin/certbot certbot --force-renewal --preferred-chain "ISRG Root X1" renew

/usr/local/bin/certbot_zimbra.sh -d

[diypbx]

@anastasios-salis Thanks

I have just tried this and unfortunately as I am running LXC on proxmox this won't work either.

[jjakob]

Doesn't what you can do depend on the OS running inside the container? AFAIK containers on Proxmox are like VMs with persistent storage, not like Docker. I'd think you can run or install whatever you want inside?

On October 5, 2021 10:23:21 AM GMT+02:00, diypbx @.> wrote: @. Thanks

I have just tried this and unfortunately as I am running LXC on proxmox this won't work either.

[diypbx]

@jjakob If it was a standard vm then yes you usually can install what you like. The LXC uses the resources of the server a different way to give quicker backups and very quick boot times

snap install --classic certbot error: system does not fully support snapd: cannot mount squashfs image using "squashfs": mount: /tmp/sanity-mountpoint-821755971: mount failed: Unknown error -1

[Jasperki]

@anastasios-salis Thanks

I have just tried this and unfortunately as I am running LXC on proxmox this won't work either.

Hey @diypbx try the below:

Make backups of all the changed files (always have backup).

Replace <yourdomain> with your domain :)

Then:

cp /etc/letsencrypt/live/<yourdomain>/privkey.pem /opt/zimbra/ssl/zimbra/commercial/commercial.key
chown zimbra:zimbra /opt/zimbra/ssl/zimbra/commercial/commercial.key
wget -O /tmp/ISRG-X1.pem https://letsencrypt.org/certs/isrgrootx1.pem
wget -O /tmp/R3.pem https://letsencrypt.org/certs/lets-encrypt-r3.pem

The files in /etc/letsencrypt/live/<yourdomain> are symbolic links to files in /etc/letsencrypt/archive/<yourdomain>. Check which files they point at (cert<num>.pem, chain<num>.pem, ...)

Then perform this, but replace the <num> part with correct one.

cat /tmp/R3.pem > /etc/letsencrypt/archive/<yourdomain>/chain<num>.pem
cat /tmp/ISRG-X1.pem >> /etc/letsencrypt/archive/<yourdomain>/chain<num>.pem

As zimbra (su - zimbra) user perform

/opt/zimbra/bin/zmcertmgr verifycrt comm /opt/zimbra/ssl/zimbra/commercial/commercial.key /etc/letsencrypt/live/<yourdomain>/cert.pem /etc/letsencrypt/live/<yourdomain>/chain.pem

If it runs successfully perform deploy. If it gives you file permissions error temporarily do this:

chmod o+rx /etc/letsencrypt/archive
chmod o+rx /etc/letsencrypt/live

And verify the cert again.

You can run certbot-zimbra deploy, or the below one:

/opt/zimbra/bin/zmcertmgr deploycrt comm /etc/letsencrypt/live/<yourdomain>/cert.pem /etc/letsencrypt/live/<yordomain>/chain.pem

Afterwards remove the extra permissions (as root)

chmod o-rx /etc/letsencrypt/archive
chmod o-rx /etc/letsencrypt/live

Restart zimbra:

zmcontrol restart

Here you can see the source information for this (adapted by me): https://blog.zimbra.com/2021/09/zimbra-skilz-how-to-use-zimbra-with-lets-encrypt-certificates/

[jjakob]

The Python venv install method that I mentioned before doesn't need snap.

On October 5, 2021 10:35:23 AM GMT+02:00, diypbx @.> wrote: @. If it was a standard vm then yes you usually can install what you like. The LXC uses the resources of the server a different way to give quicker backups and very quick boot times

snap install --classic certbot error: system does not fully support snapd: cannot mount squashfs image using "squashfs": mount: /tmp/sanity-mountpoint-821755971: mount failed: Unknown error -1

[diypbx]

@Paweł

So if I used this would I use the script to renew as normal or would I have to jump through other hoops every few months.

@jjakob
Will look in to this option also

Thanks

[Jasperki]

@paweł

So if I used this would I use the script to renew as normal or would I have to jump through other hoops every few months.

@jjakob Will look in to this option also

Thanks

@diypbx as far as I can tell, this is just for now and in future all should work as it did before.

Unfortunately I can only confirm that at about mid of December.

[Jasperki]

@paweł So if I used this would I use the script to renew as normal or would I have to jump through other hoops every few months. @jjakob Will look in to this option also Thanks

@diypbx as far as I can tell, this is just for now and in future all should work as it did before.

Unfortunately I can only confirm that at about mid of December.

As far as I was able to determine, certbot renewed the certificates, but did not renew the chain.pem. And certbot-zimbra failed at cert verify, as it compared it for the issuer_hash of old Root CA.

After the chain.pem is fixed it all should work with cerbot-zimbra from now on.

The manual part:

1. Copy the cert
2. Download Root CA and Intermediate CA
3. Combine them into chain.pem
4. Verify the cert before deploymen Is just to fix your issue.

After the cert is verified (verification is not really mandatory, but is a good check before deployment) you can use certbot-zimbra deploy or use zimbra's deploy, up to you.

Like I said above, I will confirm in December.

[jjakob]

So the same certificate can be verified by two different CAs? That's cool, I didn't know that. (I think I remember something about "cross-signing") I'd check with ssltest anyway to see if the whole chain the server presents is good. If this works, it can be an adequate workaround for people who can't or don't know how to upgrade certbot on an old system. The issue being as you said, if perhaps the modified chain files get overwritten on renewal. You could try looking into the certbot config files to see if you can convince it into using the ISRG root, even if it doesn't have the preferred-lineage option. I don't know why some people run into the issue and some not, I definitely didn't, maybe I got issued the ISRG in the past already. Maybe it has to do with the certbot version.