certbot
Issue #8070 should not have been auto-closed
(Is this what I'm supposed to do? The github-actions bot said so.)
Issue https://github.com/certbot/certbot/issues/8070 was closed by github-actions despite still being present.
The function to change ownership is called unconditionally whenever a new private key is written: https://github.com/certbot/certbot/blob/985457e57bfde52d3702dea4e500a1b54ea1c745/certbot/certbot/_internal/storage.py#L1205-L1213
And the function that does the chown just calls chown expecting it to work: https://github.com/certbot/certbot/blob/985457e57bfde52d3702dea4e500a1b54ea1c745/certbot/certbot/compat/filesystem.py#L96-L130
The original issue describes the problem well, so I suggest you close this issue and reopen the original one.
@micha030201 do you have a use case that hits this issue in practice, or is this more a comment about general practices? Does @ xiruizhao's comment work as a workaround for your use case?
My problem is self-inflicted. I created (or, rather, amended the one that came with the debian package for certbot) a systemd service file where I run certbot with
User=certbot
Group=www-data
but then created certificates using
sudo -u certbot certbot certonly --webroot -w ... -d ...
tested that renewal works with
sudo -u certbot certbot renew --no-random-sleep-on-renew
and thought that everything was good. However, the initial certificate files were owned by certbot:certbot, and when run as a service certbot tried to chown the new files in the archive directory to that as well (from certbot:www-data that they got created as by default), and failing to do so did not update the live certificates.
I only found out that my certificates weren't automatically renewing after I received an email from letsencrypt, got really confused because the last message I saw in the logs was
acme.messages.Error: urn:ietf:params:acme:error:rateLimited :: There were too many requests of a given type :: too many certificates (5) already issued for this exact set of domains in the last 168h0m0s, retry after 2025-01-07 09:02:05 UTC: see https://letsencrypt.org/docs/rate-limits/#new-certificates-per-exact-set-of-hostnames
but after scrolling back to the
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/certbot/_internal/renewal.py", line 532, in handle_renewal_request
main.renew_cert(lineage_config, plugins, renewal_candidate)
File "/usr/lib/python3/dist-packages/certbot/_internal/main.py", line 1540, in renew_cert
renewed_lineage = _get_and_save_cert(le_client, config, lineage=lineage)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3/dist-packages/certbot/_internal/main.py", line 126, in _get_and_save_cert
renewal.renew_cert(config, domains, le_client, lineage)
File "/usr/lib/python3/dist-packages/certbot/_internal/renewal.py", line 401, in renew_cert
lineage.save_successor(prior_version, new_cert, new_key.pem, new_chain, config)
File "/usr/lib/python3/dist-packages/certbot/_internal/storage.py", line 1225, in save_successor
filesystem.copy_ownership_and_apply_mode(
File "/usr/lib/python3/dist-packages/certbot/compat/filesystem.py", line 127, in copy_ownership_and_apply_mode
os.chown(dst, user_id, group_id)
PermissionError: [Errno 1] Operation not permitted: '/etc/letsencrypt/archive/.../privkey15.pem'
and reading the code I figured it out and fixed it with
sudo chown -R certbot:www-data /etc/letsencrypt/live/
sudo chown -R certbot:www-data /etc/letsencrypt/archive/
and will run certbot as
sudo -u certbot -g www-data certbot ...
from now on.
If I understand correctly, certbot prefers to be run as root and installed through either pip (with renewal being done with cron apparently) or snap, neither of which I am eager to do. I understand that I am using certbot not in a way that is officially supported, and I am willing to face the consequences of my actions.
Nonetheless, I think this is a very non-obvious and difficult to debug failure mode, and given that a dev team member commented as such on the original issue, I thought I would let you know that the behaviour hasn't changed despite the issue having been closed by the bot. I might even submit a PR for it as @ bmw described if I have the time, but before that I kind of want to know whether that comment still stands, or if you changed your mind and don't consider this to be a problem anymore (which I would also understand).
Really appreciate the detailed write up! The comment still stands that even though we don't officially support running this way, we agree that it's not the best behavior for certbot to have. If not a behavior change PR, then a better error reporting situation could work. Since in addition only two people have mentioned hitting this though, I agree that while we'd accept a well-written PR to address it, we're unlikely to write a fix for it ourselves.
Three people. Also the folks in https://github.com/certbot/certbot/issues/1795 from way back when.
My solution was just to patch it out when building a container:
# certbot
python3 -m venv /opt/certbot/
/opt/certbot/bin/pip install --upgrade pip
/opt/certbot/bin/pip install certbot
# stupid hack, this chown breaks webroot certbot for non-root users...
sed -i -e 's/os.chown/#os.chown/' /opt/certbot/lib/python*/site-packages/certbot/compat/filesystem.py
