acme-companion
acme-companion copied to clipboard
nginx-proxy
Fullchain.pem expired / force renewal doesn't renew
So, just recently i've been getting curl certificate expired errors on my ubuntu 14.04 server, however it works just fine on any other server, including my local machine
curl: (60) SSL certificate problem: certificate has expired
A bit of investigating, I believe that the Intermediate has expired or is invalid, whynopadlock result shows that, along with /app/cert_status command showing that the fullchain.pem has expired, force renewing doesn't seem to fix the issue, it 100% renews the certificate however.
##### Certificate status #####
/etc/nginx/certs/domain/fullchain.pem: EXPIRED
Certificate was issued by R3
Certificate was valid until Jan 9 05:04:47 2022 GMT
Subject Name:
- domain
Certificate is used by the following domain(s):
- domain
Info: running acme-companion version v2.1.0-28-g1785bc5
version: '2'
services:
nginx-proxy:
image: nginxproxy/nginx-proxy:alpine
container_name: nginx-proxy
ports:
- "80:80"
- "443:443"
volumes:
- conf:/etc/nginx/conf.d
- vhost:/etc/nginx/vhost.d
- html:/usr/share/nginx/html
- dhparam:/etc/nginx/dhparam
- certs:/etc/nginx/certs:ro
- /var/run/docker.sock:/tmp/docker.sock:ro
nginx-proxy-letsencrypt:
image: nginxproxy/acme-companion
container_name: nginx-proxy-letsencrypt
volumes_from:
- nginx-proxy
volumes:
- certs:/etc/nginx/certs:rw
- acme:/etc/acme.sh
- /var/run/docker.sock:/var/run/docker.sock:ro
environment:
- DEFAULT_EMAIL=
whoami:
image: jwilder/whoami
environment:
- VIRTUAL_HOST=whoami.local
volumes:
conf:
vhost:
html:
dhparam:
certs:
acme:
rendered nginx configuration
nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is successful
# configuration file /etc/nginx/nginx.conf:
user nginx;
worker_processes auto;
error_log /var/log/nginx/error.log notice;
pid /var/run/nginx.pid;
events {
worker_connections 10240;
}
http {
include /etc/nginx/mime.types;
default_type application/octet-stream;
log_format main '$remote_addr - $remote_user [$time_local] "$request" '
'$status $body_bytes_sent "$http_referer" '
'"$http_user_agent" "$http_x_forwarded_for"';
access_log /var/log/nginx/access.log main;
sendfile on;
#tcp_nopush on;
keepalive_timeout 65;
#gzip on;
include /etc/nginx/conf.d/*.conf;
}
daemon off;
# configuration file /etc/nginx/mime.types:
types {
text/html html htm shtml;
text/css css;
text/xml xml;
image/gif gif;
image/jpeg jpeg jpg;
application/javascript js;
application/atom+xml atom;
application/rss+xml rss;
text/mathml mml;
text/plain txt;
text/vnd.sun.j2me.app-descriptor jad;
text/vnd.wap.wml wml;
text/x-component htc;
image/png png;
image/svg+xml svg svgz;
image/tiff tif tiff;
image/vnd.wap.wbmp wbmp;
image/webp webp;
image/x-icon ico;
image/x-jng jng;
image/x-ms-bmp bmp;
font/woff woff;
font/woff2 woff2;
application/java-archive jar war ear;
application/json json;
application/mac-binhex40 hqx;
application/msword doc;
application/pdf pdf;
application/postscript ps eps ai;
application/rtf rtf;
application/vnd.apple.mpegurl m3u8;
application/vnd.google-earth.kml+xml kml;
application/vnd.google-earth.kmz kmz;
application/vnd.ms-excel xls;
application/vnd.ms-fontobject eot;
application/vnd.ms-powerpoint ppt;
application/vnd.oasis.opendocument.graphics odg;
application/vnd.oasis.opendocument.presentation odp;
application/vnd.oasis.opendocument.spreadsheet ods;
application/vnd.oasis.opendocument.text odt;
application/vnd.openxmlformats-officedocument.presentationml.presentation
pptx;
application/vnd.openxmlformats-officedocument.spreadsheetml.sheet
xlsx;
application/vnd.openxmlformats-officedocument.wordprocessingml.document
docx;
application/vnd.wap.wmlc wmlc;
application/wasm wasm;
application/x-7z-compressed 7z;
application/x-cocoa cco;
application/x-java-archive-diff jardiff;
application/x-java-jnlp-file jnlp;
application/x-makeself run;
application/x-perl pl pm;
application/x-pilot prc pdb;
application/x-rar-compressed rar;
application/x-redhat-package-manager rpm;
application/x-sea sea;
application/x-shockwave-flash swf;
application/x-stuffit sit;
application/x-tcl tcl tk;
application/x-x509-ca-cert der pem crt;
application/x-xpinstall xpi;
application/xhtml+xml xhtml;
application/xspf+xml xspf;
application/zip zip;
application/octet-stream bin exe dll;
application/octet-stream deb;
application/octet-stream dmg;
application/octet-stream iso img;
application/octet-stream msi msp msm;
audio/midi mid midi kar;
audio/mpeg mp3;
audio/ogg ogg;
audio/x-m4a m4a;
audio/x-realaudio ra;
video/3gpp 3gpp 3gp;
video/mp2t ts;
video/mp4 mp4;
video/mpeg mpeg mpg;
video/quicktime mov;
video/webm webm;
video/x-flv flv;
video/x-m4v m4v;
video/x-mng mng;
video/x-ms-asf asx asf;
video/x-ms-wmv wmv;
video/x-msvideo avi;
}
# configuration file /etc/nginx/conf.d/default.conf:
# If we receive X-Forwarded-Proto, pass it through; otherwise, pass along the
# scheme used to connect to this server
map $http_x_forwarded_proto $proxy_x_forwarded_proto {
default $http_x_forwarded_proto;
'' $scheme;
}
# If we receive X-Forwarded-Port, pass it through; otherwise, pass along the
# server port the client connected to
map $http_x_forwarded_port $proxy_x_forwarded_port {
default $http_x_forwarded_port;
'' $server_port;
}
# If we receive Upgrade, set Connection to "upgrade"; otherwise, delete any
# Connection header that may have been passed to this server
map $http_upgrade $proxy_connection {
default upgrade;
'' close;
}
# Apply fix for very long server names
server_names_hash_bucket_size 128;
# Default dhparam
ssl_dhparam /etc/nginx/dhparam/dhparam.pem;
# Set appropriate X-Forwarded-Ssl header based on $proxy_x_forwarded_proto
map $proxy_x_forwarded_proto $proxy_x_forwarded_ssl {
default off;
https on;
}
gzip_types text/plain text/css application/javascript application/json application/x-javascript text/xml application/xml application/xml+rss text/javascript;
log_format vhost '$host $remote_addr - $remote_user [$time_local] '
'"$request" $status $body_bytes_sent '
'"$http_referer" "$http_user_agent" '
'"$upstream_addr"';
access_log off;
ssl_protocols TLSv1.2 TLSv1.3;
ssl_ciphers 'ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384';
ssl_prefer_server_ciphers off;
resolver 127.0.0.11;
# HTTP 1.1 support
proxy_http_version 1.1;
proxy_buffering off;
proxy_set_header Host $http_host;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection $proxy_connection;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $proxy_x_forwarded_proto;
proxy_set_header X-Forwarded-Ssl $proxy_x_forwarded_ssl;
proxy_set_header X-Forwarded-Port $proxy_x_forwarded_port;
# Mitigate httpoxy attack (see README for details)
proxy_set_header Proxy "";
server {
server_name _; # This is just an invalid value which will never trigger on a real hostname.
server_tokens off;
listen 80;
access_log /var/log/nginx/access.log vhost;
return 503;
}
server {
server_name _; # This is just an invalid value which will never trigger on a real hostname.
server_tokens off;
listen 443 ssl http2;
access_log /var/log/nginx/access.log vhost;
return 503;
ssl_session_cache shared:SSL:50m;
ssl_session_tickets off;
ssl_certificate /etc/nginx/certs/default.crt;
ssl_certificate_key /etc/nginx/certs/default.key;
}
# domain
upstream domain {
## Can be connected with "nginx-proxy_default" network
# domain_nginx_1
server 172.19.0.5:80;
}
server {
server_name domain;
listen 80 ;
access_log /var/log/nginx/access.log vhost;
# Do not HTTPS redirect Let'sEncrypt ACME challenge
location ^~ /.well-known/acme-challenge/ {
auth_basic off;
auth_request off;
allow all;
root /usr/share/nginx/html;
try_files $uri =404;
break;
}
location / {
return 301 https://$host$request_uri;
}
}
server {
server_name domain;
listen 443 ssl http2 ;
access_log /var/log/nginx/access.log vhost;
ssl_session_timeout 5m;
ssl_session_cache shared:SSL:50m;
ssl_session_tickets off;
ssl_certificate /etc/nginx/certs/domain.crt;
ssl_certificate_key /etc/nginx/certs/domain.key;
ssl_dhparam /etc/nginx/certs/domain.dhparam.pem;
ssl_stapling on;
ssl_stapling_verify on;
ssl_trusted_certificate /etc/nginx/certs/domain.chain.pem;
add_header Strict-Transport-Security "max-age=31536000" always;
include /etc/nginx/vhost.d/default;
location / {
proxy_pass http://domain;
}
}
# whoami.local
upstream whoami.local {
## Can be connected with "nginx-proxy_default" network
# nginx-proxy_whoami_1
server 172.19.0.2:8000;
}
server {
server_name whoami.local;
listen 80 ;
access_log /var/log/nginx/access.log vhost;
include /etc/nginx/vhost.d/default;
location / {
proxy_pass http://whoami.local;
}
}
server {
server_name whoami.local;
listen 443 ssl http2 ;
access_log /var/log/nginx/access.log vhost;
return 500;
ssl_certificate /etc/nginx/certs/default.crt;
ssl_certificate_key /etc/nginx/certs/default.key;
}
# configuration file /etc/nginx/vhost.d/default:
## Start of configuration add by letsencrypt container
location ^~ /.well-known/acme-challenge/ {
auth_basic off;
auth_request off;
allow all;
root /usr/share/nginx/html;
try_files $uri =404;
break;
}
## End of configuration add by letsencrypt container
I have the same issue and still could not figure out if it is related to this project or not. docker exec letsencrypt /app/cert_status shows all certificates as expired however force renewal works all certificates are getting renewed. I use docker so the server being old should not be an issue.
@gurumark @fluffyst nginxproxy/acme-companion:2.1.1 has been updated to use a newer version of acme.sh (2.8.8 -> 2.9.0), could you test again with this version of the container ?
Didn't fix it for me at least. I've got the same phenomenon, that cert_status shows all fullchains as expired (Certificate was valid until Mar 5 2022), but I don't have any other issues besides that. The certificates and the renewal seems to work as usual.
It doesn't seem to make a difference if I use ACME_PREFERRED_CHAIN on a container or not. I'm on a fully updated Ubuntu 20.04 machine with the latest images.
Did someone find the fix for this? I'm facing the same issue with the fullchain.pem file
The only way to show the fullchain as OK instead of expired is to manually edit the fullchain.pem and remove the last cert in the chain.
Just to clarify: the cert_status function is a small utility that was hastily written a while ago, it's not meant to actually reflect the internal state of acme.sh in any way.
I know the preferred chain stuff has been an issue on the acme.sh side for a while, the more recent version 3.x might finally solve this but I'll have to check a few things before bumping to this version. It's fairly easy to locally build an acme-companion image with a different version of acme.sh, I can provide instructions for this if needed.
Since one year I'm running a mailcow-dockerized behind nginx-proxy. Today I'm facing exactly this issue myself from one day to the other. In my case IMAPS does not work anymore. openssl connection to the server returns: certificate has expired force renewal works but cert_status on my mailserver says that fullchain.pem has exired and: "Certificate was valid until Jun 22 06:06:14 2022 GMT" really strange
Hi, isn't this caused by the DST Root CA X3 expiration ?
I think that is so. I've created an issue on the Let's Encrypt forum. Follow the link to find the suggested solution.
I tried to find a solution myself, and the following is probably not needed anymore, but just in case, to extract the first certificate from a file:
$ awk '/BEGIN/{if (matched) exit; else matched=1}; {print}' < /etc/nginx/certs/example.com/chain.pem | less
