CURL error 28 on certificate creation/renewal

[LilianBoulard]

Bug description

The companion has been working fine for the last 6 months I've been using it, but I setup a new service last week, and the SSL certificate was not created for this one. Digging into the issue, it seems the key and crt files are not created, similar to #1042, and when running docker exec acme-companion /app/force_renew, the creation takes forever (kind of like #1049), and returns this error:

$ docker exec reverse_proxy_acme /app/force_renew
Creating/renewal service.example.com certificates... (service.example.com)
[Mon Aug 21 22:00:45 UTC 2023] Please refer to https://curl.haxx.se/libcurl/c/libcurl-errors.html for error code: 28
[Mon Aug 21 22:00:45 UTC 2023] Can not init api for: https://acme-v02.api.letsencrypt.org/directory.

for all services, new and old.

acme-companion image version

Info: running acme-companion version v2.2.8-14-g9fbd668

nginx-proxy's Docker configuration

Service docker-compose.yml

version: "3"

networks:

  db_bridge:
    internal: true

  reverse_proxy_bridge:
    external: true

services:

  postgres:
    image: postgres:latest
    container_name: postgres
    restart: unless-stopped
    expose:
      - 5432
    environment:
      - POSTGRES_USER=abc
      - POSTGRES_PASSWORD=abc
      - POSTGRES_DB=abc
    networks:
      - db_bridge

  service:
    build:
      context: ./service
    container_name: service
    depends_on:
      - postgres
    restart: unless-stopped
    expose:
      - 80
    environment:
      - VIRTUAL_HOST=service.example.com
      - LETSENCRYPT_HOST=service.example.com
      - VIRTUAL_PORT=80
      - NETWORK_ACCESS=internal
    networks:
      - db_bridge
      - reverse_proxy_bridge

Reverse proxy docker-compose.yml:

version: "3.7"

networks:

  acme_bridge:
    internal: true

  reverse_proxy_bridge:
    external: true

services:

  reverse-proxy:
    image: nginxproxy/nginx-proxy:alpine
    container_name: reverse_proxy
    restart: unless-stopped
    # We make the reverse-proxy depend on the acme companion
    # as we would prefer serving no service at all than
    # unsecure ones.
    depends_on:
      - reverse-proxy-acme
    ports:
      - 80:80
      - 443:443
    environment:
      - TRUST_DOWNSTREAM_PROXY=false
    volumes:
      - ./htpasswd:/etc/nginx/htpasswd:ro
      - ./certs:/etc/nginx/certs:ro
      - ./vhost:/etc/nginx/vhost.d
      - ./html:/usr/share/nginx/html
      - /var/run/docker.sock:/tmp/docker.sock:ro
      - ./log:/var/log
      - ./network_internal.conf:/etc/nginx/network_internal.conf:ro
    networks:
      - acme_bridge
      - reverse_proxy_bridge

  reverse-proxy-acme:
    image: nginxproxy/acme-companion:latest
    container_name: reverse_proxy_acme
    restart: unless-stopped
    volumes:
      - ./certs:/etc/nginx/certs
      - ./vhost:/etc/nginx/vhost.d
      - ./html:/usr/share/nginx/html
      - ./acme:/etc/acme.sh
      - /var/run/docker.sock:/var/run/docker.sock:ro
    environment:
      - NGINX_PROXY_CONTAINER=reverse_proxy
    networks:
      - acme_bridge

rendered nginx configuration

Please provide the rendered nginx configuration:

nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is successful
# configuration file /etc/nginx/nginx.conf:

user  nginx;
worker_processes  auto;

error_log  /var/log/nginx/error.log notice;
pid        /var/run/nginx.pid;


events {
    worker_connections  10240;
}


http {
    include       /etc/nginx/mime.types;
    default_type  application/octet-stream;

    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"';

    access_log  /var/log/nginx/access.log  main;

    sendfile        on;
    #tcp_nopush     on;

    keepalive_timeout  65;

    #gzip  on;

    include /etc/nginx/conf.d/*.conf;
}
daemon off;

# configuration file /etc/nginx/mime.types:

types {
    text/html                                        html htm shtml;
    text/css                                         css;
    text/xml                                         xml;
    image/gif                                        gif;
    image/jpeg                                       jpeg jpg;
    application/javascript                           js;
    application/atom+xml                             atom;
    application/rss+xml                              rss;

    text/mathml                                      mml;
    text/plain                                       txt;
    text/vnd.sun.j2me.app-descriptor                 jad;
    text/vnd.wap.wml                                 wml;
    text/x-component                                 htc;

    image/avif                                       avif;
    image/png                                        png;
    image/svg+xml                                    svg svgz;
    image/tiff                                       tif tiff;
    image/vnd.wap.wbmp                               wbmp;
    image/webp                                       webp;
    image/x-icon                                     ico;
    image/x-jng                                      jng;
    image/x-ms-bmp                                   bmp;

    font/woff                                        woff;
    font/woff2                                       woff2;

    application/java-archive                         jar war ear;
    application/json                                 json;
    application/mac-binhex40                         hqx;
    application/msword                               doc;
    application/pdf                                  pdf;
    application/postscript                           ps eps ai;
    application/rtf                                  rtf;
    application/vnd.apple.mpegurl                    m3u8;
    application/vnd.google-earth.kml+xml             kml;
    application/vnd.google-earth.kmz                 kmz;
    application/vnd.ms-excel                         xls;
    application/vnd.ms-fontobject                    eot;
    application/vnd.ms-powerpoint                    ppt;
    application/vnd.oasis.opendocument.graphics      odg;
    application/vnd.oasis.opendocument.presentation  odp;
    application/vnd.oasis.opendocument.spreadsheet   ods;
    application/vnd.oasis.opendocument.text          odt;
    application/vnd.openxmlformats-officedocument.presentationml.presentation
                                                     pptx;
    application/vnd.openxmlformats-officedocument.spreadsheetml.sheet
                                                     xlsx;
    application/vnd.openxmlformats-officedocument.wordprocessingml.document
                                                     docx;
    application/vnd.wap.wmlc                         wmlc;
    application/wasm                                 wasm;
    application/x-7z-compressed                      7z;
    application/x-cocoa                              cco;
    application/x-java-archive-diff                  jardiff;
    application/x-java-jnlp-file                     jnlp;
    application/x-makeself                           run;
    application/x-perl                               pl pm;
    application/x-pilot                              prc pdb;
    application/x-rar-compressed                     rar;
    application/x-redhat-package-manager             rpm;
    application/x-sea                                sea;
    application/x-shockwave-flash                    swf;
    application/x-stuffit                            sit;
    application/x-tcl                                tcl tk;
    application/x-x509-ca-cert                       der pem crt;
    application/x-xpinstall                          xpi;
    application/xhtml+xml                            xhtml;
    application/xspf+xml                             xspf;
    application/zip                                  zip;

    application/octet-stream                         bin exe dll;
    application/octet-stream                         deb;
    application/octet-stream                         dmg;
    application/octet-stream                         iso img;
    application/octet-stream                         msi msp msm;

    audio/midi                                       mid midi kar;
    audio/mpeg                                       mp3;
    audio/ogg                                        ogg;
    audio/x-m4a                                      m4a;
    audio/x-realaudio                                ra;

    video/3gpp                                       3gpp 3gp;
    video/mp2t                                       ts;
    video/mp4                                        mp4;
    video/mpeg                                       mpeg mpg;
    video/quicktime                                  mov;
    video/webm                                       webm;
    video/x-flv                                      flv;
    video/x-m4v                                      m4v;
    video/x-mng                                      mng;
    video/x-ms-asf                                   asx asf;
    video/x-ms-wmv                                   wmv;
    video/x-msvideo                                  avi;
}

# configuration file /etc/nginx/conf.d/default.conf:
# nginx-proxy version : 1.3.1-36-g6f0be58
# Networks available to the container running docker-gen (which are assumed to
# match the networks available to the container running nginx):
#     reverse-proxy_acme_bridge
#     reverse_proxy_bridge
# If we receive X-Forwarded-Proto, pass it through; otherwise, pass along the
# scheme used to connect to this server
map $http_x_forwarded_proto $proxy_x_forwarded_proto {
    default $scheme;
    '' $scheme;
}
map $http_x_forwarded_host $proxy_x_forwarded_host {
    default $http_host;
    '' $http_host;
}
# If we receive X-Forwarded-Port, pass it through; otherwise, pass along the
# server port the client connected to
map $http_x_forwarded_port $proxy_x_forwarded_port {
    default $server_port;
    '' $server_port;
}
# If the request from the downstream client has an "Upgrade:" header (set to any
# non-empty value), pass "Connection: upgrade" to the upstream (backend) server.
# Otherwise, the value for the "Connection" header depends on whether the user
# has enabled keepalive to the upstream server.
map $http_upgrade $proxy_connection {
    default upgrade;
    '' $proxy_connection_noupgrade;
}
map $upstream_keepalive $proxy_connection_noupgrade {
    # Preserve nginx's default behavior (send "Connection: close").
    default close;
    # Use an empty string to cancel nginx's default behavior.
    true '';
}
# Abuse the map directive (see <https://stackoverflow.com/q/14433309>) to ensure
# that $upstream_keepalive is always defined.  This is necessary because:
#   - The $proxy_connection variable is indirectly derived from
#     $upstream_keepalive, so $upstream_keepalive must be defined whenever
#     $proxy_connection is resolved.
#   - The $proxy_connection variable is used in a proxy_set_header directive in
#     the http block, so it is always fully resolved for every request -- even
#     those where proxy_pass is not used (e.g., unknown virtual host).
map "" $upstream_keepalive {
    # The value here should not matter because it should always be overridden in
    # a location block (see the "location" template) for all requests where the
    # value actually matters.
    default false;
}
# Apply fix for very long server names
server_names_hash_bucket_size 128;
# Default dhparam
ssl_dhparam /etc/nginx/dhparam/dhparam.pem;
# Set appropriate X-Forwarded-Ssl header based on $proxy_x_forwarded_proto
map $proxy_x_forwarded_proto $proxy_x_forwarded_ssl {
    default off;
    https on;
}
gzip_types text/plain text/css application/javascript application/json application/x-javascript text/xml application/xml application/xml+rss text/javascript;
log_format vhost '$host $remote_addr - $remote_user [$time_local] "$request" $status $body_bytes_sent "$http_referer" "$http_user_agent" "$upstream_addr"';
access_log off;
    ssl_protocols TLSv1.2 TLSv1.3;
    ssl_ciphers 'ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384:DHE-RSA-CHACHA20-POLY1305';
    ssl_prefer_server_ciphers off;
error_log /dev/stderr;
resolver 127.0.0.11;
# HTTP 1.1 support
proxy_http_version 1.1;
proxy_buffering off;
proxy_set_header Host $http_host;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection $proxy_connection;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Host $proxy_x_forwarded_host;
proxy_set_header X-Forwarded-Proto $proxy_x_forwarded_proto;
proxy_set_header X-Forwarded-Ssl $proxy_x_forwarded_ssl;
proxy_set_header X-Forwarded-Port $proxy_x_forwarded_port;
proxy_set_header X-Original-URI $request_uri;
# Mitigate httpoxy attack (see README for details)
proxy_set_header Proxy "";
server {
    server_name _; # This is just an invalid value which will never trigger on a real hostname.
    server_tokens off;
    http2 on;
    listen 80;
    listen 443 ssl;
    ssl_session_cache shared:SSL:50m;
    ssl_session_tickets off;
    access_log /var/log/nginx/access.log vhost;
    ssl_certificate /etc/nginx/certs/default.crt;
    ssl_certificate_key /etc/nginx/certs/default.key;
    return 503;
}
# service.example.com/
upstream service.example.com {
    # Container: service
    #     networks:
    #         service_db_bridge (unreachable)
    #         reverse_proxy_bridge (reachable)
    #     IP address: 172.29.0.4
    #     exposed ports: 80/tcp
    #     default port: 80
    #     using port: 80
    server 172.29.0.4:80;
}
server {
    server_name service.example.com;
    http2 on;
    access_log /var/log/nginx/access.log vhost;
    listen 80 ;
    listen 443 ssl ;
    # No certificate found for this vhost, so use the default certificate and
    # return an error code if the user connects via https.
    ssl_certificate /etc/nginx/certs/default.crt;
    ssl_certificate_key /etc/nginx/certs/default.key;
    if ($https) {
        return 500;
    }
    include /etc/nginx/vhost.d/default;
    location / {
        proxy_pass http://service.example.com;
        set $upstream_keepalive false;
    }
}

# configuration file /etc/nginx/vhost.d/default:
## Start of configuration add by letsencrypt container
location ^~ /.well-known/acme-challenge/ {
    auth_basic off;
    auth_request off;
    allow all;
    root /usr/share/nginx/html;
    try_files $uri =404;
    break;
}
## End of configuration add by letsencrypt container

# configuration file /etc/nginx/network_internal.conf:
# Only allow traffic from internal clients
allow 127.0.0.0/8;
allow 10.0.0.0/8;
allow 192.168.0.0/16;
allow 192.168.27.64/27;
allow 192.168.27.0/27;
allow 172.16.0.0/12;  # FIXME: check/add ipv6 nat support
allow fc00::/7; # IPv6 local address range
deny all;

Containers logs

$ docker logs reverse_proxy_acme
Info: running acme-companion version v2.2.8-14-g9fbd668
Info: 4096 bits RFC7919 Diffie-Hellman group found, generation skipped.
Reloading nginx proxy (reverse_proxy)...
2023/08/21 21:58:33 Generated '/etc/nginx/conf.d/default.conf' from 22 containers
2023/08/21 21:58:33 Generated '/app/letsencrypt_service_data' from 22 containers
2023/08/21 21:58:33 Running '/app/signal_le_service'
2023/08/21 21:58:33 Watching docker events
2023/08/21 21:58:33 Contents of /app/letsencrypt_service_data did not change. Skipping notification '/app/signal_le_service'
Reloading nginx proxy (reverse_proxy)...
2023/08/21 21:58:33 Generated '/etc/nginx/conf.d/default.conf' from 22 containers
Creating/renewal service.example.com certificates... (service.example.com)
[Mon Aug 21 22:00:43 UTC 2023] Please refer to https://curl.haxx.se/libcurl/c/libcurl-errors.html for error code: 28
[Mon Aug 21 22:00:43 UTC 2023] Can not init api for: https://acme-v02.api.letsencrypt.org/directory.

$ docker logs reverse_proxy
Info: running nginx-proxy version 1.3.1-36-g6f0be58
Setting up DH Parameters..
forego      | starting dockergen.1 on port 5000
forego      | starting nginx.1 on port 5100
dockergen.1 | 2023/08/21 22:09:03 Generated '/etc/nginx/conf.d/default.conf' from 22 containers
dockergen.1 | 2023/08/21 22:09:03 Running 'nginx -s reload'
dockergen.1 | 2023/08/21 22:09:03 Watching docker events
dockergen.1 | 2023/08/21 22:09:03 Generated '/etc/nginx/conf.d/default.conf' from 22 containers
dockergen.1 | 2023/08/21 22:09:03 Running 'nginx -s reload'

Please provide the logs of:

• your acme-companion container
• your nginx-proxy container (or nginx and docker-gen container in a three containers setup)

docker logs name-of-the-companion-container

Docker host

• OS: Debian 11
• Docker version:

 Version:           24.0.5
 API version:       1.43
 Go version:        go1.20.6
 Git commit:        ced0996
 Built:             Fri Jul 21 20:35:45 2023
 OS/Arch:           linux/amd64
 Context:           default

Server: Docker Engine - Community
 Engine:
  Version:          24.0.5
  API version:      1.43 (minimum version 1.12)
  Go version:       go1.20.6
  Git commit:       a61e2b4
  Built:            Fri Jul 21 20:35:45 2023
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.6.22
  GitCommit:        8165feabfdfe38c65b599c4993d227328c231fca
 runc:
  Version:          1.1.8
  GitCommit:        v1.1.8-0-g82f18fe
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

[Steiniche]

I am seeing something very similar with 2.2.9 When the Acme Companion starts it correctly outputs its version number and then nothing. Running with DEBUG=1 yield no more logs.

Trying to execute docker exec acme-companion /app/force_renew the command just hangs indefinitely. As it seems 2.2.9 has updated many dependencies based on depandabot recommendations maybe this is the culprit?

I tried introducing a fault where the Acme Companion did not know of the nginx-proxy container by not:

1. Providing the volumes-from directive
2. Providing the label
3. Providing the env var

This still did not give any error messages.

I have reverted back to 2.2.8 for now as this has fixed the problem. Further, 2.2.8 will provide error messages for what to do when in a faulty state as described above.

Happy to help test what ever is necessary to find the problem.

[buchdag]

@Steiniche : https://github.com/nginx-proxy/acme-companion/issues/1084#issuecomment-1875764063

There is a known issue with the version of curl currently shipped with Alpine 3.19 and 3.x kernels, causing an endless loop to hang containers, usually on startup. The 3.x kernel line has been End of Life since 2017, but some prebuilt systems such as older Synology NAS units are still running it.

While a fix has been applied to the affected upstream library we do not have an ETA for if or when it will find its way into the version of curl shipped with Alpine 3.19.