3 container setup + nginx replicas

[mmcinnestaylor]

trafficstars

Context

I use the 3 container setup (~18 months now) as the entrypoint to a collection of web services hosted in a single node Docker Swarm environment. I use helderco/docker-gen in place of nginx-proxy/docker-gen to support the dynamic container names used in Swarm mode.

The Problem

The 3 container setup works without issue when only one Nginx replica is used. Upon scaling up to two or more replicas, acme-companion begins generating these logs: Note: This is an example using 2 replicas, and the printed container id is 128 characters.

Reloading nginx (using separate container c19ed05645edfe3984c1aff6fa7946c06fe9170388997fe8ab78af68415f7c28
6607c1c9fa31fc6bbc7b2e161ca9d908afd1cda4123a140d7c2983190c0de7d8)...

Error: nginx-proxy container c19ed05645edfe3984c1aff6fa7946c06fe9170388997fe8ab78af68415f7c28
6607c1c9fa31fc6bbc7b2e161ca9d908afd1cda4123a140d7c2983190c0de7d8 isn't running.

It appears that the above error is generated in check_nginx_proxy_container_run in functions.sh. After doing some digging, I think I've isolated the problem to the following:

• All Nginx replicas use the same container label com.example.nginx_proxy
• On any call to get_nginx_proxy_container in functions.sh the function returns a single id string of length 64 * n_replicas
  • Docker full container IDs are 64 characters
• Both check_nginx_proxy_container_run and reload_nginx in functions.sh use the container id returned from get_nginx_proxy_container but do not account for the possibility that the string contains multiple ids.

The Fix?

I am not intimately familiar with the codebase, but it appears that check_nginx_proxy_container_run and reload_nginx simply need to parse the concatenated string and loop over each id while performing the state check or calling docker_kill.

EDIT: added Stack YAML

version: "3.8"
services:
    nginx:
        image: nginx:mainline
        labels:
            - com.github.jrcs.letsencrypt_nginx_proxy_companion.nginx_proxy=true
        ports:
            - 80:80
            - 443:443
        environment:
            TZ: America/New_York
        volumes:
            - certs:/etc/nginx/certs:ro
            - conf:/etc/nginx/conf.d
            - html:/usr/share/nginx/html
            - vhost:/etc/nginx/vhost.d
            - /var/log/nginx:/var/log/nginx:rw
        networks:
            - acme
            - backend
        deploy:
            replicas: 2
            restart_policy:
                delay: 5s
                max_attempts: 10
                window: 60s
            rollback_config:
                delay: 5s
                monitor: 60s
            update_config:
                parallelism: 1
                delay: 5s
                monitor: 60s
                failure_action: rollback 
    docker-gen:
        image: helder/docker-gen:latest
        labels:
            - com.github.jrcs.letsencrypt_nginx_proxy_companion.docker_gen=true
        command: -notify "docker-label-sighup com.github.jrcs.letsencrypt_nginx_proxy_companion.nginx_proxy" -watch -wait 10s:30s /etc/docker-gen/templates/nginx.tmpl /etc/nginx/conf.d/default.conf
        environment:
            RESOLVERS: 1.1.1.1
        volumes:
            - certs:/etc/nginx/certs:ro
            - conf:/etc/nginx/conf.d
            - html:/usr/share/nginx/html
            - vhost:/etc/nginx/vhost.d
            - ./config/nginx.tmpl:/etc/docker-gen/templates/nginx.tmpl:ro
            - /var/run/docker.sock:/tmp/docker.sock:ro
        networks:
            - acme
            - backend
        deploy:
            restart_policy:
                delay: 5s
                max_attempts: 10
                window: 60s
            rollback_config:
                monitor: 60s
            update_config:
                monitor: 60s
                failure_action: rollback
    acme-companion:
        image: nginxproxy/acme-companion:latest
        environment:
            DEFAULT_EMAIL: <<redacted>>
            CERTS_UPDATE_INTERVAL: 86400 # 86400s = 1 day
        volumes:
            - acme:/etc/acme.sh
            - certs:/etc/nginx/certs:rw
            - conf:/etc/nginx/conf.d
            - html:/usr/share/nginx/html
            - vhost:/etc/nginx/vhost.d
            - /var/run/docker.sock:/var/run/docker.sock:ro
        networks:
            - acme
        deploy:
            restart_policy:
                delay: 5s
                max_attempts: 10
                window: 60s
            rollback_config:
                monitor: 60s
            update_config:
                monitor: 60s
                failure_action: rollback
volumes:
    acme:
    certs:
    conf:
    html:
    vhost:
networks:
    acme:
        name: acme
    backend:
        name: proxied-apps