Clarification on deployment with Traefik, Docker-Compose, Swarm mode (from dockerswarm.rocks)

[tsh356]

I have a question about deployment using the ideas from dockerswarm.rocks.

Is the Traefik service found here: https://dockerswarm.rocks/traefik/

version: '3.3'

services:

  traefik:
    # Use the latest Traefik image
    image: traefik:v2.2
    ports:
      # Listen on port 80, default for HTTP, necessary to redirect to HTTPS
      - 80:80
      # Listen on port 443, default for HTTPS
      - 443:443
    deploy:
      placement:
        constraints:
          # Make the traefik service run only on the node with this label
          # as the node with it has the volume for the certificates
          - node.labels.traefik-public.traefik-public-certificates == true
      labels:
        # Enable Traefik for this service, to make it available in the public network
        - traefik.enable=true
        # Use the traefik-public network (declared below)
        - traefik.docker.network=traefik-public
        # Use the custom label "traefik.constraint-label=traefik-public"
        # This public Traefik will only use services with this label
        # That way you can add other internal Traefik instances per stack if needed
        - traefik.constraint-label=traefik-public
        # admin-auth middleware with HTTP Basic auth
        # Using the environment variables USERNAME and HASHED_PASSWORD
        - traefik.http.middlewares.admin-auth.basicauth.users=${USERNAME?Variable not set}:${HASHED_PASSWORD?Variable not set}
        # https-redirect middleware to redirect HTTP to HTTPS
        # It can be re-used by other stacks in other Docker Compose files
        - traefik.http.middlewares.https-redirect.redirectscheme.scheme=https
        - traefik.http.middlewares.https-redirect.redirectscheme.permanent=true
        # traefik-http set up only to use the middleware to redirect to https
        # Uses the environment variable DOMAIN
        - traefik.http.routers.traefik-public-http.rule=Host(`${DOMAIN?Variable not set}`)
        - traefik.http.routers.traefik-public-http.entrypoints=http
        - traefik.http.routers.traefik-public-http.middlewares=https-redirect
        # traefik-https the actual router using HTTPS
        # Uses the environment variable DOMAIN
        - traefik.http.routers.traefik-public-https.rule=Host(`${DOMAIN?Variable not set}`)
        - traefik.http.routers.traefik-public-https.entrypoints=https
        - traefik.http.routers.traefik-public-https.tls=true
        # Use the special Traefik service api@internal with the web UI/Dashboard
        - traefik.http.routers.traefik-public-https.service=api@internal
        # Use the "le" (Let's Encrypt) resolver created below
        - traefik.http.routers.traefik-public-https.tls.certresolver=le
        # Enable HTTP Basic auth, using the middleware created above
        - traefik.http.routers.traefik-public-https.middlewares=admin-auth
        # Define the port inside of the Docker service to use
        - traefik.http.services.traefik-public.loadbalancer.server.port=8080
    volumes:
      # Add Docker as a mounted volume, so that Traefik can read the labels of other services
      - /var/run/docker.sock:/var/run/docker.sock:ro
      # Mount the volume to store the certificates
      - traefik-public-certificates:/certificates
    command:
      # Enable Docker in Traefik, so that it reads labels from Docker services
      - --providers.docker
      # Add a constraint to only use services with the label "traefik.constraint-label=traefik-public"
      - --providers.docker.constraints=Label(`traefik.constraint-label`, `traefik-public`)
      # Do not expose all Docker services, only the ones explicitly exposed
      - --providers.docker.exposedbydefault=false
      # Enable Docker Swarm mode
      - --providers.docker.swarmmode
      # Create an entrypoint "http" listening on address 80
      - --entrypoints.http.address=:80
      # Create an entrypoint "https" listening on address 80
      - --entrypoints.https.address=:443
      # Create the certificate resolver "le" for Let's Encrypt, uses the environment variable EMAIL
      - --certificatesresolvers.le.acme.email=${EMAIL?Variable not set}
      # Store the Let's Encrypt certificates in the mounted volume
      - --certificatesresolvers.le.acme.storage=/certificates/acme.json
      # Use the TLS Challenge for Let's Encrypt
      - --certificatesresolvers.le.acme.tlschallenge=true
      # Enable the access log, with HTTP requests
      - --accesslog
      # Enable the Traefik log, for configurations and errors
      - --log
      # Enable the Dashboard and API
      - --api
    networks:
      # Use the public network created to be shared between Traefik and
      # any other service that needs to be publicly available with HTTPS
      - traefik-public

volumes:
  # Create a volume to store the certificates, there is a constraint to make sure
  # Traefik is always deployed to the same Docker node with the same volume containing
  # the HTTPS certificates
  traefik-public-certificates:

networks:
  # Use the previously created public network "traefik-public", shared with other
  # services that need to be publicly available via this Traefik
  traefik-public:
    external: true

meant to replace/merge with the “proxy” service in docker-compose.yml below? Or do we need both? If we need both, what is their purpose?

version: "3.3"
services:

  proxy:
    image: traefik:v2.2
    networks:
      - ${TRAEFIK_PUBLIC_NETWORK?Variable not set}
      - default
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
    command:
      # Enable Docker in Traefik, so that it reads labels from Docker services
      - --providers.docker
      # Add a constraint to only use services with the label for this stack
      # from the env var TRAEFIK_TAG
      - --providers.docker.constraints=Label(`traefik.constraint-label-stack`, `${TRAEFIK_TAG?Variable not set}`)
      # Do not expose all Docker services, only the ones explicitly exposed
      - --providers.docker.exposedbydefault=false
      # Enable Docker Swarm mode
      - --providers.docker.swarmmode
      # Enable the access log, with HTTP requests
      - --accesslog
      # Enable the Traefik log, for configurations and errors
      - --log
      # Enable the Dashboard and API
      - --api
    deploy:
      placement:
        constraints:
          - node.role == manager
      labels:
        # Enable Traefik for this service, to make it available in the public network
        - traefik.enable=true
        # Use the traefik-public network (declared below)
        - traefik.docker.network=${TRAEFIK_PUBLIC_NETWORK?Variable not set}
        # Use the custom label "traefik.constraint-label=traefik-public"
        # This public Traefik will only use services with this label
        - traefik.constraint-label=${TRAEFIK_PUBLIC_TAG?Variable not set}
        # traefik-http set up only to use the middleware to redirect to https
        - traefik.http.middlewares.${STACK_NAME?Variable not set}-https-redirect.redirectscheme.scheme=https
        - traefik.http.middlewares.${STACK_NAME?Variable not set}-https-redirect.redirectscheme.permanent=true
        # Handle host with and without "www" to redirect to only one of them
        # Uses environment variable DOMAIN
        # To disable www redirection remove the Host() you want to discard, here and
        # below for HTTPS
        - traefik.http.routers.${STACK_NAME?Variable not set}-proxy-http.rule=Host(`${DOMAIN?Variable not set}`) || Host(`www.${DOMAIN?Variable not set}`)
        - traefik.http.routers.${STACK_NAME?Variable not set}-proxy-http.entrypoints=http
        # traefik-https the actual router using HTTPS
        - traefik.http.routers.${STACK_NAME?Variable not set}-proxy-https.rule=Host(`${DOMAIN?Variable not set}`) || Host(`www.${DOMAIN?Variable not set}`)
        - traefik.http.routers.${STACK_NAME?Variable not set}-proxy-https.entrypoints=https
        - traefik.http.routers.${STACK_NAME?Variable not set}-proxy-https.tls=true
        # Use the "le" (Let's Encrypt) resolver created below
        - traefik.http.routers.${STACK_NAME?Variable not set}-proxy-https.tls.certresolver=le
        # Define the port inside of the Docker service to use
        - traefik.http.services.${STACK_NAME?Variable not set}-proxy.loadbalancer.server.port=80
        # Handle domain with and without "www" to redirect to only one
        # To disable www redirection remove the next line
        - traefik.http.middlewares.${STACK_NAME?Variable not set}-www-redirect.redirectregex.regex=^https?://(www.)?(${DOMAIN?Variable not set})/(.*)
        # Redirect a domain with www to non-www
        # To disable it remove the next line
        - traefik.http.middlewares.${STACK_NAME?Variable not set}-www-redirect.redirectregex.replacement=https://${DOMAIN?Variable not set}/$${3}
        # Redirect a domain without www to www
        # To enable it remove the previous line and uncomment the next
        # - traefik.http.middlewares.${STACK_NAME}-www-redirect.redirectregex.replacement=https://www.${DOMAIN}/$${3}
        # Middleware to redirect www, to disable it remove the next line 
        - traefik.http.routers.${STACK_NAME?Variable not set}-proxy-https.middlewares=${STACK_NAME?Variable not set}-www-redirect
        # Middleware to redirect www, and redirect HTTP to HTTPS
        # to disable www redirection remove the section: ${STACK_NAME?Variable not set}-www-redirect,
        - traefik.http.routers.${STACK_NAME?Variable not set}-proxy-http.middlewares=${STACK_NAME?Variable not set}-www-redirect,${STACK_NAME?Variable not set}-https-redirect

[phortonssf]

In a similar position looking to deploy and get certs for HTTPS. +1

[paxcodes]

You'd need both. The discussion in issue #116 might help especially starting with @tiangolo's comment

[untilyou58]

Do I have to create traefik service(as in https://dockerswarm.rocks/traefik/) in docker-compose.yml or do I have to modify the proxy in docker-compose.override.yml?

[jakkritz]

Do I have to create traefik service(as in https://dockerswarm.rocks/traefik/) in docker-compose.yml or do I have to modify the proxy in docker-compose.override.yml?

You need both.

[phortonssf]

@tsh356 Were you able to resolve this? I can't get this SSL certs to work properly.

[tsh356]

Yes, I resolved this. You need both Traefik services, traefik.yml is deployed to its own stack and then this project is deployed to a separate stack. Essentially just follow the steps here https://dockerswarm.rocks/ and then here https://dockerswarm.rocks/traefik/ sequentially. Here's was the confusing part for me:

Create the public network:

sudo docker network create --driver=overlay traefik-public
export NODE_ID=$(sudo docker info -f '{{.Swarm.NodeID}}')
sudo docker node update --label-add traefik-public.traefik-public-certificates-live=true $NODE_ID

Then deploy stack #1, the public facing Traefik (note: this DOMAIN is the domain for the traefik web app, not your app domain):

curl -L dockerswarm.rocks/traefik.yml -o traefik.yml
export [email protected] && export DOMAIN=traefik.example.com && export USERNAME=admin && export PASSWORD=xxxxxx && export HASHED_PASSWORD=$(openssl passwd -apr1 $PASSWORD)
docker stack deploy -c traefik.yml traefik

You can then deploy your app stack which contains the second Traefik service (proxy) as well as all the other services contained in docker-compose.yml using scripts/build.sh and scripts/deploy.sh

[phortonssf]

@tsh356 I didn't realize there were deploy scripts. I think I am hung up where you were on the DOMAIN env variable. So the two different stacks are using the same env variable name, DOMAIN but with different values The values change from the script / build order do. Is that correct?

[caraseli02]

@tsh356 thanks for clarification.

[nareto]

thanks to everybody in this thread, I hadn't understood at all that one is supposed to have two nested traefik containers. I'm not sure where in the docs this would fit, but I think it should be explained more explicitly/verbosely somewhere.

(I NOW SOLVED THIS, see edit below) The theory is now clear (or so it seemed), however I'm only able to reach the dashboard of the public facing traefik container and none of the services. I've set DNS records for both traefik.sys.mydomain.com and mydomain.com to my server's ip, used the former as DOMAIN when creating the outer traefik service and the latter as DOMAIN when running scripts/deploy.sh.

Going to mydomain.com and mydomain.com/docs shows a 404 error (the logs of the public facing traefik show the GET request but no error), while pgadmin.mydomain.com shows the browser's "Page not found" error (and nothing shows up in the logs of either traefik container). In the dashboard at traefik.sys.mydomain.com I see services sname-pgadmin, sname-flower and sname-proxy alongside their routers (where sname is the value I set for STACK_NAME when deploying the stack), but nothing about backend... I noticed that pgadmin, flower and proxy have the label

deploy:
  labels:
    - traefik.constraint-label=${TRAEFIK_PUBLIC_TAG?Variable not set}

while the backend has

deploy:
  labels:
    - traefik.constraint-label-stack=${TRAEFIK_TAG?Variable not set}

So then is the idea to have pgadmin, flower and proxy exposed (and anyway, why can't I reach them)? And have backend reachable only through proxy, so it would show up theoretically in the dashboard of the "inner" traefik (i.e. proxy service) but not in the public facing one? Why not make also pgadmin and flower go through the proxy then?

EDIT: I'm embarassed to admit, after going with a comb through all the files... I had misspelled my domain when deploying the stack 😔 Now everything works as expected.

[totalhack]

I'm also trying to get this working and was surprised to find out that two Traefiks are necessary. The dockerswarm.rocks docs state:

But doing it in a way that allows you to have other Traefik services inside each stack without interfering with each other, to redirect based on path in the same stack (e.g. one container handles / for a web frontend and another handles /api for an API under the same domain), or to redirect from HTTP to HTTPS selectively.

...which makes it sound like having two is really for if you want to run multiple stacks. But if you just want to run a single stack would it be possible to get everything done with a single Traefik instance?

[gerazenobi]

+1 for a more detailed explanation/documentation 😓 (a diagram would be awesome)

I was imagining the 1 traefik public instance would listen/handle all services in the different nodes; not sure I quite get the need for a second internal traefik nor do I understand why some 'internal' services are exposed in the external traefik rather than in the internal stack traefik (e.g. pgadmin, flower)

the idea for different stacks is for different env versions of the same app stack (staging, production), or different applications altogether ?

[karolzlot]

Is it necessary to copy all project files / clone repo to server? Or maybe is it possible to deploy from local computer to server directly?

Edit: it is necessary. I will probably automate it somehow with pyinfra or ci/cd.