Multiple caddy instances

[revnelson]

Could somebody please provide an example of how to run multiple caddy containers? Not in a swarm or kubernetes. I'd like to have a main caddy that handles whole-server services like portainer and matomo. Then, in a separate docker-compose, I want a caddy that handles project-specific services like php-fpm, wordpress, etc.

My thought process:

• I can't mount project-specific static files to the main caddy instance (I'd like to be able to easily add/remove/move projects without needing to change the main server configs)
• I want to keep my request chain all caddy. i.e. it seems like a shame to use an apache-served wordpress to go caddy -> apache -> wordpress static files instead I'd prefer to have the project-level caddy act as a file_server for the static files with a php_fastcgi directive pointing to the project-level php-fpm service.

This is what I've tried:

Server-level docker-compose

version: "3.8"

services:
  caddy:
    container_name: caddy
    image: ${CADDY_IMAGE?err}
    restart: unless-stopped
    ports:
      - 80:80
      - 443:443
    environment:
      - CADDY_INGRESS_NETWORKS=caddy
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
      - caddy:/data
      - ./caddy/config:/config
    labels:
      caddy.email: ${EMAIL?err}
    networks:
      - caddy
    logging:
      driver: syslog
      options:
        tag: docker-caddy

  matomo_db:
    container_name: matomo_db
    image: mariadb:${MATOMO_MARIADB_VERSION?err}
    restart: unless-stopped
    command: --max-allowed-packet=128MB
    environment:
      - MYSQL_ROOT_PASSWORD=${MATOMO_DB_ROOT_PASSWORD?err}
      - MYSQL_DATABASE=${MATOMO_DB_DATABASE?err}
      - MYSQL_USER=${MATOMO_DB_USER?err}
      - MYSQL_PASSWORD=${MATOMO_DB_PASSWORD?err}
      - MARIADB_INITDB_SKIP_TZINFO=1
    volumes:
      - matomo_db:/var/lib/mysql
    networks:
      - internal
    logging:
      driver: syslog
      options:
        tag: docker-matomo_db

  matomo:
    container_name: matomo
    image: matomo:${MATOMO_VERSION?err}
    restart: unless-stopped
    environment:
      - MATOMO_DATABASE_HOST=matomo_db
      - MATOMO_DATABASE_ADAPTER=mysql
      - MATOMO_DATABASE_TABLES_PREFIX=matomo_
      - MATOMO_DATABASE_USERNAME=${MATOMO_DB_USER?err}
      - MATOMO_DATABASE_PASSWORD=${MATOMO_DB_PASSWORD?err}
      - MATOMO_DATABASE_DBNAME=${MATOMO_DB_DATABASE?err}
      - PHP_MEMORY_LIMIT=2048M
    volumes:
      - matomo:/var/www/html/
      - ./matomo/config:/var/www/html/config:rw
    labels:
      caddy: ${MATOMO_URL?err}
      caddy.reverse_proxy: "{{upstreams}}"
    networks:
      - caddy
      - internal
    depends_on:
      - matomo_db
    logging:
      driver: syslog
      options:
        tag: docker-matomo

volumes:
  caddy:
  matomo:
  matomo_db:

networks:
  caddy:
    name: caddy
  internal:
    name: server_internal
    external: false

That is working fine and I can access the matomo dashboard.

Project-level docker-compose

version: "3.8"

services:
  caddy:
    container_name: ${PROJECT_NAME?err}_caddy
    image: ${CADDY_IMAGE?err}
    restart: unless-stopped
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
      - caddy:/data
      - ./files:/var/www
    environment:
      - CADDY_DOCKER_LABEL_PREFIX=project-caddy
    labels:
      caddy: ${DOMAIN?err}
      caddy.reverse_proxy: "{{upstreams 80}}"
      project-caddy: localhost
      project-caddy.root: "* /var/www/public"
      project-caddy.file_server:
      project-caddy.encode: gzip
      project-caddy.php_fastcgi: "php-fpm:9000"
    networks:
      - caddy
      - internal
    logging:
      driver: syslog
      options:
        tag: docker-${PROJECT_NAME?err}-caddy

  php-fpm:
    container_name: ${PROJECT_NAME?err}_php
    image: php:${PHP_VERSION?err}-fpm
    restart: unless-stopped
    volumes:
      - ./files:/var/www
    networks:
      - internal
    logging:
      driver: syslog
      options:
        tag: docker-${PROJECT_NAME?err}-caddy


volumes:
  caddy:
    name: ${PROJECT_NAME?err}_caddy

networks:
  caddy:
    external: true
  internal:
    name: ${PROJECT_NAME?err}_internal
    external: false

To clarify, in my .env file I have CADDY_IMAGE=lucaslorentz/caddy-docker-proxy:ci-alpine and my ./files directory currently has a public folder that currently just has index.html and info.php until I get this working.

Originally posted by @revnelson in https://github.com/lucaslorentz/caddy-docker-proxy/issues/53#issuecomment-1455221003

[pikeas]

I'm interested in this as well, for a different reason: exposing one instance of a service under multiple domains, managed by different Caddy services.

services:
  foo:
    ...
    deploy:
      labels:
        caddy_1: foo.internal-corp.company.com
        caddy_2: app.company.com

The service can be attached to two Caddy networks, or each Caddy can be attached to this stack's network.

This doesn't work well, because the internal Caddy picks up the external directive, with DNS it can't serve. And vice versa for the external Caddy picking up the internal directive.