ansible-role-docker-compose-generator

ansible-role-docker-compose-generator copied to clipboard

Change Summary

First of all thanks for the idea of combining all compose files together! It's quite nice to be able to have one compose per machine with all services being included.

This PR adds a fix where multiple compose files with different top-level sections (services, networks, volumes, configs, secrets) were improperly mixed, producing invalid compose file.

How it works is that it goes over all these sections in a nested loops, extracts and accumulates each section values from all compose files, one-by-one, then saves into a variable which is used in producing a final template.

Additionally, some small changes were added:

• role files were ansible-linted,
• compose files are sorted and filtered upfront.

There are some shortcomings with this approach though:

• comments are not preserved,
• even if some sections didn't exists, they're added, such as configs or secrets (probably can be tuned with additional logic, but the output compose file is still valid, so working on it felt it being art for art's sake),
• the list items are not indented, as this is a know issue with the underlying library used by to_nice_yaml filter - PyYAML: https://github.com/yaml/pyyaml/issues/234

---

Example Compose files

$ tree services
services
└── my.domain.local
    ├── 01-reverse-proxy
    │   └── compose.yaml
    └── 02-dns-dhcp
        └── compose.yaml

# 01-reverse-proxy/compose.yaml
---
services:
  reverse-proxy:
    image: traefik:v3.4.1
    container_name: traefik
    restart: unless-stopped
    command: --api.insecure=true --providers.docker=true
    ports:
      - "80:80"
      - "443:443"
      - "8080:8080" # Traefik dashboard
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
      - ./config/traefik.yaml:/etc/traefik/traefik.yaml:ro

volumes:
  config:

networks:
  frontend:
    external: true

# 02-dns-dhcp/compose.yaml
---
services:
  technitium:
    image: technitium/dns-server:13.6.0
    restart: unless-stopped
    # host network mode is required for DHCP
    network_mode: host
    # ports:
    #   - "53:53/udp"
    #   - "53:53/tcp"
    #   - "67:67/udp"     # DHCP
    #   - "5380:5380/tcp" # Web UI
    volumes:
      - config:/etc/dns
    # https://github.com/TechnitiumSoftware/DnsServer/blob/master/DockerEnvironmentVariables.md
    environment:
      - DNS_SERVER_DOMAIN=dns-server
      - FOOVAR={{ foovar }}

volumes:
  config:

Output before PR

Contains multiple mixed-up and improperly indented top-level sections such as services or volumes.

# Generated by ironicbadger.docker-compose-generator
# badger badger badger mushroom mushrooooom...

services:
  services:
  reverse-proxy:
    image: traefik:v3.4.1
    container_name: traefik
    restart: unless-stopped
    command: --api.insecure=true --providers.docker=true
    ports:
      - "80:80"
      - "443:443"
      - "8080:8080" # Traefik dashboard
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
      - ./config/traefik.yaml:/etc/traefik/traefik.yaml:ro

volumes:
  config:

networks:
  frontend:
    external: true

  services:
  technitium:
    image: technitium/dns-server:13.6.0
    restart: unless-stopped
    # host network mode is required for DHCP
    network_mode: host
    # ports:
    #   - "53:53/udp"
    #   - "53:53/tcp"
    #   - "67:67/udp"     # DHCP
    #   - "5380:5380/tcp" # Web UI
    volumes:
      - config:/etc/dns
    # https://github.com/TechnitiumSoftware/DnsServer/blob/master/DockerEnvironmentVariables.md
    environment:
      - DNS_SERVER_DOMAIN=dns-server
      - FOOVAR=bar

volumes:
  config:

Output after PR

Irons-out the above shortcomings by merging all sections first.

# Generated by ironicbadger.docker-compose-generator
# badger badger badger mushroom mushrooooom...

services:
  reverse-proxy:
    image: traefik:v3.4.1
    container_name: traefik
    restart: unless-stopped
    command: --api.insecure=true --providers.docker=true
    ports:
    - 80:80
    - 443:443
    - 8080:8080
    volumes:
    - /var/run/docker.sock:/var/run/docker.sock
    - ./config/traefik.yaml:/etc/traefik/traefik.yaml:ro
  technitium:
    image: technitium/dns-server:13.6.0
    restart: unless-stopped
    network_mode: host
    volumes:
    - config:/etc/dns
    environment:
    - DNS_SERVER_DOMAIN=dns-server
    - FOOVAR=bar

networks:
  frontend:
    external: true

volumes:
  config: null

configs: {}

secrets: {}

---

Additional benefit (or disadvantage) of this method is that if given section items (ex.: one of networks) is duplicated in multiple compose files, such as external frontend/proxy network, it's "de-duplicated" and appears in the output only once. It does it in the last-present-wins way and it works for children of all top-level sections (so services with same names will be overridden too).

If this PR was to be merged, it's probably worth documenting that (I can add it to the README.md).

Example

Assume each block is in a separate compose file, and they're already ordered:

networks:
  frontend:
    external: true

networks:
  frontend:
    external: true
  foo_network:
    name: FooNetwork

# foo_network will override the previous one
networks:
  foo_network:
  bar_network:

Final Output

networks:
  frontend:
    external: true
  foo_network: null
  bar_network: null

Since all compose files are merged anyway, one could define networks just in services/host.example.com/99-networks/compose.yaml to be sure that the networks do not override each other.