traefik icon indicating copy to clipboard operation
traefik copied to clipboard
Published 4 months ago • verified publisher icon traefik

Traefik 3.6 404s all containers

Open jinks opened this issue 5 months ago • 5 comments

Welcome!

  • [x] Yes, I've searched similar issues on GitHub and didn't find any.
  • [x] Yes, I've searched similar issues on the Traefik community forum and didn't find any.

What did you do?

I updated my traefik podman container from 3.5 to 3.6 and expected seamless continuation of all services.

What did you see instead?

All proxied containers instantly returned 404, this includes traefik's own dashboard.

What version of Traefik are you using?

Version: 3.6.0 Codename: ramequin Go version: go1.24.10 Built: 2025-11-07T15:22:11Z OS/Arch: linux/amd64

What is your environment & configuration?

[tls]
  [[tls.certificates]]
    certFile = "REDACTED"
    keyFile = "REDACTED"

  [tls.stores]
    [tls.stores.default]
      [tls.stores.default.defaultCertificate]
        certFile = "REDACTED"
        keyFile = "REDACTED"

Command line: '--api.dashboard=true' --providers.docker '--providers.docker.exposedbydefault=false' '--entrypoints.web.address=:80' '--entryPoints.web.http.redirections.entryPoint.to=websecure' '--entryPoints.web.htt p.redirections.entryPoint.scheme=https' '--entrypoints.websecure.address=:443' '--providers.file.filename=/etc/traefik/docker.toml' '--log.level=INFO'

If applicable, please paste the log output in DEBUG level

Nov 09 08:39:21 host systemd[525]: Started traefik.service.
Nov 09 08:39:21 host traefik[21127]: 1375bef2019c9e940257535e1b6c028246cf7f0f7375d5b0c18ee1a1f2169d76
Nov 09 08:39:22 host systemd-traefik[21227]: 2025-11-09T07:39:22Z INF Traefik version 3.6.0 built on 2025-11-07T15:22:11Z version=3.6.0
Nov 09 08:39:22 host systemd-traefik[21227]: 2025-11-09T07:39:22Z INF 
Nov 09 08:39:22 host systemd-traefik[21227]: Stats collection is disabled.
Nov 09 08:39:22 host systemd-traefik[21227]: Help us improve Traefik by turning this feature on :)
Nov 09 08:39:22 host systemd-traefik[21227]: More details on: https://doc.traefik.io/traefik/contributing/data-collection/
Nov 09 08:39:22 host systemd-traefik[21227]: 
Nov 09 08:39:22 host systemd-traefik[21227]: 2025-11-09T07:39:22Z INF Starting provider aggregator *aggregator.ProviderAggregator
Nov 09 08:39:22 host systemd-traefik[21227]: 2025-11-09T07:39:22Z INF Starting provider *file.Provider
Nov 09 08:39:22 host systemd-traefik[21227]: 2025-11-09T07:39:22Z INF Starting provider *traefik.Provider
Nov 09 08:39:22 host systemd-traefik[21227]: 2025-11-09T07:39:22Z INF Starting provider *docker.Provider
Nov 09 08:39:22 host systemd-traefik[21227]: 2025-11-09T07:39:22Z INF Starting provider *acme.ChallengeTLSALPN
Nov 09 08:39:22 host systemd-traefik[21227]: 2025-11-09T07:39:22Z ERR Failed to list containers for docker error="Error response from daemon: getting container from store \"750b0a783e617fd9b4f1b3d60725e87439cb17639373033c575e1728b0c52309\": container not known" providerName=docker
Nov 09 08:39:22 host systemd-traefik[21227]: 2025-11-09T07:39:22Z ERR Provider error, retrying in 262.232322ms error="Error response from daemon: getting container from store \"750b0a783e617fd9b4f1b3d60725e87439cb17639373033c575e1728b0c52309\": container not known" providerName=docker
Nov 09 08:39:22 host systemd-traefik[21227]: 2025-11-09T07:39:22Z ERR Failed to list containers for docker error="Error response from daemon: getting container from store \"750b0a783e617fd9b4f1b3d60725e87439cb17639373033c575e1728b0c52309\": container not known" providerName=docker
Nov 09 08:39:22 host systemd-traefik[21227]: 2025-11-09T07:39:22Z ERR Provider error, retrying in 490.361676ms error="Error response from daemon: getting container from store \"750b0a783e617fd9b4f1b3d60725e87439cb17639373033c575e1728b0c52309\": container not known" providerName=docker
Nov 09 08:39:23 host systemd-traefik[21227]: 2025-11-09T07:39:23Z ERR Failed to list containers for docker error="Error response from daemon: getting container from store \"750b0a783e617fd9b4f1b3d60725e87439cb17639373033c575e1728b0c52309\": container not known" providerName=docker
Nov 09 08:39:23 host systemd-traefik[21227]: 2025-11-09T07:39:23Z ERR Provider error, retrying in 1.505756874s error="Error response from daemon: getting container from store \"750b0a783e617fd9b4f1b3d60725e87439cb17639373033c575e1728b0c52309\": container not known" providerName=docker
Nov 09 08:39:25 host systemd-traefik[21227]: 2025-11-09T07:39:25Z ERR Failed to list containers for docker error="Error response from daemon: getting container from store \"750b0a783e617fd9b4f1b3d60725e87439cb17639373033c575e1728b0c52309\": container not known" providerName=docker
Nov 09 08:39:25 host systemd-traefik[21227]: 2025-11-09T07:39:25Z ERR Provider error, retrying in 1.652034971s error="Error response from daemon: getting container from store \"750b0a783e617fd9b4f1b3d60725e87439cb17639373033c575e1728b0c52309\": container not known" providerName=docker
Nov 09 08:39:27 host systemd-traefik[21227]: 2025-11-09T07:39:27Z ERR Failed to list containers for docker error="Error response from daemon: getting container from store \"750b0a783e617fd9b4f1b3d60725e87439cb17639373033c575e1728b0c52309\": container not known" providerName=docker
Nov 09 08:39:27 host systemd-traefik[21227]: 2025-11-09T07:39:27Z ERR Provider error, retrying in 2.051909076s error="Error response from daemon: getting container from store \"750b0a783e617fd9b4f1b3d60725e87439cb17639373033c575e1728b0c52309\": container not known" providerName=docker
Nov 09 08:39:29 host systemd-traefik[21227]: 2025-11-09T07:39:29Z ERR Failed to list containers for docker error="Error response from daemon: getting container from store \"750b0a783e617fd9b4f1b3d60725e87439cb17639373033c575e1728b0c52309\": container not known" providerName=docker
Nov 09 08:39:29 host systemd-traefik[21227]: 2025-11-09T07:39:29Z ERR Provider error, retrying in 4.140889781s error="Error response from daemon: getting container from store \"750b0a783e617fd9b4f1b3d60725e87439cb17639373033c575e1728b0c52309\": container not known" providerName=docker
Nov 09 08:39:33 host systemd-traefik[21227]: 2025-11-09T07:39:33Z ERR Failed to list containers for docker error="Error response from daemon: getting container from store \"750b0a783e617fd9b4f1b3d60725e87439cb17639373033c575e1728b0c52309\": container not known" providerName=docker
Nov 09 08:39:33 host systemd-traefik[21227]: 2025-11-09T07:39:33Z ERR Provider error, retrying in 6.687928005s error="Error response from daemon: getting container from store \"750b0a783e617fd9b4f1b3d60725e87439cb17639373033c575e1728b0c52309\": container not known" providerName=docker
Nov 09 08:39:40 host systemd-traefik[21227]: 2025-11-09T07:39:40Z ERR Failed to list containers for docker error="Error response from daemon: getting container from store \"750b0a783e617fd9b4f1b3d60725e87439cb17639373033c575e1728b0c52309\": container not known" providerName=docker
Nov 09 08:39:40 host systemd-traefik[21227]: 2025-11-09T07:39:40Z ERR Provider error, retrying in 6.826821194s error="Error response from daemon: getting container from store \"750b0a783e617fd9b4f1b3d60725e87439cb17639373033c575e1728b0c52309\": container not known" providerName=docker
Nov 09 08:39:47 host systemd-traefik[21227]: 2025-11-09T07:39:47Z ERR Failed to list containers for docker error="Error response from daemon: getting container from store \"750b0a783e617fd9b4f1b3d60725e87439cb17639373033c575e1728b0c52309\": container not known" providerName=docker
Nov 09 08:39:47 host systemd-traefik[21227]: 2025-11-09T07:39:47Z ERR Provider error, retrying in 17.534615934s error="Error response from daemon: getting container from store \"750b0a783e617fd9b4f1b3d60725e87439cb17639373033c575e1728b0c52309\": container not known" providerName=docker
Nov 09 08:40:05 host systemd-traefik[21227]: 2025-11-09T07:40:05Z ERR Failed to list containers for docker error="Error response from daemon: getting container from store \"750b0a783e617fd9b4f1b3d60725e87439cb17639373033c575e1728b0c52309\": container not known" providerName=docker
Nov 09 08:40:05 host systemd-traefik[21227]: 2025-11-09T07:40:05Z ERR Provider error, retrying in 332.25873ms error="Error response from daemon: getting container from store \"750b0a783e617fd9b4f1b3d60725e87439cb17639373033c575e1728b0c52309\": container not known" providerName=docker
Nov 09 08:40:05 host systemd-traefik[21227]: 2025-11-09T07:40:05Z ERR Failed to list containers for docker error="Error response from daemon: getting container from store \"750b0a783e617fd9b4f1b3d60725e87439cb17639373033c575e1728b0c52309\": container not known" providerName=docker
Nov 09 08:40:05 host systemd-traefik[21227]: 2025-11-09T07:40:05Z ERR Provider error, retrying in 1.115357378s error="Error response from daemon: getting container from store \"750b0a783e617fd9b4f1b3d60725e87439cb17639373033c575e1728b0c52309\": container not known" providerName=docker
Nov 09 08:40:07 host systemd-traefik[21227]: 2025-11-09T07:40:07Z ERR Failed to list containers for docker error="Error response from daemon: getting container from store \"750b0a783e617fd9b4f1b3d60725e87439cb17639373033c575e1728b0c52309\": container not known" providerName=docker
Nov 09 08:40:07 host systemd-traefik[21227]: 2025-11-09T07:40:07Z ERR Provider error, retrying in 1.389800745s error="Error response from daemon: getting container from store \"750b0a783e617fd9b4f1b3d60725e87439cb17639373033c575e1728b0c52309\": container not known" providerName=docker
Nov 09 08:40:08 host systemd-traefik[21227]: 2025-11-09T07:40:08Z ERR Failed to list containers for docker error="Error response from daemon: getting container from store \"750b0a783e617fd9b4f1b3d60725e87439cb17639373033c575e1728b0c52309\": container not known" providerName=docker
Nov 09 08:40:08 host systemd-traefik[21227]: 2025-11-09T07:40:08Z ERR Provider error, retrying in 1.850972141s error="Error response from daemon: getting container from store \"750b0a783e617fd9b4f1b3d60725e87439cb17639373033c575e1728b0c52309\": container not known" providerName=docker
Nov 09 08:40:10 host systemd-traefik[21227]: 2025-11-09T07:40:10Z ERR Failed to list containers for docker error="Error response from daemon: getting container from store \"750b0a783e617fd9b4f1b3d60725e87439cb17639373033c575e1728b0c52309\": container not known" providerName=docker
Nov 09 08:40:10 host systemd-traefik[21227]: 2025-11-09T07:40:10Z ERR Provider error, retrying in 2.046880094s error="Error response from daemon: getting container from store \"750b0a783e617fd9b4f1b3d60725e87439cb17639373033c575e1728b0c52309\": container not known" providerName=docker
Nov 09 08:40:12 host systemd-traefik[21227]: 2025-11-09T07:40:12Z ERR Failed to list containers for docker error="Error response from daemon: getting container from store \"750b0a783e617fd9b4f1b3d60725e87439cb17639373033c575e1728b0c52309\": container not known" providerName=docker
Nov 09 08:40:12 host systemd-traefik[21227]: 2025-11-09T07:40:12Z ERR Provider error, retrying in 1.899916593s error="Error response from daemon: getting container from store \"750b0a783e617fd9b4f1b3d60725e87439cb17639373033c575e1728b0c52309\": container not known" providerName=docker
Nov 09 08:40:14 host systemd-traefik[21227]: 2025-11-09T07:40:14Z ERR Failed to list containers for docker error="Error response from daemon: getting container from store \"750b0a783e617fd9b4f1b3d60725e87439cb17639373033c575e1728b0c52309\": container not known" providerName=docker
Nov 09 08:40:14 host systemd-traefik[21227]: 2025-11-09T07:40:14Z ERR Provider error, retrying in 4.40974574s error="Error response from daemon: getting container from store \"750b0a783e617fd9b4f1b3d60725e87439cb17639373033c575e1728b0c52309\": container not known" providerName=docker
Nov 09 08:40:19 host systemd-traefik[21227]: 2025-11-09T07:40:19Z ERR Failed to list containers for docker error="Error response from daemon: getting container from store \"750b0a783e617fd9b4f1b3d60725e87439cb17639373033c575e1728b0c52309\": container not known" providerName=docker
Nov 09 08:40:19 host systemd-traefik[21227]: 2025-11-09T07:40:19Z ERR Provider error, retrying in 12.284968334s error="Error response from daemon: getting container from store \"750b0a783e617fd9b4f1b3d60725e87439cb17639373033c575e1728b0c52309\": container not known" providerName=docker
Nov 09 08:40:31 host systemd-traefik[21227]: 2025-11-09T07:40:31Z ERR Failed to list containers for docker error="Error response from daemon: getting container from store \"750b0a783e617fd9b4f1b3d60725e87439cb17639373033c575e1728b0c52309\": container not known" providerName=docker
Nov 09 08:40:31 host systemd-traefik[21227]: 2025-11-09T07:40:31Z ERR Provider error, retrying in 12.40279407s error="Error response from daemon: getting container from store \"750b0a783e617fd9b4f1b3d60725e87439cb17639373033c575e1728b0c52309\": container not known" providerName=docker
Nov 09 08:40:33 host systemd[525]: Stopping traefik.service...
Nov 09 08:40:35 host systemd-traefik[21227]: 2025-11-09T07:40:35Z INF I have to go...
Nov 09 08:40:35 host systemd-traefik[21227]: 2025-11-09T07:40:35Z INF Stopping server gracefully
Nov 09 08:40:36 host systemd-traefik[21227]: 2025-11-09T07:40:36Z INF Shutting down
Nov 09 08:40:36 host systemd-traefik[21227]: 2025-11-09T07:40:36Z ERR Cannot retrieve data error="context canceled" providerName=docker
Nov 09 08:40:36 host systemd-traefik[21227]: 2025-11-09T07:40:36Z INF Server stopped
Nov 09 08:40:36 host podman[21794]: 2025-11-09 08:40:36.226158733 +0100 CET m=+2.546368130 container died 1375bef2019c9e940257535e1b6c028246cf7f0f7375d5b0c18ee1a1f2169d76 (image=docker.io/library/traefik:v3.6, name=systemd-traefik, traefik.http.routers.api.middlewares=auth, traefik.http.routers.api.rule=( Host(`traefik.XXXXX.XXX`) || Host(`traefik.XXXXX.XXX`) ) && ( PathPrefix(`/api`) || PathPrefix(`/dashboard`) ), traefik.enable=true, org.opencontainers.image.description=A modern reverse-proxy, traefik.http.routers.api.service=api@internal, org.opencontainers.image.source=https://github.com/traefik/traefik, PODMAN_SYSTEMD_UNIT=traefik.service, org.opencontainers.image.documentation=https://docs.traefik.io, traefik.http.routers.api.tls=true, traefik.http.middlewares.auth.basicauth.users=admin:$apr1$eJN8MKjF$XBmJuCqsJaVmqf4vfh2NG0, traefik.http.routers.api.entrypoints=websecure, org.opencontainers.image.title=Traefik, org.opencontainers.image.vendor=Traefik Labs, io.containers.autoupdate=registry, org.opencontainers.image.url=https://traefik.io, org.opencontainers.image.version=v3.6.0)
Nov 09 08:40:36 host podman[21794]: 2025-11-09 08:40:36.378547152 +0100 CET m=+2.698756539 container remove 1375bef2019c9e940257535e1b6c028246cf7f0f7375d5b0c18ee1a1f2169d76 (image=docker.io/library/traefik:v3.6, name=systemd-traefik, org.opencontainers.image.title=Traefik, org.opencontainers.image.version=v3.6.0, traefik.http.routers.api.rule=( Host(`traefik.XXXXX.XXX`) || Host(`traefik.XXXXX.XXX`) ) && ( PathPrefix(`/api`) || PathPrefix(`/dashboard`) ), traefik.http.routers.api.service=api@internal, traefik.enable=true, org.opencontainers.image.source=https://github.com/traefik/traefik, traefik.http.routers.api.tls=true, io.containers.autoupdate=registry, org.opencontainers.image.url=https://traefik.io, traefik.http.routers.api.entrypoints=websecure, org.opencontainers.image.documentation=https://docs.traefik.io, org.opencontainers.image.vendor=Traefik Labs, traefik.http.routers.api.middlewares=auth, traefik.http.middlewares.auth.basicauth.users=admin:$apr1$eJN8MKjF$XBmJuCqsJaVmqf4vfh2NG0, PODMAN_SYSTEMD_UNIT=traefik.service, org.opencontainers.image.description=A modern reverse-proxy)
Nov 09 08:40:36 host traefik[21794]: systemd-traefik
Nov 09 08:40:36 host systemd[525]: Stopped traefik.service.
Nov 09 08:40:36 host systemd[525]: traefik.service: Consumed 3.370s CPU time, 383.3M memory peak.
Nov 09 08:40:36 host systemd[525]: Starting traefik.service...
Nov 09 08:40:36 host podman[21849]: 2025-11-09 08:40:36.521512632 +0100 CET m=+0.067280039 container create 0c8b8d2d4997f24b195daf9aea60380ee740a9cf64b214ac2554e435b00bac0f (image=docker.io/library/traefik:v3.5, name=systemd-traefik, org.opencontainers.image.title=Traefik, org.opencontainers.image.url=https://traefik.io, traefik.http.middlewares.auth.basicauth.users=admin:$apr1$eJN8MKjF$XBmJuCqsJaVmqf4vfh2NG0, org.opencontainers.image.description=A modern reverse-proxy, org.opencontainers.image.documentation=https://docs.traefik.io, PODMAN_SYSTEMD_UNIT=traefik.service, traefik.http.routers.api.middlewares=auth, io.containers.autoupdate=registry, org.opencontainers.image.source=https://github.com/traefik/traefik, traefik.http.routers.api.tls=true, org.opencontainers.image.vendor=Traefik Labs, traefik.enable=true, org.opencontainers.image.version=v3.5.6, traefik.http.routers.api.service=api@internal, traefik.http.routers.api.rule=( Host(`traefik.XXXXX.XXX`) || Host(`traefik.XXXXX.XXX`) ) && ( PathPrefix(`/api`) || PathPrefix(`/dashboard`) ), traefik.http.routers.api.entrypoints=websecure)
Nov 09 08:40:36 host podman[21849]: 2025-11-09 08:40:36.475572105 +0100 CET m=+0.021339531 image pull 50542ae5aab61184f1865f9120650fbb8761f54266b21c1709b3096f9add0fa2 docker.io/traefik:v3.5
Nov 09 08:40:36 host podman[21849]: 2025-11-09 08:40:36.617469535 +0100 CET m=+0.163236972 container init 0c8b8d2d4997f24b195daf9aea60380ee740a9cf64b214ac2554e435b00bac0f (image=docker.io/library/traefik:v3.5, name=systemd-traefik, org.opencontainers.image.version=v3.5.6, org.opencontainers.image.source=https://github.com/traefik/traefik, org.opencontainers.image.title=Traefik, io.containers.autoupdate=registry, org.opencontainers.image.description=A modern reverse-proxy, traefik.http.routers.api.tls=true, traefik.http.routers.api.service=api@internal, traefik.http.routers.api.middlewares=auth, traefik.http.routers.api.rule=( Host(`traefik.XXXXX.XXX`) || Host(`traefik.XXXXX.XXX`) ) && ( PathPrefix(`/api`) || PathPrefix(`/dashboard`) ), org.opencontainers.image.url=https://traefik.io, traefik.enable=true, org.opencontainers.image.vendor=Traefik Labs, PODMAN_SYSTEMD_UNIT=traefik.service, traefik.http.middlewares.auth.basicauth.users=admin:$apr1$eJN8MKjF$XBmJuCqsJaVmqf4vfh2NG0, traefik.http.routers.api.entrypoints=websecure, org.opencontainers.image.documentation=https://docs.traefik.io)
Nov 09 08:40:36 host podman[21849]: 2025-11-09 08:40:36.620067168 +0100 CET m=+0.165834574 container start 0c8b8d2d4997f24b195daf9aea60380ee740a9cf64b214ac2554e435b00bac0f (image=docker.io/library/traefik:v3.5, name=systemd-traefik, traefik.http.routers.api.tls=true, traefik.http.routers.api.service=api@internal, traefik.http.routers.api.middlewares=auth, traefik.http.routers.api.rule=( Host(`traefik.XXXXX.XXX`) || Host(`traefik.XXXXX.XXX`) ) && ( PathPrefix(`/api`) || PathPrefix(`/dashboard`) ), traefik.http.routers.api.entrypoints=websecure, org.opencontainers.image.title=Traefik, traefik.enable=true, org.opencontainers.image.description=A modern reverse-proxy, org.opencontainers.image.vendor=Traefik Labs, PODMAN_SYSTEMD_UNIT=traefik.service, traefik.http.middlewares.auth.basicauth.users=admin:$apr1$eJN8MKjF$XBmJuCqsJaVmqf4vfh2NG0, org.opencontainers.image.url=https://traefik.io, org.opencontainers.image.version=v3.5.6, io.containers.autoupdate=registry, org.opencontainers.image.documentation=https://docs.traefik.io, org.opencontainers.image.source=https://github.com/traefik/traefik)
Nov 09 08:40:36 host systemd[525]: Started traefik.service.
Nov 09 08:40:36 host traefik[21849]: 0c8b8d2d4997f24b195daf9aea60380ee740a9cf64b214ac2554e435b00bac0f
Nov 09 08:40:36 host systemd-traefik[21925]: 2025-11-09T07:40:36Z INF Traefik version 3.5.6 built on 2025-11-07T13:47:38Z version=3.5.6
Nov 09 08:40:36 host systemd-traefik[21925]: 2025-11-09T07:40:36Z INF 
Nov 09 08:40:36 host systemd-traefik[21925]: Stats collection is disabled.
Nov 09 08:40:36 host systemd-traefik[21925]: Help us improve Traefik by turning this feature on :)
Nov 09 08:40:36 host systemd-traefik[21925]: More details on: https://doc.traefik.io/traefik/contributing/data-collection/
Nov 09 08:40:36 host systemd-traefik[21925]: 
Nov 09 08:40:36 host systemd-traefik[21925]: 2025-11-09T07:40:36Z INF Starting provider aggregator *aggregator.ProviderAggregator
Nov 09 08:40:36 host systemd-traefik[21925]: 2025-11-09T07:40:36Z INF Starting provider *file.Provider
Nov 09 08:40:36 host systemd-traefik[21925]: 2025-11-09T07:40:36Z INF Starting provider *docker.Provider
Nov 09 08:40:36 host systemd-traefik[21925]: 2025-11-09T07:40:36Z INF Starting provider *traefik.Provider
Nov 09 08:40:36 host systemd-traefik[21925]: 2025-11-09T07:40:36Z INF Starting provider *acme.ChallengeTLSALPN

jinks avatar Nov 09 '25 07:11 jinks

After I upgraded to Ubuntu 24.04 LTS, and upgraded the traefik v3.6.0 at the same time, including Docker to v29.0.0, now all my Docker container labels no longer work, including the traefik dashboard itself, BUT, all the config.yaml file-based proxy is working. I tried to downgrade to v3.5.4, and it still does not work. No Traefik logs, too.

Turns out it is issued #12253 , the issue seems to be related to your.

luinniuxx avatar Nov 12 '25 02:11 luinniuxx

Hello @jinks and thanks for opening this,

This is likely a duplicate of https://github.com/traefik/traefik/issues/12253. Therefore, I am closing this issue as a duplicate.

kevinpollet avatar Nov 12 '25 08:11 kevinpollet

This is still an issue with Traefik 3.6.2. With #12253 now closed this seems to be a different issue.

jinks avatar Nov 20 '25 00:11 jinks

Thanks for your report. Could you share a simple reproducible use case?

juliens avatar Nov 20 '25 14:11 juliens