Bad Gateway after containers have been updated by Watchtower

[fortysix2ahead]

I experience the following behaviour for now several months now (have been using caddy-proxy for much longer time already): I am running dozens of containers which are exposed by caddy-proxy. They live in two docker networks ("caddy" and "caddy_int"). Both are configured via the environment variable CADDY_INGRESS_NETWORKS. So far, so good, everything's running smoothly - until a container is updated automatically by Watchtower. After Watchtower takes down a container and spins up a new one, they service is not accessible any longer. Caddy responds with 502 - bad gateway. This does not happen every time, but on a regular basis. After restarting the caddy container everything is fine again.

My guess is the following: when the container is updated and gets the same IP inside the docker network, all is fine. When the new container gets a new IP, different from the old one, caddy responds with 502 because all requests are sent into nirvana now. It looks like caddy is not notified that a new IP is used.

I am using caddy.reverse_proxy: {{upstreams http xxxx}} which resolves to an ip address. And if this address changes, things break.

I can't tell when I started to observe this behaviour. I am a long term user and I think caddy-proxy 2.8.x was still fine. 2.9 or 2.10 introduced the behaviour for me.

[vgallegob]

I have a similar issue, but i'm not using whatchtower.

If I restart a container, I can not reach it anymore. After I restart Caddy things go back to normal.

[fortysix2ahead]

I don't think it has to do with Watchtower. It's just the situation when a container is recreated and receives a new IP in the docker network.

[vgallegob]

Yes I agree, but how do we handle that?

[fortysix2ahead]

I could think of a workaround (without knowing the internals of caddy-proxy): I am using snippets in a dedicated Caddyfile so that in the compose file of a service there's only a single line as a label:

labels:
  caddy: "import reverse_proxy {{upstreams 8888}} whoami"

The snippet reverse_proxy contains the actual reverse proxy stuff, along with logging definiton and so on. In the end it resolves to

whoami.mydomain.com {
  reverse_proxy <docker_ip_address>:8888
  ... more stuff ...
}

This is because the template function {{upstreams 8888}} resolves to i.e. http://172.17.0.17:8888 (as documented in caddy-proxy). After container recreation the new container maybe has 172.17.0.18 and the old reverse proxy directive points to nowhere (because caddy-proxy is not notified by the change for whatever reason).

A workaround could be to get rid of the template function and use the container name:

whoami.mydomain.com {
	reverse_proxy whoami:8888
}

Using the upstreams template function is convenient, but if it breaks things ..? Didn't try that out though, because I need to rewrite my Caddyfile with the adapted snippets ...

[fortysix2ahead]

I have changed my setup now from using {{upstreams XXXX}} to containername:XXXX. Let's see if that helps.

As an observation: after taking down a stack via compose, changing from upstreams to container name and restarting the stack, Caddy gives a Bad Gateway again. This is an indicator that a change in the configuration is not detected by Caddy Proxy and not forwarded to Caddy.

[misilot]

Could caddy-docker-proxy be setup to route to servicenames instead of the ips?

[jum]

A good workaround is to use Unix domain sockets (if the software supports it). If I can, I run all my containers via sockets in a /run/containers directory that I mount into all containers. I am also not good at remembering numbers, so this is much nicer for me than the port numbers.

[fortysix2ahead]

@misilot yes, see my comment above. That's what I have changed. It's a bit more typing (service name + port), but it's not much more. Seems you can't get around providing at least the port manually.

[misilot]

@misilot yes, see my comment above. That's what I have changed. It's a bit more typing (service name + port), but it's not much more. Seems you can't get around providing at least the port manually.

Thanks! I will try that. Sorry, I read what you wrote, and than your observation, that made it sound like it wasn't possible?