True zero-downtime deployment by request draining

[wowu]

Currently there's no way of telling Traefik that old container is going to be stopped, so it might route requests to a container that is shutting down. I'm creating this issue to track the progress of figuring out what is the best way of implementing this.

The problem was mentioned in this StackOverflow question: https://stackoverflow.com/questions/75918681/how-to-avoid-downtime-when-using-docker-rollout-with-traefik

Current idea

The easiest way seems to fail healthchecks before the container is going to be stopped, so Traefik is not routing new requests to the unhealthy container(s). This can be achieved by adding ! test -f /drain to container healthcheck, that is "fail if there exists a file named drain in /", and docker-rollout can create this file before stopping the old container.

I'm not sure if this behavior should be hardcoded in the tool, as there might be better ways of implementing request draining for proxies other than Traefik / nginx. Implementing hook support would allow docker-rollout users to implement true zero downtime deployment in two steps:

1. Add && ! test -f /drain to current container healtcheck in compose file
2. Add a hook like --before-stop "docker exec $1 touch /drain && sleep 10" to create the file manually

[rogerdz]

https://doc.traefik.io/traefik/middlewares/http/retry/ maybe this can help ?

[immortaly007]

I like the current idea, but wouldn't the "sleep" timout need to be at least the interval of the health check times the "retry" amount for the health check? Otherwise the container might not enter the "unhealthy" state before the shutdown sequence starts.

Maybe another option (specifically for traefik): in the --before-stop hook, add the label traefik.enable=false. (note that I have not tested how quickly traefik would pick up this change).

[wowu]

Thats a useful insight, thanks! We should describe the requirements for sleep in docs if we go for the hooks solution, or wait a required time in docker-rollout itself.

Docker container labels are immutable (at least for now: https://github.com/moby/moby/issues/21721), so we cannot use them to deregister the container from Traefik.