watchtower icon indicating copy to clipboard operation
watchtower copied to clipboard

Incorrect docker version detection

Open abubaca4 opened this issue 1 month ago • 28 comments

Describe the bug

After updating Docker to 29.0.0, the container detects the API version as 1.25 and does not work. docker version displays the correct API version. System reboot not help.

Steps to reproduce

Run the container on the system specified in the environment. I think the Debian version specified should also have this issue.

Expected behavior

Just work normal.

Screenshots

No response

Environment

  • Openmediavault 7.7.20-1 (Sandworm) (Debian GNU/Linux 12 (bookworm))
  • x86_64
  • 29.0.0

Your logs

time="2025-11-11T04:00:52+03:00" level=debug msg="Sleeping for a second to ensure the docker api client has been properly initialized."
time="2025-11-11T04:00:53+03:00" level=debug msg="Making sure everything is sane before starting"
time="2025-11-11T04:00:53+03:00" level=debug msg="Retrieving running, stopped, restarting and exited containers"
time="2025-11-11T04:00:53+03:00" level=error msg="Error response from daemon: client version 1.25 is too old. Minimum supported API version is 1.44, please upgrade your client to a newer version"
time="2025-11-11T04:00:53+03:00" level=info msg="Waiting for the notification goroutine to finish" notify=no
time="2025-11-11T04:00:54+03:00" level=debug msg="Sleeping for a second to ensure the docker api client has been properly initialized."
time="2025-11-11T04:00:55+03:00" level=debug msg="Making sure everything is sane before starting"
time="2025-11-11T04:00:55+03:00" level=debug msg="Retrieving running, stopped, restarting and exited containers"
time="2025-11-11T04:00:55+03:00" level=error msg="Error response from daemon: client version 1.25 is too old. Minimum supported API version is 1.44, please upgrade your client to a newer version"
time="2025-11-11T04:00:55+03:00" level=info msg="Waiting for the notification goroutine to finish" notify=no
time="2025-11-11T04:00:55+03:00" level=debug msg="Sleeping for a second to ensure the docker api client has been properly initialized."
time="2025-11-11T04:00:56+03:00" level=debug msg="Making sure everything is sane before starting"
time="2025-11-11T04:00:56+03:00" level=debug msg="Retrieving running, stopped, restarting and exited containers"
time="2025-11-11T04:00:56+03:00" level=error msg="Error response from daemon: client version 1.25 is too old. Minimum supported API version is 1.44, please upgrade your client to a newer version"
time="2025-11-11T04:00:56+03:00" level=info msg="Waiting for the notification goroutine to finish" notify=no
time="2025-11-11T04:00:57+03:00" level=debug msg="Sleeping for a second to ensure the docker api client has been properly initialized."
time="2025-11-11T04:00:58+03:00" level=debug msg="Making sure everything is sane before starting"
time="2025-11-11T04:00:58+03:00" level=debug msg="Retrieving running, stopped, restarting and exited containers"
time="2025-11-11T04:00:58+03:00" level=error msg="Error response from daemon: client version 1.25 is too old. Minimum supported API version is 1.44, please upgrade your client to a newer version"
time="2025-11-11T04:00:58+03:00" level=info msg="Waiting for the notification goroutine to finish" notify=no
time="2025-11-11T04:00:59+03:00" level=debug msg="Sleeping for a second to ensure the docker api client has been properly initialized."

Additional context

No response

abubaca4 avatar Nov 11 '25 01:11 abubaca4

Hi there! 👋🏼 As you're new to this repo, we'd like to suggest that you read our code of conduct as well as our contribution guidelines. Thanks a bunch for opening your first issue! 🙏

github-actions[bot] avatar Nov 11 '25 01:11 github-actions[bot]

Same issue here

Ubuntu 24.04.3 LTS Docker version 29.0.0, build 3d4129b

zacharyshale95 avatar Nov 11 '25 01:11 zacharyshale95

Use this repo it works: https://github.com/nicholas-fedor/watchtower

Node815 avatar Nov 11 '25 01:11 Node815

Traefik has an identical issue (https://github.com/traefik/traefik/issues/12253): the OP there flagged it as caused by https://docs.docker.com/engine/deprecated/#deprecate-legacy-api-versions and there are workarounds posted for Ubuntu and Debian (tl;dr: downgrade docker-ce and related packages to v28)

Use this repo it works: https://github.com/nicholas-fedor/watchtower

Oh wow the last commit on this (containrrr/watchtower) repo is from 2 years ago :| gl;hf everyone?

cmrtdev avatar Nov 11 '25 01:11 cmrtdev

@piksel sorry to ping you directly but you're the last member of the containrrr project who committed to watchtower (in Dec. 2023). Any chance you know anything about the fate of the project?

Without an update to the Docker API, this version of watchtower will not work anymore.

cmrtdev avatar Nov 11 '25 01:11 cmrtdev

Traefik has an identical issue (traefik/traefik#12253): the OP there flagged it as caused by https://docs.docker.com/engine/deprecated/#deprecate-legacy-api-versions and there are workarounds posted for Ubuntu and Debian (tl;dr: downgrade docker-ce and related packages to v28)

Use this repo it works: https://github.com/nicholas-fedor/watchtower

Oh wow the last commit on this (containrrr/watchtower) repo is from 2 years ago :| gl;hf everyone?

Yes I am also seeing an issue on cAdvisor and others, seems to be a truly breaking change for a lot of things

zacharyshale95 avatar Nov 11 '25 01:11 zacharyshale95

Same error in Debian 12, but using the docker API version argument to set it to 1.44 seems to work (https://containrrr.dev/watchtower/arguments/#docker_api_version)

  Argument: --api-version, -a
Environment Variable: DOCKER_API_VERSION
                Type: String
             Default: "1.24"

But the default value should eventually be changed in the code. https://github.com/containrrr/watchtower/blob/76f9cea516593fabb8ca91ff13de55caa6aa0a8b/internal/flags/flags.go#L20

Kramoule avatar Nov 11 '25 01:11 Kramoule

I'm facing the same issue. I really like Watchtower. I hope I don't have to migrate to another tool (e.g. Renovate, WUD, or Diun).

ih8d8 avatar Nov 11 '25 01:11 ih8d8

Use this repo it works: nicholas-fedor/watchtower

Thanks ! 900 commits ahead !

Use this fork in your docker-compose.yaml:

image:  ghcr.io/nicholas-fedor/watchtower:latest

Write avatar Nov 11 '25 02:11 Write

Use this repo it works: https://github.com/nicholas-fedor/watchtower

Thanks, man! It works well, and the developer is actively working on it. Meanwhile, the containrrr version hasn't been updated for two years!

It's time to switch to this repository. Just replace the image: line in you compose from: image: containrrr/watchtower

to: image: nickfedor/watchtower

Feriman22 avatar Nov 11 '25 03:11 Feriman22

Traefik has an identical issue (traefik/traefik#12253): the OP there flagged it as caused by https://docs.docker.com/engine/deprecated/#deprecate-legacy-api-versions and there are workarounds posted for Ubuntu and Debian (tl;dr: downgrade docker-ce and related packages to v28)

Use this repo it works: https://github.com/nicholas-fedor/watchtower

Oh wow the last commit on this (containrrr/watchtower) repo is from 2 years ago :| gl;hf everyone?

thanks, downgrading worked, I think I'll pin my docker version to v28 and never upgrade lol..

ShlomiD83 avatar Nov 11 '25 06:11 ShlomiD83

Seems like a bad couple of weeks for homelab user. It was security fix on runc opencontainers/runc/issues/4968 and now legacy api deprecation on docker.

kdpuvvadi avatar Nov 11 '25 06:11 kdpuvvadi

Same issue here. Last watchtower commit was 2 years ago. How likely is an update?

BulldozerPete avatar Nov 11 '25 07:11 BulldozerPete

Temp fix for docker-compose.yaml

---
services:
  watchtower:
[…]
     environment:
       DOCKER_API_VERSION: 1.52

(check your API version with docker version)

kampfflunder avatar Nov 11 '25 07:11 kampfflunder

add environment fix it -e DOCKER_API_VERSION=1.51 \

Temp fix for docker-compose.yaml

---
services:
  watchtower:
[…]
     environment:
       DOCKER_API_VERSION: 1.52

(check your API version with docker version)

thanks! Using this environment temporarily fixed the problem. It's about time to find an alternative.

atlanticfirst avatar Nov 11 '25 09:11 atlanticfirst

add environment fix it -e DOCKER_API_VERSION=1.51 \

Temp fix for docker-compose.yaml

---
services:
  watchtower:
[…]
     environment:
       DOCKER_API_VERSION: 1.52

(check your API version with docker version)

thanks! Using this environment temporarily fixed the problem. It's about time to find an alternative.

Here is the alternative, which is working without this trick, and actively developed.

Feriman22 avatar Nov 11 '25 09:11 Feriman22

Found this: https://github.com/portainer/portainer/issues/12925#issuecomment-3516549977 Solve for me all issues. Switch to nick ether.

Wikibear avatar Nov 11 '25 14:11 Wikibear

If you wish to continue using the Containrrr version of Watchtower, then you will need to specify your Docker version (Containrrr Documentation).

As others have noted, my version works fine because I opted to integrate Docker's automatic version negotiation to avoid this issue altogether.

nicholas-fedor avatar Nov 11 '25 21:11 nicholas-fedor

The same here with AlmaLinux 9.6 Docker version 29.0.0, build 3d4129b and containrrr/watchtower:latest. I fixed it changing the repo from this image: containrrr/watchtower to this image: nickfedor/watchtower. Thanks @Node815!

mlopezcoria avatar Nov 12 '25 04:11 mlopezcoria

@nicholas-fedor Your fork seems to work fine. Thank you!

mboehm21 avatar Nov 12 '25 11:11 mboehm21

If you want to keep using the current Watchtower version, run this script manually or via crontab when convenient

#!/bin/bash

API_VERSION=$(docker version --format '{{.Server.APIVersion}}')

docker stop watchtower >/dev/null 2>&1

docker rm watchtower >/dev/null 2>&1

docker run -d --restart=always --name="watchtower" -v /var/run/docker.sock:/var/run/docker.sock containrrr/watchtower --api-version $(docker version --format '{{.Server.APIVersion}}')

bakunin1848 avatar Nov 12 '25 17:11 bakunin1848

Same error in Debian 12, but using the docker API version argument to set it to 1.44 seems to work (https://containrrr.dev/watchtower/arguments/#docker_api_version)

  Argument: --api-version, -a
Environment Variable: DOCKER_API_VERSION
                Type: String
             Default: "1.24"

But the default value should eventually be changed in the code.

watchtower/internal/flags/flags.go

Line 20 in 76f9cea

const DockerAPIMinVersion string = "1.25"

You're not the hero we deserve, but the hero we need.

MqlBql avatar Nov 16 '25 13:11 MqlBql

This docker.service edit did it here (Debian 12) without changing repo: https://github.com/portainer/portainer/issues/12925#issuecomment-3516549977

Image

memo-567 avatar Nov 17 '25 08:11 memo-567

This docker.service edit did it here (Debian 12) without changing repo: portainer/portainer#12925 (comment)

Image

It's great, but changing the Docker service affects all Docker containers. Meanwhile, changing the watchtower repository to this: https://github.com/nicholas-fedor/watchtower

is a better way. The Nick's version is actively developed, whereas the containrrr version has not been updated for over two years.

Feriman22 avatar Nov 17 '25 08:11 Feriman22

"... changing the Docker service affects all Docker containers."

I'm really not a developer and yeah - containrrr seems to be very, very old, but if I read the explanation on

https://www.docker.com/blog/docker-engine-version-29/

then that's probably the right / better way. ¯\_(ツ)_/¯

Edit: At least if you want to stick with the current bsky dev version.

memo-567 avatar Nov 17 '25 09:11 memo-567

At the end, I changed to image: nick fedor/watchtower too. Didn't notice containrrr/watchtower isn't made by bsky devs. 🙄

memo-567 avatar Nov 17 '25 11:11 memo-567

I think we are going to go ahead and switch to this fork for the PDS image — thanks for the discussion, everybody!

axfelix avatar Nov 17 '25 23:11 axfelix

docker run --rm
-v /var/run/docker.sock:/var/run/docker.sock
containrrr/watchtower --run-once --api-version 1.44

BasRuggeberg avatar Dec 09 '25 10:12 BasRuggeberg

Same ! need a update for watchtower ! :(

NoisyBoyFR avatar Dec 13 '25 01:12 NoisyBoyFR

you need to check your docker api version by this command

docker version --format '{{.Server.APIVersion}}'

then in watchtower docker compose add this env variable :

- DOCKER_API_VERSION=1.52 #(use the version you got from the command )

MotazHakim avatar Dec 15 '25 16:12 MotazHakim