watchtower icon indicating copy to clipboard operation
watchtower copied to clipboard
verified publisher icon containrrr

WatchTower upadate image used by x container

Open rFond opened this issue 2 years ago • 8 comments

Is your feature request related to a problem? Please describe.

Hello,

I have 1 image docker used with 2 container (for my exemple).

My docker compose :

  mcdLeds:
    image: clauger3e.azurecr.io/mcd/leds:dev-latest
    container_name: mcdleds
    restart: always
    volumes:
      - "/etc/timezone:/etc/timezone:ro"
      - "/etc/localtime:/etc/localtime:ro"
      - "/sys/class/leds/user-led1-green/brightness:/sys/class/leds/user-led1-green/brightness"
      - "/sys/class/leds/user-led2-green/brightness:/sys/class/leds/user-led2-green/brightness"
      - "/sys/class/leds/user-led1-red/brightness:/sys/class/leds/user-led1-red/brightness"
      - "/sys/class/leds/user-led2-red/brightness:/sys/class/leds/user-led2-red/brightness"
    env_file: ./MyClaugerDetect_Leds/.env
    networks:
      vpcbr_leds:

  mcdLeds2:
    image: clauger3e.azurecr.io/mcd/leds:dev-latest
    container_name: mcdleds2
    restart: always
    volumes:
      - "/etc/timezone:/etc/timezone:ro"
      - "/etc/localtime:/etc/localtime:ro"
      - "/sys/class/leds/user-led1-green/brightness:/sys/class/leds/user-led1-green/brightness"
      - "/sys/class/leds/user-led2-green/brightness:/sys/class/leds/user-led2-green/brightness"
      - "/sys/class/leds/user-led1-red/brightness:/sys/class/leds/user-led1-red/brightness"
      - "/sys/class/leds/user-led2-red/brightness:/sys/class/leds/user-led2-red/brightness"
    env_file: ./MyClaugerDetect_Leds/.env
    networks:
      vpcbr_leds2:

Watchtower update only one.

time="2023-04-26T09:22:59+02:00" level=info msg="Found new clauger3e.azurecr.io/mcd/leds:dev-latest image (1b1b444cf68a)"
time="2023-04-26T09:31:01+02:00" level=info msg="Stopping /mcdleds (8dbec57bf10a) with SIGTERM"
time="2023-04-26T09:31:12+02:00" level=info msg="Creating /mcdleds"
time="2023-04-26T09:31:13+02:00" level=info msg="Removing image b2a6457d44cb"

Describe the solution you'd like

I would like if possible, (with option in conf). WatchTower check all container, stop, restart 2 or more same processus.

Describe alternatives you've considered

Use another image ?

Additional context

No response

rFond avatar Apr 26 '23 07:04 rFond

Hi there! 👋🏼 As you're new to this repo, we'd like to suggest that you read our code of conduct as well as our contribution guidelines. Thanks a bunch for opening your first issue! 🙏

github-actions[bot] avatar Apr 26 '23 07:04 github-actions[bot]

Hmm, that should be the normal behaviour of watchtower... Not sure how it misses the second one. How are you running watchtower, and what does the report say after a session (the number of checked/updated/skipped containers)?

piksel avatar Apr 26 '23 09:04 piksel

I m in last version 1.5.3, Watchtower is dockerised.

This is configuration of my WatchTower :

  watchtower:
    container_name: watchtower
    hostname: watchtower
    image: containrrr/watchtower:1.5.3
    restart: always
    environment:
      TZ: "Europe/Paris"
      WATCHTOWER_CLEANUP: "true"
      WATCHTOWER_POLL_INTERVAL: 86400
      WATCHTOWER_INCLUDE_STOPPED: "true"
      WATCHTOWER_REVIVE_STOPPED: "true"
      WATCHTOWER_ROLLING_RESTART: "true"
      WATCHTOWER_INCLUDE_RESTARTING: "true"
      WATCHTOWER_REVIVE_STOPPED: "true"
      WATCHTOWER_SCHEDULE : "0 * * * * *"
      #WATCHTOWER_RUN_ONCE: "true"
      WATCHTOWER_HTTP_API_UPDATE: "true"
      WATCHTOWER_HTTP_API_TOKEN: "xxxxxxxxxxx"
      WATCHTOWER_HTTP_API_PERIODIC_POLLS: "true"
    volumes:
      - /root/.docker/config.json:/config.json
      - /var/run/docker.sock:/var/run/docker.sock
    networks:
      vpcbr_external:
        ipv4_address: 172.18.0.7
      vpcbr_watchtower:

all message :

time="2023-04-26T09:22:04+02:00" level=info msg="Updates triggered by HTTP API request."
time="2023-04-26T09:22:59+02:00" level=info msg="Found new clauger3e.azurecr.io/mcd/leds:dev-latest image (1b1b444cf68a)"
time="2023-04-26T09:31:01+02:00" level=info msg="Stopping /mcdleds (8dbec57bf10a) with SIGTERM"
time="2023-04-26T09:31:12+02:00" level=info msg="Creating /mcdleds"
time="2023-04-26T09:31:13+02:00" level=info msg="Removing image b2a6457d44cb"
time="2023-04-26T09:31:13+02:00" level=error msg="Error response from daemon: conflict: unable to delete b2a6457d44cb (cannot be forced) - image is being used by running container 0d7ca5478a47"
time="2023-04-26T09:31:13+02:00" level=info msg="Session done" Failed=0 Scanned=19 Updated=1 notify=no

rFond avatar Apr 26 '23 09:04 rFond 

time="2023-04-26T09:31:13+02:00" level=error msg="Error response from daemon: conflict: unable to delete b2a6457d44cb (cannot be forced) - image is being used by running container 0d7ca5478a47"

This seems like it could be an issue. It should probably recreate both before trying to remove the image. This could very well be the source of the issues you are experiencing.

simskij avatar Jul 24 '23 17:07 simskij

I think the watchtower should do:

  • Step 1 : Check repos and image différence
  • Step 2 : Download new image
  • Step 3 : Check all docker container with image used
  • Step 4 : Stop and remove all containers
  • Step 5 : Restart all containers with a new image
  • Step 6 : Delete old image

rFond avatar Jul 26 '23 16:07 rFond

Well, it seems like it just ignores the second container for some reason, and then when it tries to remove the old image for container 1 it can't since it's still in use by container 2.

Watchtower is actually doing it the way you describe above (although it checks every container's image for updates, not checking all images and then containers).

You could try running with:

 watchtower:
   command: "mcleds2?"
   environment:
     WATCHTOWER_DEBUG: "true"

It should show why it's skipping it...

piksel avatar Jul 27 '23 09:07 piksel

With logs :


time="2023-08-17T16:53:32+02:00" level=debug msg="Trying to load authentication credentials." container=/mcdleds image="*********************/mcd/leds:dev-latest"
time="2023-08-17T16:53:32+02:00" level=debug msg="Loaded auth credentials for user MCDDev, on registry *********************/mcd/leds:dev-latest, from file /config.json"
time="2023-08-17T16:53:32+02:00" level=debug msg="Got image name: *********************/mcd/leds:dev-latest"
time="2023-08-17T16:53:32+02:00" level=debug msg="Credentials loaded"
time="2023-08-17T16:53:32+02:00" level=debug msg="Checking if pull is needed" container=/mcdleds image="*********************/mcd/leds:dev-latest"
time="2023-08-17T16:53:32+02:00" level=debug msg="Building challenge URL" URL="https://*********************/v2/"
time="2023-08-17T16:53:32+02:00" level=debug msg="Got response to challenge request" header="Bearer realm=\"https://*********************/oauth2/token\",service=\"*********************\"" status="401 Unauthorized"
time="2023-08-17T16:53:32+02:00" level=debug msg="Checking challenge header content" realm="https://*********************/oauth2/token" service=*********************
time="2023-08-17T16:53:32+02:00" level=debug msg="Setting scope for auth token" image=*********************/mcd/leds scope="repository:*********************/mcd/leds:pull"
time="2023-08-17T16:53:32+02:00" level=debug msg="Credentials found."
time="2023-08-17T16:53:33+02:00" level=debug msg="Parsing image ref" host=********************* image=mcd/leds normalized="*********************/mcd/leds:dev-latest" tag=dev-latest
time="2023-08-17T16:53:33+02:00" level=debug msg="Doing a HEAD request to fetch a digest" url="https://*********************/v2/mcd/leds/manifests/dev-latest"
time="2023-08-17T16:53:33+02:00" level=debug msg="Could not do a head request for \"*********************/mcd/leds:dev-latest\", falling back to regular pull." container=/mcdleds image="*********************/mcd/leds:dev-latest"
time="2023-08-17T16:53:33+02:00" level=debug msg="Reason: registry responded to head request with \"401 Unauthorized\", auth: \"Bearer realm=\\\"https://*********************/oauth2/token\\\",service=\\\"*********************\\\",scope=\\\"repository:mcd/leds:pull\\\"\"" container=/mcdleds image="*********************/mcd/leds:dev-latest"
time="2023-08-17T16:53:33+02:00" level=debug msg="Pulling image" container=/mcdleds image="*********************/mcd/leds:dev-latest"
time="2023-08-17T16:53:34+02:00" level=info msg="Found new *********************/mcd/leds:dev-latest image (9a7b0432f3a3)"
time="2023-08-17T16:53:34+02:00" level=debug msg="Trying to load authentication credentials." container=/mcdleds2 image="*********************/mcd/leds:dev-latest"
time="2023-08-17T16:53:34+02:00" level=debug msg="Loaded auth credentials for user MCDDev, on registry *********************/mcd/leds:dev-latest, from file /config.json"
time="2023-08-17T16:53:34+02:00" level=debug msg="Got image name: *********************/mcd/leds:dev-latest"
time="2023-08-17T16:53:34+02:00" level=debug msg="Credentials loaded"
time="2023-08-17T16:53:34+02:00" level=debug msg="Checking if pull is needed" container=/mcdleds2 image="*********************/mcd/leds:dev-latest"
time="2023-08-17T16:53:34+02:00" level=debug msg="Building challenge URL" URL="https://*********************/v2/"
time="2023-08-17T16:53:34+02:00" level=debug msg="Got response to challenge request" header="Bearer realm=\"https://*********************/oauth2/token\",service=\"*********************\"" status="401 Unauthorized"
time="2023-08-17T16:53:34+02:00" level=debug msg="Checking challenge header content" realm="https://*********************/oauth2/token" service=*********************
time="2023-08-17T16:53:34+02:00" level=debug msg="Setting scope for auth token" image=*********************/mcd/leds scope="repository:*********************/mcd/leds:pull"
time="2023-08-17T16:53:34+02:00" level=debug msg="Credentials found."
time="2023-08-17T16:53:35+02:00" level=debug msg="Parsing image ref" host=********************* image=mcd/leds normalized="*********************/mcd/leds:dev-latest" tag=dev-latest
time="2023-08-17T16:53:35+02:00" level=debug msg="Doing a HEAD request to fetch a digest" url="https://*********************/v2/mcd/leds/manifests/dev-latest"
time="2023-08-17T16:53:35+02:00" level=debug msg="Could not do a head request for \"*********************/mcd/leds:dev-latest\", falling back to regular pull." container=/mcdleds2 image="*********************/mcd/leds:dev-latest"
time="2023-08-17T16:53:35+02:00" level=debug msg="Reason: registry responded to head request with \"401 Unauthorized\", auth: \"Bearer realm=\\\"https://*********************/oauth2/token\\\",service=\\\"*********************\\\",scope=\\\"repository:mcd/leds:pull\\\"\"" container=/mcdleds2 image="*********************/mcd/leds:dev-latest"
time="2023-08-17T16:53:35+02:00" level=debug msg="Pulling image" container=/mcdleds2 image="*********************/mcd/leds:dev-latest"
time="2023-08-17T16:53:35+02:00" level=info msg="Found new *********************/mcd/leds:dev-latest image (9a7b0432f3a3)"
time="2023-08-17T16:53:47+02:00" level=info msg="Stopping /mcdleds2 (69a695e1cdf3) with SIGTERM"
time="2023-08-17T16:53:57+02:00" level=debug msg="Removing container 69a695e1cdf3"
time="2023-08-17T16:53:57+02:00" level=info msg="Creating /mcdleds2"
time="2023-08-17T16:53:57+02:00" level=debug msg="Starting container /mcdleds2 (d516d9f41044)"
time="2023-08-17T16:53:58+02:00" level=info msg="Stopping /mcdleds (56c76d924571) with SIGTERM"
time="2023-08-17T16:54:09+02:00" level=debug msg="Removing container 56c76d924571"
time="2023-08-17T16:54:09+02:00" level=info msg="Creating /mcdleds"
time="2023-08-17T16:54:09+02:00" level=debug msg="Starting container /mcdleds (ee2194bfb287)"
time="2023-08-17T16:54:11+02:00" level=info msg="Removing image e689624463e6"
time="2023-08-17T16:54:11+02:00" level=info msg="Session done" Failed=0 Scanned=16 Updated=2 notify=no

No problem this time.

rFond avatar Aug 17 '23 15:08 rFond

Maybe the second container was stopped in the first run? You can include stopped containers using --include-stopped.

Otherwise, we can't do much without logs from when it didn't work. 🤷‍♀️

Glad to see that it is working for you now, though!

piksel avatar Aug 18 '23 11:08 piksel