watchtower Could not do a head request: invalid character

Describe the bug

Issues with the crowdsec image.

  crowdsec:
    image: crowdsecurity/crowdsec:latest
    container_name: Crowdsec

Error:

Could not do a head request for "crowdsecurity/crowdsec:latest", falling back to regular pull.
Reason: invalid character 'S' looking for beginning of value

Steps to reproduce

Watchtower config:

  watchtower:
    image: containrrr/watchtower
    container_name: Watchtower
    hostname: Watchtower
    networks:
      - isonet
    labels: 
      - autoheal=true
    environment:
      TZ: $TZ
      WATCHTOWER_CLEANUP: "true"
      WATCHTOWER_REMOVE_VOLUMES: "false"
      WATCHTOWER_INCLUDE_STOPPED: "true"
      WATCHTOWER_NO_STARTUP_MESSAGE: "true"
      WATCHTOWER_TIMEOUT: 30s
      WATCHTOWER_SCHEDULE: "0 0 6 * * *" # Everyday at 04:30
      WATCHTOWER_NOTIFICATIONS: shoutrrr
      WATCHTOWER_NOTIFICATION_URL: "telegram://$TGRAM_BOT_TOKEN@telegram?channels=$TGRAM_CHAT_ID"
      WATCHTOWER_NOTIFICATIONS_LEVEL: info
      DOCKER_HOST: unix:///var/run/docker.sock
    volumes:
     - /var/run/docker.sock:/var/run/docker.sock
    logging:
      driver: "local"
      options:
        max-size: 10m
        max-file: "3"
    restart: always
    mem_limit: 250m
    mem_reservation: 60m

Expected behavior

To update.

Screenshots

Environment

Ubuntu 22.04
x86
Docker version 20.10.22, build 3a2c30b
Crowdsec

Your logs

time="2022-12-26T06:00:35Z" level=info msg="Session done" Failed=0 Scanned=43 Updated=0 notify=no
time="2022-12-27T06:00:39Z" level=info msg="Session done" Failed=0 Scanned=43 Updated=0 notify=no
time="2022-12-28T06:00:04Z" level=warning msg="Could not do a head request for \"crowdsecurity/crowdsec:latest\", falling back to regular pull." container=/Crowdsec image="crowdsecurity/crowdsec:latest"
time="2022-12-28T06:00:04Z" level=warning msg="Reason: invalid character 'S' looking for beginning of value" container=/Crowdsec image="crowdsecurity/crowdsec:latest"
time="2022-12-28T06:00:15Z" level=info msg="Found new linuxserver/bookstack:latest image (f532d54a179c)"
time="2022-12-28T06:00:23Z" level=info msg="Found new linuxserver/sonarr:latest image (10d61a66d74e)"
time="2022-12-28T06:00:51Z" level=info msg="Found new linuxserver/netbootxyz:latest image (2d07183b1077)"
time="2022-12-28T06:00:53Z" level=info msg="Stopping /Netbootxyz (14c6cda3588b) with SIGTERM"
time="2022-12-28T06:01:34Z" level=info msg="Stopping /Sonarr (d19f614e614d) with SIGTERM"
time="2022-12-28T06:01:45Z" level=info msg="Stopping /Bookstack (7a1565f59d9e) with SIGTERM"
time="2022-12-28T06:01:51Z" level=info msg="Creating /Bookstack"
time="2022-12-28T06:01:52Z" level=info msg="Creating /Sonarr"
time="2022-12-28T06:01:53Z" level=info msg="Creating /Netbootxyz"
time="2022-12-28T06:01:59Z" level=info msg="Removing image 455a86d2b778"
time="2022-12-28T06:01:59Z" level=info msg="Removing image f4e288f594a6"
time="2022-12-28T06:02:00Z" level=info msg="Removing image 4866c5508f26"
time="2022-12-28T06:02:00Z" level=info msg="Session done" Failed=0 Scanned=43 Updated=3 notify=no

Additional context

Unsure if related to https://github.com/containrrr/watchtower/issues/1050

Doing a manual run does not trigger any issues.

docker run --rm \
    -v /var/run/docker.sock:/var/run/docker.sock \
    containrrr/watchtower \
    --debug --run-once \
    Crowdsec
time="2022-12-29T05:56:32Z" level=debug msg="Sleeping for a second to ensure the docker api client has been properly initialized."
time="2022-12-29T05:56:33Z" level=debug msg="Making sure everything is sane before starting"
time="2022-12-29T05:56:33Z" level=info msg="Watchtower 1.5.1"
time="2022-12-29T05:56:33Z" level=info msg="Using no notifications"
time="2022-12-29T05:56:33Z" level=info msg="Only checking containers which name matches \"Crowdsec\""
time="2022-12-29T05:56:33Z" level=info msg="Running a one time update."
time="2022-12-29T05:56:33Z" level=debug msg="Checking containers for updated images"
time="2022-12-29T05:56:33Z" level=debug msg="Retrieving running containers"
time="2022-12-29T05:56:35Z" level=debug msg="Trying to load authentication credentials." container=/Crowdsec image="crowdsecurity/crowdsec:latest"
time="2022-12-29T05:56:35Z" level=debug msg="No credentials for crowdsecurity found" config_file=/config.json
time="2022-12-29T05:56:35Z" level=debug msg="Got image name: crowdsecurity/crowdsec:latest"
time="2022-12-29T05:56:35Z" level=debug msg="Checking if pull is needed" container=/Crowdsec image="crowdsecurity/crowdsec:latest"
time="2022-12-29T05:56:35Z" level=debug msg="Building challenge URL" URL="https://index.docker.io/v2/"
time="2022-12-29T05:56:35Z" level=debug msg="Got response to challenge request" header="Bearer realm=\"https://auth.docker.io/token\",service=\"registry.docker.io\"" status="401 Unauthorized"
time="2022-12-29T05:56:35Z" level=debug msg="Checking challenge header content" realm="https://auth.docker.io/token" service=registry.docker.io
time="2022-12-29T05:56:35Z" level=debug msg="Setting scope for auth token" image=crowdsecurity/crowdsec scope="repository:crowdsecurity/crowdsec:pull"
time="2022-12-29T05:56:35Z" level=debug msg="No credentials found."
time="2022-12-29T05:56:36Z" level=debug msg="Parsing image ref" host=index.docker.io image=crowdsecurity/crowdsec normalized="docker.io/crowdsecurity/crowdsec:latest" tag=latest
time="2022-12-29T05:56:36Z" level=debug msg="Doing a HEAD request to fetch a digest" url="https://index.docker.io/v2/crowdsecurity/crowdsec/manifests/latest"
time="2022-12-29T05:56:36Z" level=debug msg="Found a remote digest to compare with" remote="sha256:edd3e6aca6c7b0c99f2c542b502c70cebd94a5b867084dd24de041cbe48bbc66"
time="2022-12-29T05:56:36Z" level=debug msg=Comparing local="sha256:edd3e6aca6c7b0c99f2c542b502c70cebd94a5b867084dd24de041cbe48bbc66" remote="sha256:edd3e6aca6c7b0c99f2c542b502c70cebd94a5b867084dd24de041cbe48bbc66"
time="2022-12-29T05:56:36Z" level=debug msg="Found a match"
time="2022-12-29T05:56:36Z" level=debug msg="No pull needed. Skipping image."
time="2022-12-29T05:56:36Z" level=debug msg="No new images found for /Crowdsec"
time="2022-12-29T05:56:36Z" level=info msg="Session done" Failed=0 Scanned=1 Updated=0 notify=no
time="2022-12-29T05:56:36Z" level=info msg="Waiting for the notification goroutine to finish" notify=no

Dec 29 '22 05:12 modem7

Seems like there was some kind of temporary network error. During your first session, it only warned you that it could not perform a "cheap" check for updates, falling back to a normal pull.

Whatever error message the server responded with is probably now gone and it works normally.

Dec 29 '22 06:12 piksel

I'm having the same issue on Raspberry Pi 4 (AARCH64). I couldn't revert back to docker-compose 1.28.2, as no binary for aarch64 existed. It looks like there is a problem with docker-compose v.2.

Dec 30 '22 16:12 UniverseM5

I have exactly the same issue. Occasionally I get this error-message for alternating images (not the same each time). I have the impression that this message has been occurring since the last Watchtower update.

Jan 04 '23 08:01 hoorna

@UniverseXXX and @hoorna what part do you mean is the same? Do you also get the invalid character 'S' looking for beginning of value reason?

I don't see how docker-compose version is related to this issue. Nor should there be anything in the last release that alters this behaviour...

Jan 04 '23 17:01 piksel

@piksel, I also get the "invalid character 'S' looking for beginning of value".

Today I got for example an email message with the following text: "Could not do a head request for "containrrr/watchtower:latest", falling back to regular pull. Reason: invalid character 'S' looking for beginning of value"

Before the latest Watchtower update I never got such a message.

Jan 04 '23 18:01 hoorna

The message is a bit cryptic, but what it means is that the repository responds with something that is not valid JSON. Instead of starting with {, it starts with S, which is probably a proxy error. Watchtower still falls back to doing a regular pull though, so I'm not sure what else we can do...

Jan 04 '23 20:01 piksel

The message is a bit cryptic, but what it means is that the repository responds with something that is not valid JSON. Instead of starting with {, it starts with S, which is probably a proxy error. Watchtower still falls back to doing a regular pull though, so I'm not sure what else we can do...

If the message is cryptic and making it difficult to troubleshoot, I wonder if it would be worthwhile investigating a way to make that particular error more verbose (but not run the entire container verbose as that might just cause noise, even more so if it's an intermittent issue (seems to be in my case))?

Jan 04 '23 20:01 modem7

I'm not sure if related, but for the past 2 weeks I've been received the following notification (never had an issue previously)

Could not do a head request for "nicolargo/glances:latest-full", falling back to regular pull.
Reason: registry responded to head request with "404 Not Found", auth: "not present"

Sometimes there is then a subsequent message, e.g.,

Found new nicolargo/glances:latest-full image (ce3e6aaf669f)

Everything seems to be present OK on Docker Hub - https://hub.docker.com/r/nicolargo/glances/tags I've also not noticed any issues with more generally with updates.

Jan 19 '23 04:01 EDIflyer

I'm not sure if related, but for the past 2 weeks I've been received the following notification (never had an issue previously)
Could not do a head request for "nicolargo/glances:latest-full", falling back to regular pull.
Reason: registry responded to head request with "404 Not Found", auth: "not present"
Sometimes there is then a subsequent message, e.g.,
Found new nicolargo/glances:latest-full image (ce3e6aaf669f)
Everything seems to be present OK on Docker Hub - https://hub.docker.com/r/nicolargo/glances/tags I've also not noticed any issues with more generally with updates.

Same for me with the image koenkk/zigbee2mqtt:latest.

The log says:

time="2023-01-13T23:53:08Z" level=warning msg="Could not do a head request for \"koenkk/zigbee2mqtt:latest\", falling back to regular pull." container=/Zigbee2MQTT image="koenkk/zigbee2mqtt:latest" 
time="2023-01-13T23:53:08Z" level=warning msg="Reason: registry responded to head request with \"404 Not Found\", auth: \"not present\"" container=/Zigbee2MQTT image="koenkk/zigbee2mqtt:latest"

Does anyone know the reason why this is happening? Is it a problem from dockerhub, is there something wrong with the published images, or is it a problem with watchtower?

Jan 19 '23 08:01 ochstobi

I'm not sure if related, but for the past 2 weeks I've been received the following notification (never had an issue previously)
Could not do a head request for "nicolargo/glances:latest-full", falling back to regular pull.
Reason: registry responded to head request with "404 Not Found", auth: "not present"
Sometimes there is then a subsequent message, e.g.,
Found new nicolargo/glances:latest-full image (ce3e6aaf669f)
Everything seems to be present OK on Docker Hub - https://hub.docker.com/r/nicolargo/glances/tags I've also not noticed any issues with more generally with updates.

I'm having the exact same issue.

My other machines are still running the old v2tec/watchtower and are not having this issue so it's something with this fork.

Jan 19 '23 18:01 LeeThompson

I am also seeing this issue for crowdsec, started within last few days

Jan 19 '23 22:01 dreary-ennui

Hmm OK I'm now seeing this on other images too and on another server...

Could not do a head request for "nicolargo/glances:latest-full", falling back to regular pull.
Reason: registry responded to head request with "404 Not Found", auth: "not present"
Found new squidfunk/mkdocs-material:latest image (5d27b1bf0c29)
Could not do a head request for "docker.io/amir20/dozzle:latest", falling back to regular pull.
Reason: registry responded to head request with "404 Not Found", auth: "not present"
Found new docker.io/amir20/dozzle:latest image (d2ba0922ba56)
Stopping /dozzle (8c68ed1a0222) with SIGTERM
Creating /mkdocs-checkforupdates
Creating /dozzle
Removing image 4d8ef68aa252
Removing image effbbffdde8c

Jan 20 '23 00:01 EDIflyer

Same for me with the koenkk/zigbee2mqtt:latest

Jan 20 '23 13:01 olylak

I'm also receivng a 404 on the images matrixdotorg/dendrite-monolith:latest and pihole/pihole:latest

time="2023-01-20T18:06:32Z" level=warning msg="Could not do a head request for \"pihole/pihole:latest\", falling back to regular pull." container=/pihole image="pihole/pihole:latest"
time="2023-01-20T18:06:32Z" level=warning msg="Reason: registry responded to head request with \"404 Not Found\", auth: \"not present\"" container=/pihole image="pihole/pihole:latest"
time="2023-01-20T18:06:34Z" level=warning msg="Could not do a head request for \"matrixdotorg/dendrite-monolith:latest\", falling back to regular pull." container=/dendrite-monolith-1 image="matrixdotorg/dendrite-monolith:latest"
time="2023-01-20T18:06:34Z" level=warning msg="Reason: registry responded to head request with \"404 Not Found\", auth: \"not present\"" container=/dendrite-monolith-1 image="matrixdotorg/dendrite-monolith:latest"
time="2023-01-20T18:06:39Z" level=info msg="Session done" Failed=0 Scanned=10 Updated=0 notify=no

Jan 20 '23 16:01 WilsontheWolf

Plenty of other repos are also broken ....

netdata/netdata:latest amir20/dozzle:latest shlinkio/shlink:latest zabbix/zabbix-*:latest

Could not do a head request for "zabbix/zabbix-agent:alpine-latest", falling back to regular pull.
Reason: registry responded to head request with "404 Not Found", auth: "not present"
Could not do a head request for "zabbix/zabbix-server-mysql:alpine-latest", falling back to regular pull.
Reason: registry responded to head request with "404 Not Found", auth: "not present"
Could not do a head request for "zabbix/zabbix-web-nginx-mysql:alpine-latest", falling back to regular pull.
Reason: registry responded to head request with "404 Not Found", auth: "not present"
Could not do a head request for "amir20/dozzle:latest", falling back to regular pull.
Reason: registry responded to head request with "404 Not Found", auth: "not present"
Could not do a head request for "shlinkio/shlink:latest", falling back to regular pull.
Reason: registry responded to head request with "404 Not Found", auth: "not present"

Jan 20 '23 16:01 xpliz

This happens for me too when I try to update docker images private from ghcr.io.

Watchtower updates on <myserver>
Could not do a head request for "ghcr.io/georg-jung/<myrepo>:latest", falling back to regular pull.
Reason: registry responded to head request with "404 Not Found", auth: "not present"

Jan 20 '23 18:01 georg-jung

Plenty of other repos are also broken ....

netdata/netdata:latest amir20/dozzle:latest shlinkio/shlink:latest zabbix/zabbix-*:latest

Could not do a head request for "zabbix/zabbix-agent:alpine-latest", falling back to regular pull.
Reason: registry responded to head request with "404 Not Found", auth: "not present"
Could not do a head request for "zabbix/zabbix-server-mysql:alpine-latest", falling back to regular pull.
Reason: registry responded to head request with "404 Not Found", auth: "not present"
Could not do a head request for "zabbix/zabbix-web-nginx-mysql:alpine-latest", falling back to regular pull.
Reason: registry responded to head request with "404 Not Found", auth: "not present"
Could not do a head request for "amir20/dozzle:latest", falling back to regular pull.
Reason: registry responded to head request with "404 Not Found", auth: "not present"
Could not do a head request for "shlinkio/shlink:latest", falling back to regular pull.
Reason: registry responded to head request with "404 Not Found", auth: "not present"

Pretty sure it's not the repos at this point.

Jan 20 '23 18:01 modem7

Pretty sure it's not the repos at this point.

I know it's not repos, docker pull works just fine on them.

But there can be something different on that repos since watchtower has problems with some and not with others repos. Maybe something in underlying API changed dunno.

Jan 20 '23 20:01 xpliz

Pretty sure it's not the repos at this point.

I know it's not repos, docker pull works just fine on them.

But there can be something different on that repos since watchtower has problems with some and not with others repos. Maybe something in underlying API changed dunno.

For me, it's intermittent, and on different repos. Not always the same one.

Jan 20 '23 20:01 modem7

Since #1528 was opened for the 404 errors, let's use it for those errors, since it seems to be something different from the original issue here.

Jan 20 '23 21:01 piksel

Same here :( With homarr

Jan 23 '23 00:01 vilmotz

This happens for me too when I try to update docker images private from ghcr.io.

Watchtower updates on <myserver>
Could not do a head request for "ghcr.io/georg-jung/<myrepo>:latest", falling back to regular pull.
Reason: registry responded to head request with "404 Not Found", auth: "not present"

If you are building your own docker images using GitHub Actions and experience this issue, like I did, this solution might work for you as a workaround:

  - name: Build and push
    uses: docker/build-push-action@v3
    with:
      push: true
+     provenance: false
      tags: user/app:latest

Jan 26 '23 11:01 georg-jung

Since #1528 was opened for the 404 errors, let's use it for those errors, since it seems to be something different from the original issue here.

@piksel Should #1528 not be re-opened then? (the issue is closed as "completed" but it's clearly still occurring.)

Jan 29 '23 06:01 notDavid

@notDavid it's closed because the issue is fixed in our code base. If you want to test it before it's released you can use the containrrr/watchtower:latest-dev, but since it's bleeding edge, it's not recommended for daily use.

Jan 29 '23 10:01 piksel

I still get this error

Notifications: Container errors Streams: Vector - Containers --- [Event Definition] --- Title: Container errors Type: aggregation-v1 Timestamp: 2023-10-12T23:00:08.000Z

--- [Log] --- Source: docker_logs Container: Watchtower Level: warning Message: Message: Could not do a head request for
Message Link: https://10.10.40.190:9000/messages/vector_7/17144332-6953-11ee-819f-0242ac170004

Container errors Streams: Vector - Containers --- [Event Definition] --- Title: Container errors Type: aggregation-v1 Timestamp: 2023-10-12T23:00:08.000Z

--- [Log] --- Source: docker_logs Container: Watchtower Level: warning Message: Message: Reason: invalid character 'S' looking for beginning of value Message Link: https://10.10.40.190:9000/messages/vector_7/17146a40-6953-11ee-819f-0242ac170004

Oct 12 '23 23:10 lordraiden

@lordraiden It's not an error, but rather a warning. If you don't care about the warning you can turn it off. It means that watchtower gets an invalid response from the repository and can't perform a cheap change check and falls back to a regular pull.

This issue go hijacked a bit for the occurring 404 errors (which was due to a change in docker buildx). I'll reopen this and clean it from unrelated comments.

Oct 13 '23 06:10 piksel

The message is a bit cryptic, but what it means is that the repository responds with something that is not valid JSON. Instead of starting with {, it starts with S, which is probably a proxy error. Watchtower still falls back to doing a regular pull though, so I'm not sure what else we can do...

If the message is cryptic and making it difficult to troubleshoot, I wonder if it would be worthwhile investigating a way to make that particular error more verbose (but not run the entire container verbose as that might just cause noise, even more so if it's an intermittent issue (seems to be in my case))?

The response should be logged in full to aid in further investigation. Reopening this.

Oct 13 '23 06:10 piksel