Unable to authenticate with the `/v1/metrics` endpoint

[radeklat]

Describe the bug

I am trying to configure prometheus 2.45.3 to scrape metrics made available from watchtower from the /v1/metrics endpoint. I ran into a dead end when prometheus keeps getting a 401 Unauthorized HTTP error.

Steps to reproduce

1. Run prometheus and watchtower from docker-compose
2. In the Targets section of the prometheus UI, observe the watchtower target to fail with a 401 HTTP error

Expected behavior

No authorization error, metrics are scraped.

Screenshots

No response

Environment

• Platform: Synology DSM420+
• Architecture: Linux
• Docker Version: 20.10.23, build 876964a
• Prometheus version: 2.45.3

Your logs

watchtower  | time="2024-02-28T16:34:01Z" level=debug msg="Sleeping for a second to ensure the docker api client has been properly initialized."
watchtower  | time="2024-02-28T16:34:02Z" level=debug msg="Making sure everything is sane before starting"
watchtower  | time="2024-02-28T16:34:02Z" level=debug msg="Retrieving running and restarting containers"
watchtower  | time="2024-02-28T16:34:02Z" level=debug msg="There are no additional watchtower containers"
watchtower  | time="2024-02-28T16:34:02Z" level=info msg="Watchtower 1.7.1"
watchtower  | time="2024-02-28T16:34:02Z" level=info msg="Using no notifications"
watchtower  | time="2024-02-28T16:34:02Z" level=info msg="Checking all containers (except explicitly disabled with label)"
watchtower  | time="2024-02-28T16:34:02Z" level=info msg="Scheduling first run: 2024-02-29 16:34:02 +0000 UTC"
watchtower  | time="2024-02-28T16:34:02Z" level=info msg="Note that the first check will be performed in 23 hours, 59 minutes, 59 seconds"
watchtower  | time="2024-02-28T16:34:02Z" level=warning msg="Trace level enabled: log will include sensitive information as credentials and tokens"

Additional context

I used the guide from https://containrrr.dev/watchtower/metrics/ to configure both watchtower and prometheus. However, it seems that bearer_token: demo_token is no longer supported in the latest LTS version of prometheus (2.45). I couldn't find any documentation of it. What is documented instead is:

authorization:
  type: Bearer
  credentials: "demo_token"

(Note that I'm using and actual long random alphanum string, not demo_token.)

I checked what prometheus actually sends as header with this configuration and it is authorization: Bearer demo_token. So that seem correct. I used wget from the prometheus container to simulate a request to watchtower with the same header:

docker-compose exec prometheus sh
wget -O - --header "authorization: Bearer demo_token" http://watchtower:8080/v1/metrics

and get the data back successfully. So I don't understand where this authorization error comes from. I tried to increase logging to trace level but I'm not seeing anything after start up.

Is there anything else I can try to see why the authorization fails? It would be great if I could disable the authorization althogether. Both prometheus and watchtower run on the same internal docker network, so the 8080 port doesn't even have to be exposed to the internet.

Here's how I'm running watchtower from docker-compose:

  watchtower:
    container_name: watchtower
    image: containrrr/watchtower
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
    environment:
      - WATCHTOWER_CLEANUP=true
      - WATCHTOWER_INCLUDE_RESTARTING=true
      - WATCHTOWER_HTTP_API_METRICS=true
      - WATCHTOWER_HTTP_API_TOKEN=demo_token
      - WATCHTOWER_TRACE=true
      - WATCHTOWER_LOG_LEVEL=trace
    restart: unless-stopped

And here is the prometheus config:

global:
  scrape_interval: 1m
  evaluation_interval: 1m

rule_files:
  - "/etc/prometheus/rules/*"

scrape_configs:
  - job_name: watchtower
    scrape_interval: 5s
    metrics_path: /v1/metrics
    authorization:
      type: Bearer
      credentials: "demo_token"
    static_configs:
      - targets: [ 'watchtower:8080' ]

[github-actions[bot]]

Hi there! 👋🏼 As you're new to this repo, we'd like to suggest that you read our code of conduct as well as our contribution guidelines. Thanks a bunch for opening your first issue! 🙏

[e7d]

Hello @radeklat,

I'm facing exactly the same issue. I fiddled in the same way, without any success. I'm not on a NAS, but on a Debian-based server using Docker 26.0.1 and Prometheus 2.48.0, but with the same results.

Did you find any way around that to make it work in the end?

[radeklat]

@e7d Unfortunately I didn't :( I'm not collecting any metrics at the moment.

[zkvvoob]

Hello,

I found this issue, because I, too, had a problem getting Watchtower metrics to Prometheus. After some trial and error, I managed to get them working together like this:

Watchtower docker-compose

WATCHTOWER_HTTP_API_TOKEN: 'watchtowertoken'
WATCHTOWER_HTTP_API_METRICS: 'true'

Prometheus config

scrape_configs:
  - job_name: watchtower
    metrics_path: /v1/metrics
    authorization:
      type: Bearer
      credentials: "watchtowertoken"
    static_configs:
      - targets: ['watchtower:8080']

I think what actually did the trick was changing the Watchtower API token once (after I had initially set it up). Read something along those lines in another issue.