docker image does not recognise timezone appropriately

[instantdreams]

What did you do? My compose.yaml file looks like this:

services:
  alertmanager:
    image: quay.io/prometheus/alertmanager:latest
    container_name: alertmanager
    hostname: alertmanager
    privileged: true
    command: --config.file=/etc/alertmanager/alertmanager.yaml --log.level=info
    ports:
      - 9093:9093 # web ui
    environment:
      - TZ=America/Edmonton
      - BOT_TOKEN=${BOT_TOKEN}
      - CHAT_ID=${CHAT_ID}
    volumes:
      - ${DIRECTORY_DATA}:/data
      - ${DIRECTORY_ETCALERTMANAGER}:/etc/alertmanager
      - /etc/timezone:/etc/timezone:ro
      - /etc/localtime:/etc/localtime:ro
    restart: unless-stopped

What did you expect to see? The time in the container to represent the timezone passed by the compose file.

What did you see instead? Under which circumstances? Using the following commands:

$ docker exec alertmanager date
Mon Apr 15 17:00:34 UTC 2024

$ docker exec alertmanager cat /etc/timezone
America/Edmonton

The timezone is available to the container but is not being recognised.

Environment

• System information:

  Linux 6.1.0-18-amd64 x86_64

• Alertmanager version:

$ docker exec alertmanager alertmanager --version
alertmanager, version 0.27.0 (branch: HEAD, revision: 0aa3c2aad14cff039931923ab16b26b7481783b5)
build user:       root@22cd11f671e9
  build date:       20240228-11:51:20
  go version:       go1.21.7
  platform:         linux/amd64
  tags:             netgo

• Prometheus version:

$ docker exec prometheus prometheus --version
prometheus, version 2.51.2 (branch: HEAD, revision: b4c0ab52c3e9b940ab803581ddae9b3d9a452337)
  build user:       root@b63f02a423d9
  build date:       20240410-14:05:54
  go version:       go1.22.2
  platform:         linux/amd64
  tags:             netgo,builtinassets,stringlabels

• Alertmanager configuration file:

global:
  resolve_timeout: 10m

route:
  group_by: ['alertname']
  group_wait: 30s
  group_interval: 30s
  repeat_interval: 5m
  receiver: 'telegram'

receivers:
- name: 'telegram'
  telegram_configs:
  - send_resolved: true
    api_url: https://api.telegram.org
    bot_token: $BOT_TOKEN
    chat_id: $CHAT_ID
    parse_mode: HTML

• Prometheus configuration file: n/a

• Logs:

alertmanager  | ts=2024-03-20T15:40:52.935Z caller=main.go:181 level=info msg="Starting Alertmanager" version="(version=0.27.0, branch=HEAD, revision=0aa3c2aad14cff039931923ab16b26b7481783b5)"
alertmanager  | ts=2024-03-20T15:40:52.935Z caller=main.go:182 level=info build_context="(go=go1.21.7, platform=linux/amd64, user=root@22cd11f671e9, date=20240228-11:51:20, tags=netgo)"
alertmanager  | ts=2024-03-20T15:40:52.936Z caller=cluster.go:186 level=info component=cluster msg="setting advertise address explicitly" addr=192.168.93.141 port=9094
alertmanager  | ts=2024-03-20T15:40:52.937Z caller=cluster.go:683 level=info component=cluster msg="Waiting for gossip to settle..." interval=2s
alertmanager  | ts=2024-03-20T15:40:53.008Z caller=coordinator.go:113 level=info component=configuration msg="Loading configuration file" file=/etc/alertmanager/alertmanager.yaml
alertmanager  | ts=2024-03-20T15:40:53.009Z caller=coordinator.go:126 level=info component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/alertmanager.yaml
alertmanager  | ts=2024-03-20T15:40:53.013Z caller=tls_config.go:313 level=info msg="Listening on" address=[::]:9093
alertmanager  | ts=2024-03-20T15:40:53.013Z caller=tls_config.go:316 level=info msg="TLS is disabled." http2=false address=[::]:9093
alertmanager  | ts=2024-03-20T15:40:54.938Z caller=cluster.go:708 level=info component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.000347913s
alertmanager  | ts=2024-03-20T15:41:02.940Z caller=cluster.go:700 level=info component=cluster msg="gossip settled; proceeding" elapsed=10.002336569s

[grobinson-grafana]

Hi! 👋 Your timezone in /etc/localtime is plain text, but as far as I understand it needs to be a symlink to a timezone file in /usr/share/zoneinfo:

The /etc/localtime file configures the system-wide timezone of the local system that is used by applications for presentation to the user. It should be an absolute or relative symbolic link pointing to /usr/share/zoneinfo/, followed by a timezone identifier such as "Europe/Berlin" or "Etc/UTC". The resulting link should lead to the corresponding binary tzfile(5) timezone data for the configured timezone.

https://www.freedesktop.org/software/systemd/man/latest/localtime.html

Can you try this and see if it works?

[instantdreams]

My current approach works for the other 36+ containers I run. It works for Promtail. It doesn't work for Alertmanager, Loki, Prometheus, or Node-Exporter.

I did check the state of the files in the container:

~$ docker exec alertmanager cat /etc/timezone
America/Edmonton


$ docker exec alertmanager cat /etc/localtime
TZif2[binary-data]LMTMDTMSTMWTMPT
MST7MDT,M3.2.0,M11.1.0


$ docker exec alertmanager date
Wed Apr 17 14:56:40 UTC 2024

[grobinson-grafana]

D'oh! I misread the original comment. ~Your /etc/localtime does look correct.~ When I do this though, the date is correct. For example:

/alertmanager # date
Wed Apr 17 15:01:14 UTC 2024
/alertmanager # ln -sf /usr/share/zoneinfo/America/Edmonton /etc/localtime
/alertmanager # date
Wed Apr 17 09:01:48 MDT 2024

[instantdreams]

Here's my result:

$ docker exec -it alertmanager sh
/alertmanager $ date
Wed Apr 17 15:12:26 UTC 2024
/alertmanager $ exit
$ date
Wed 17 Apr 2024 09:12:28 AM MDT

Are you running alertmanager inside a docker image? Can you share your sanitised compose file details?

Addendum: To make sure both the host and the container output the date in the same format, I used the following:

$ docker exec -it alertmanager sh
/alertmanager $ date -R
Wed, 17 Apr 2024 15:14:26 +0000
/alertmanager $ exit
$ date --rfc-email
Wed, 17 Apr 2024 09:14:30 -0600

[grobinson-grafana]

OK! I think I know what the issue is – I suspect the base image which the docker images for Prometheus and Alertmanager are built on do not have tzdata installed. This means the TZ environment variable doesn't work. Can you create an issue in https://github.com/prometheus/busybox?

[grobinson-grafana]

Hi! 👋 I opened an issue for this but I suspect it will be rejected. Please see the following in the FAQ https://prometheus.io/docs/introduction/faq/#can-i-change-the-timezone-why-is-everything-in-utc.

[jkroepke]

Reading the issues, alermanger and prometheus works only in UTC exklusive.

In our company, we are using Grafana to manage the Alertmanager alerts and Grafana will covert the UTC times into local time. Create and silence with local time in grafana will be forwarded in UTC at AM.