Prometheus container keeps restarting

[eeganlf]

Provide a general summary of the issue in the Title above

If you're describing a bug, tell us what should happen If you're suggesting a change/improvement, tell us how it should work

Expected Behaviour

Prometheus container to remain in running state

Current Behaviour

Prometheus is constantly in restarting state

Possible Solution

I'm at a loss

Steps to Reproduce (for bugs)

Provide a link to a live example, or steps to reproduce this bug. Include code to reproduce, if relevant:

1. git clone https://github.com/vegasbrianc/prometheus.git
2. cd prometheus
3. docker compose up -d
4. all containers run and can be visited via IPADDRESS:port except for prometheus
5. docker compose ps shows prometheus in restarting state

Your Environment

Include as many relevant details about the environment you experienced the bug in

• Docker version docker version (e.g. Docker 17.0.05 ): 20.10.17
• Operating System and version (e.g. Linux, Windows, MacOS): Ubuntu 20.04

[github-actions[bot]]

Awesome! Thanks for taking the time to open an issue. We will have a look and answer as soon as we can.' first issue

[th0mk]

I think I'm having the same issue, here are some logs from my prometheus container:

ts=2022-06-16T07:10:12.860Z caller=main.go:972 level=info msg="Starting TSDB ..."
ts=2022-06-16T07:10:12.860Z caller=tls_config.go:195 level=info component=web msg="TLS is disabled." http2=false
ts=2022-06-16T07:10:12.862Z caller=head.go:493 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any"
ts=2022-06-16T07:10:12.862Z caller=head.go:536 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=1.052µs
ts=2022-06-16T07:10:12.862Z caller=head.go:542 level=info component=tsdb msg="Replaying WAL, this may take a while"
ts=2022-06-16T07:10:12.932Z caller=head.go:613 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=1
ts=2022-06-16T07:10:12.932Z caller=head.go:613 level=info component=tsdb msg="WAL segment loaded" segment=1 maxSegment=1
ts=2022-06-16T07:10:12.932Z caller=head.go:619 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=12.042µs wal_replay_duration=70.285218ms total_replay_duration=70.309253ms
ts=2022-06-16T07:10:12.934Z caller=main.go:993 level=info fs_type=EXT4_SUPER_MAGIC
ts=2022-06-16T07:10:12.934Z caller=main.go:996 level=info msg="TSDB started"
ts=2022-06-16T07:10:12.934Z caller=main.go:1177 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml
ts=2022-06-16T07:10:12.935Z caller=main.go:1214 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=978.058µs db_storage=501ns remote_storage=1.593µs web_handler=501ns query_engine=611ns scrape=161.942µs scrape_sd=81.853µs notify=15.519µs notify_sd=4.588µs rules=415.026µs tracing=3.226µs
ts=2022-06-16T07:10:12.935Z caller=main.go:957 level=info msg="Server is ready to receive web requests."
ts=2022-06-16T07:10:12.935Z caller=manager.go:937 level=info component="rule manager" msg="Starting rule manager..."
ts=2022-06-16T07:18:15.885Z caller=main.go:807 level=warn msg="Received SIGTERM, exiting gracefully..."
ts=2022-06-16T07:18:15.885Z caller=main.go:831 level=info msg="Stopping scrape discovery manager..."
ts=2022-06-16T07:18:15.885Z caller=main.go:845 level=info msg="Stopping notify discovery manager..."
ts=2022-06-16T07:18:15.885Z caller=manager.go:951 level=info component="rule manager" msg="Stopping rule manager..."
ts=2022-06-16T07:18:15.885Z caller=manager.go:961 level=info component="rule manager" msg="Rule manager stopped"
ts=2022-06-16T07:18:15.885Z caller=main.go:827 level=info msg="Scrape discovery manager stopped"
ts=2022-06-16T07:18:15.885Z caller=main.go:882 level=info msg="Stopping scrape manager..."
ts=2022-06-16T07:18:15.885Z caller=main.go:841 level=info msg="Notify discovery manager stopped"
ts=2022-06-16T07:18:15.885Z caller=main.go:874 level=info msg="Scrape manager stopped"
ts=2022-06-16T07:18:15.886Z caller=notifier.go:599 level=info component=notifier msg="Stopping notification manager..."
ts=2022-06-16T07:18:15.886Z caller=main.go:1103 level=info msg="Notifier manager stopped"
ts=2022-06-16T07:18:15.886Z caller=main.go:1115 level=info msg="See you next time!"

The issue is in "Received SIGTERM, exiting gracefully...".

It's somehow getting the signal to terminate, even though there are no health checks configured.

[eeganlf]

In the docker-compose.yml file in the root directory of the project, change the Prometheus image to use the latest image:

services:

  prometheus:
    image: prom/prometheus:latest

A later version than the one on this repository may work that isn't the latest but I didn't try any others.

[vegasbrianc]

@eeganlf I believe this is now resolved. Let me know if you find anything else.

[github-actions[bot]]

This issue has seen no activity and is marked as stale