prometheus-prometheus-kube-prometheus-prometheus-0 pod keeps restarting with high Memory and CPU share

[vedavalliradhika]

The prometheus-prometheus-kube-prometheus-prometheus-0 pod keeps restarting with the below message, "Readiness probe failed: HTTP probe failed with statuscode: 503" The memory share sometimes goes upto 6305% and CPU share upto 1570%

Below are the logs. Please suggest what would be cause of the issue.

[prometheus-prometheus-kube-prometheus-prometheus-0 config-reloader]  level=info ts=2021-05-10T14:05:06.542286287Z caller=main.go:147 msg="Starting prometheus-config-reloader" version="(version=0.44.0, branch=refs/tags/pkg/apis/monitoring/v0.44.0, revision=35c9101c332b9371172e1d6cc5a57c065f14eddf)"
[prometheus-prometheus-kube-prometheus-prometheus-0 config-reloader]  level=info ts=2021-05-10T14:05:06.542434168Z caller=main.go:148 build_context="(go=go1.14.12, user=paulfantom, date=20201202-15:44:08)"
[prometheus-prometheus-kube-prometheus-prometheus-0 config-reloader]  level=info ts=2021-05-10T14:05:06.542941238Z caller=main.go:182 msg="Starting web server for metrics" listen=:8080
[prometheus-prometheus-kube-prometheus-prometheus-0 config-reloader]  level=error ts=2021-05-10T14:05:06.559701989Z caller=runutil.go:98 msg="function failed. Retrying in next tick" err="trigger reload: reload request failed: Post \"http://127.0.0.1:9090/-/reload\": dial tcp 127.0.0.1:9090: connect: connection refused"
[prometheus-prometheus-kube-prometheus-prometheus-0 prometheus]  level=error ts=2021-05-10T14:06:09.602Z caller=db.go:1020 component=tsdb msg="Failed to read meta.json for a block during reload. Skipping" dir=/prometheus/01EWM4N3V49DBDMT40DDRTJ195 err="open /prometheus/01EWM4N3V49DBDMT40DDRTJ195/meta.json: no such file or directory"
[prometheus-prometheus-kube-prometheus-prometheus-0 prometheus]  level=error ts=2021-05-10T14:06:09.605Z caller=db.go:1020 component=tsdb msg="Failed to read meta.json for a block during reload. Skipping" dir=/prometheus/01EWM4Q6TQMZCV6T7RJSBZSFMQ err="open /prometheus/01EWM4Q6TQMZCV6T7RJSBZSFMQ/meta.json: no such file or directory"
[prometheus-prometheus-kube-prometheus-prometheus-0 prometheus]  level=error ts=2021-05-10T14:06:09.607Z caller=db.go:1020 component=tsdb msg="Failed to read meta.json for a block during reload. Skipping" dir=/prometheus/01EWM4SBW8SF9W58P1NJ27JMZQ err="open /prometheus/01EWM4SBW8SF9W58P1NJ27JMZQ/meta.json: no such file or directory"
[prometheus-prometheus-kube-prometheus-prometheus-0 prometheus]  level=error ts=2021-05-10T14:06:09.609Z caller=db.go:1020 component=tsdb msg="Failed to read meta.json for a block during reload. Skipping" dir=/prometheus/01EWM4VFAYPJ7B2SZ8PD7EDEKK err="open /prometheus/01EWM4VFAYPJ7B2SZ8PD7EDEKK/meta.json: no such file or directory"
[prometheus-prometheus-kube-prometheus-prometheus-0 prometheus]  level=error ts=2021-05-10T14:06:09.611Z caller=db.go:1020 component=tsdb msg="Failed to read meta.json for a block during reload. Skipping" dir=/prometheus/01EWM4XJXZ7JJMXDXXAZT1QXFX err="open /prometheus/01EWM4XJXZ7JJMXDXXAZT1QXFX/meta.json: no such file or directory"
[prometheus-prometheus-kube-prometheus-prometheus-0 prometheus]  level=error ts=2021-05-10T14:06:09.613Z caller=db.go:1020 component=tsdb msg="Failed to read meta.json for a block during reload. Skipping" dir=/prometheus/01EWM4ZNK9CH5PB9EDHRCT4SQ5 err="open /prometheus/01EWM4ZNK9CH5PB9EDHRCT4SQ5/meta.json: no such file or directory"
[prometheus-prometheus-kube-prometheus-prometheus-0 prometheus]  level=error ts=2021-05-10T14:06:09.616Z caller=db.go:1020 component=tsdb msg="Failed to read meta.json for a block during reload. Skipping" dir=/prometheus/01EWM51TQDG5TH4X4HX447539G err="open /prometheus/01EWM51TQDG5TH4X4HX447539G/meta.json: no such file or directory"
[prometheus-prometheus-kube-prometheus-prometheus-0 prometheus]  level=error ts=2021-05-10T14:06:09.616Z caller=db.go:1020 component=tsdb msg="Failed to read meta.json for a block during reload. Skipping" dir=/prometheus/01EWM53XXQB5ZTE7V9RJF8GSBV err="open /prometheus/01EWM53XXQB5ZTE7V9RJF8GSBV/meta.json: no such file or directory"
[prometheus-prometheus-kube-prometheus-prometheus-0 prometheus]  level=error ts=2021-05-10T14:06:09.618Z caller=db.go:1020 component=tsdb msg="Failed to read meta.json for a block during reload. Skipping" dir=/prometheus/01EWM560VVSB5QFZTD973D5BYW err="open /prometheus/01EWM560VVSB5QFZTD973D5BYW/meta.json: no such file or directory"
[prometheus-prometheus-kube-prometheus-prometheus-0 prometheus]  level=error ts=2021-05-10T14:06:09.619Z caller=db.go:1020 component=tsdb msg="Failed to read meta.json for a block during reload. Skipping" dir=/prometheus/01EWM5842DYCPG13WY7VRVTFHP err="open /prometheus/01EWM5842DYCPG13WY7VRVTFHP/meta.json: no such file or directory"
[prometheus-prometheus-kube-prometheus-prometheus-0 prometheus]  level=error ts=2021-05-10T14:06:09.621Z caller=db.go:1020 component=tsdb msg="Failed to read meta.json for a block during reload. Skipping" dir=/prometheus/01EWM5A7HC3CZPXSK9QCG3PAD6 err="open /prometheus/01EWM5A7HC3CZPXSK9QCG3PAD6/meta.json: no such file or directory"
[prometheus-prometheus-kube-prometheus-prometheus-0 prometheus]  level=error ts=2021-05-10T14:06:09.623Z caller=db.go:1020 component=tsdb msg="Failed to read meta.json for a block during reload. Skipping" dir=/prometheus/01EWM5CB76GQ3X0D1E0PBRBGHP err="open /prometheus/01EWM5CB76GQ3X0D1E0PBRBGHP/meta.json: no such file or directory"
[prometheus-prometheus-kube-prometheus-prometheus-0 prometheus]  level=error ts=2021-05-10T14:06:09.625Z caller=db.go:1020 component=tsdb msg="Failed to read meta.json for a block during reload. Skipping" dir=/prometheus/01EWM5EEP64FJ0YK1J452FR5W4 err="open /prometheus/01EWM5EEP64FJ0YK1J452FR5W4/meta.json: no such file or directory"
[prometheus-prometheus-kube-prometheus-prometheus-0 prometheus]  level=error ts=2021-05-10T14:06:09.627Z caller=db.go:1020 component=tsdb msg="Failed to read meta.json for a block during reload. Skipping" dir=/prometheus/01EWM5GJ7H0PW595XQFH4V87MR err="open /prometheus/01EWM5GJ7H0PW595XQFH4V87MR/meta.json: no such file or directory"
[prometheus-prometheus-kube-prometheus-prometheus-0 prometheus]  level=error ts=2021-05-10T14:06:09.629Z caller=db.go:1020 component=tsdb msg="Failed to read meta.json for a block during reload. Skipping" dir=/prometheus/01EWM5JMX01PVVS4D7K130ZP9Y err="open /prometheus/01EWM5JMX01PVVS4D7K130ZP9Y/meta.json: no such file or directory"
[prometheus-prometheus-kube-prometheus-prometheus-0 prometheus]  level=error ts=2021-05-10T14:06:09.631Z caller=db.go:1020 component=tsdb msg="Failed to read meta.json for a block during reload. Skipping" dir=/prometheus/01EWM5MS9PYZ0DET6A6MF315Q7 err="open /prometheus/01EWM5MS9PYZ0DET6A6MF315Q7/meta.json: no such file or directory"
[prometheus-prometheus-kube-prometheus-prometheus-0 prometheus]  level=error ts=2021-05-10T14:06:09.632Z caller=db.go:1020 component=tsdb msg="Failed to read meta.json for a block during reload. Skipping" dir=/prometheus/01EWM5PXCW5697W529EEFR3KVV err="open /prometheus/01EWM5PXCW5697W529EEFR3KVV/meta.json: no such file or directory"
[prometheus-prometheus-kube-prometheus-prometheus-0 prometheus]  level=error ts=2021-05-10T14:06:09.634Z caller=db.go:1020 component=tsdb msg="Failed to read meta.json for a block during reload. Skipping" dir=/prometheus/01EWM5S175AT1ACNPK32ZQJ0EZ err="open /prometheus/01EWM5S175AT1ACNPK32ZQJ0EZ/meta.json: no such file or directory"
[prometheus-prometheus-kube-prometheus-prometheus-0 prometheus]  level=error ts=2021-05-10T14:06:09.636Z caller=db.go:1020 component=tsdb msg="Failed to read meta.json for a block during reload. Skipping" dir=/prometheus/01EWM5V42777R854YZBNATQVJV err="open /prometheus/01EWM5V42777R854YZBNATQVJV/meta.json: no such file or directory"
[prometheus-prometheus-kube-prometheus-prometheus-0 prometheus]  level=error ts=2021-05-10T14:06:09.638Z caller=db.go:1020 component=tsdb msg="Failed to read meta.json for a block during reload. Skipping" dir=/prometheus/01EWM5X74YAK4V9WP821CH2TDD err="open /prometheus/01EWM5X74YAK4V9WP821CH2TDD/meta.json: no such file or directory"
[prometheus-prometheus-kube-prometheus-prometheus-0 prometheus]  level=error ts=2021-05-10T14:06:09.641Z caller=db.go:1020 component=tsdb msg="Failed to read meta.json for a block during reload. Skipping" dir=/prometheus/01EWM5ZA2J4XMFHGAQXVD8MZN9 err="open /prometheus/01EWM5ZA2J4XMFHGAQXVD8MZN9/meta.json: no such file or directory"
[prometheus-prometheus-kube-prometheus-prometheus-0 prometheus]  level=error ts=2021-05-10T14:06:09.643Z caller=db.go:1020 component=tsdb msg="Failed to read meta.json for a block during reload. Skipping" dir=/prometheus/01EWM61BT3SFQ93BRY8APXPGBK err="open /prometheus/01EWM61BT3SFQ93BRY8APXPGBK/meta.json: no such file or directory"
[prometheus-prometheus-kube-prometheus-prometheus-0 prometheus]  level=error ts=2021-05-10T14:06:09.643Z caller=db.go:1020 component=tsdb msg="Failed to read meta.json for a block during reload. Skipping" dir=/prometheus/01EWM63GV297V1VT636SNCAPMK err="open /prometheus/01EWM63GV297V1VT636SNCAPMK/meta.json: no such file or directory"
[prometheus-prometheus-kube-prometheus-prometheus-0 prometheus]  level=error ts=2021-05-10T14:06:09.645Z caller=db.go:1020 component=tsdb msg="Failed to read meta.json for a block during reload. Skipping" dir=/prometheus/01EWM65MBAVQ8A36M2X8VFJJMZ err="open /prometheus/01EWM65MBAVQ8A36M2X8VFJJMZ/meta.json: no such file or directory"
[prometheus-prometheus-kube-prometheus-prometheus-0 prometheus]  level=info ts=2021-05-10T14:06:17.668Z caller=head.go:642 component=tsdb msg="Replaying on-disk memory mappable chunks if any"

[paulfantom]

It looks like your prometheus tries to replay WAL on start. This operation is very memory intensive and it is possible that you don't have enough memory.

If you can afford to lose up to 2h of last data in prometheus, I can recommend removing the content of /prometheus/wal/ directory and restarting pod. If you cannot afford to lose data, you need to increase how much RAM a container is allowed to use.

For prometheus capacity planning I recommend reading and using the following: https://www.robustperception.io/how-much-ram-does-prometheus-2-x-need-for-cardinality-and-ingestion

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had any activity in the last 60 days. Thank you for your contributions.

[github-actions[bot]]

This issue was closed because it has not had any activity in the last 120 days. Please reopen if you feel this is still valid.