Thanos ruler: Alertmanager v0.27.0 /api/v1 endpoints deprecated

[federicopires]

Thanos, Prometheus and Golang version used:

v0.34.1

Object Storage Provider: AWS S3

What happened: Thanos ruler is randomly failing to send alerts to Alertmanager v0.27.0 with bad response status 410 Gone

What you expected to happen: No errors while sending alerts

How to reproduce it (as minimally and precisely as possible): Install latest Alertmanager v0.27.0 (I'm using latest kube-prometheus-stack version with their implementation of thanos ruler) and Thanos ruler 0.34.1 setup to send alerts to it

Full logs to relevant components: Thanos ruler:

err="bad response status 410 Gone from \"http://prometheus-kube-prometheus-alertmanager.monitoring.svc.cluster.local:9093/api/v1/alerts

Alertmanager: ts=2024-03-13T14:32:22.436Z caller=v1_deprecation_router.go:51 level=warn component=api version=v1 msg="v1 API received a request on a removed endpoint" path=/api/v1/alerts method=POST

Anything else we need to know: Related to https://github.com/prometheus/alertmanager/pull/2970

[jacobbaungard]

v2 will be default by the next release https://github.com/thanos-io/thanos/pull/7123 . Until then you can change the API version in the alertmanager config: https://thanos.io/v0.34/components/rule.md/#configuration

[MichaHoffmann]

This is fixed by upgrading to a recent thanos version, Ill close this; please reopen if I missed something!