spilo-role replica label missing from one of the patroni replica pods

[shubham-yewalekar]

• Operator Version? - ghcr.io/zalando/postgres-operator:v1.8.2
• Where do you run it - cloud or metal? Bare-metal K8s
• Type of issue? Bug Report

Hi,

I saw that one of the patroni replica pods was stuck in "starting" state because it was not able to connect via the replication k8s service. I checked that was because the endpoints were missing for this k8s service.

The endpoint wasn't created because none of the patroni pods had spilo-role=replica label set, so the endpoint isn't able to select any target

postgres operator logs

time="2024-07-05T09:36:00Z" level=debug msg="making GET http request: http://[fd74:ca9b:3a09:868c:172:18:0:6844]:8008/patroni" cluster-name= pkg=cluster time="2024-07-05T09:36:02Z" level=debug msg="making GET http request: http://[fd74:ca9b:3a09:868c:172:18:0:6643]:8008/patroni" cluster-name= pkg=cluster time="2024-07-05T09:36:02Z" level=debug msg="syncing pod disruption budgets" cluster-name= pkg=cluster time="2024-07-05T09:36:02Z" level=debug msg="syncing roles" cluster-name= pkg=cluster time="2024-07-05T09:36:02Z" level=debug msg="closing database connection" cluster-name= pkg=cluster time="2024-07-05T09:36:02Z" level=debug msg="syncing databases" cluster-name= pkg=cluster time="2024-07-05T09:36:02Z" level=debug msg="closing database connection" cluster-name= pkg=cluster time="2024-07-05T09:36:02Z" level=debug msg="syncing prepared databases with schemas" cluster-name= pkg=cluster time="2024-07-05T09:36:02Z" level=debug msg="syncing connection pooler (master, replica) from (false, nil) to (false, nil)" cluster-name= pkg=cluster time="2024-07-05T09:36:02Z" level=info msg="cluster has been synced" cluster-name= pkg=controller worker=0

whenever I try to label the pod manually, postgres-operator removes the spilo-role label

I don't see anything useful in the logs