postgres-operator icon indicating copy to clipboard operation
postgres-operator copied to clipboard

Published 20 hours ago • verified publisher icon CrunchyData

UnableToCreateStanzas warning with wrong address for backup repo pod

Open burigolucas opened this issue 9 months ago • 2 comments

Overview

By creating a postgres cluster, a warning event with reason UnableToCreateStanzas is created with following message:

command terminated with exit code 49: ERROR: [049]: unable to get address for 'db1-repo-host-0.db1-pods.demo-database.svc.kubernetes.default.': [-2] Name or service not known

The cronjob for the backup is not created resulting in the WAL filling up the disk of the postgres instance.

Environment

Please provide the following details:

  • Platform: ACK Serverless (Alibaba)
  • Platform Version: Kubernetes version 1.28.3
  • PGO Image Tag: ubi8-5.5.1-0
  • Postgres Version 16
  • Storage: storage class alicloud-disk-alltype, provisioner diskplugin.csi.alibabacloud.com

Steps to Reproduce

REPRO

Provide steps to get to the error condition:

Sample of postgres cluster definition:

apiVersion: postgres-operator.crunchydata.com/v1beta1
kind: PostgresCluster
metadata:
  name: db1
  namespace: demo-database
spec:
  backups:
    pgbackrest:
      global:
        repo1-retention-full: "4"
        repo1-retention-full-type: time
      repos:
      - name: repo1
        schedules:
          full: 10 1 * * *
        volume:
          volumeClaimSpec:
            accessModes:
            - ReadWriteOnce
            resources:
              requests:
                storage: 50Gi
  instances:
  - dataVolumeClaimSpec:
      accessModes:
      - ReadWriteOnce
      resources:
        requests:
          storage: 100Gi
    name: instance1
    replicas: 1
  patroni:
    dynamicConfiguration:
      postgresql:
        parameters:
          archive_mode: "on"
          archive_timeout: 60
          max_connections: 1000
          max_parallel_workers: 2
          max_worker_processes: 2
          shared_buffers: 1GB
          shared_preload_libraries: timescaledb
          work_mem: 2MB
    leaderLeaseDurationSeconds: 30
    port: 8008
    syncPeriodSeconds: 10
  port: 5432
  postgresVersion: 16
  users:
  - name: postgres

EXPECTED

  1. Provide the behavior that you expected.

cronjob for backup to be created and backups to local repo1 to happen as per schedule.

ACTUAL

  1. Describe what actually happens

Please, see below logs of the operator:

Logs

time="2024-05-02T11:13:08Z" level=debug msg="reconciled instance" controller=postgrescluster controllerGroup=postgres-operator.crunchydata.com controllerKind=PostgresCluster instance=db1-instance1-k5mj name=db1 namespace=demo-database postgresCluster=demo-database/db1 reconcileID=9630b809-771b-4df5-a8e1-5542c456f87f version=5.5.1-0-amd64
time="2024-05-02T11:13:08Z" level=debug msg="reconciled instance set" controller=postgrescluster controllerGroup=postgres-operator.crunchydata.com controllerKind=PostgresCluster instance-set=instance1 name=db1 namespace=demo-database postgresCluster=demo-database/db1 reconcileID=9630b809-771b-4df5-a8e1-5542c456f87f version=5.5.1-0-amd64
time="2024-05-02T11:13:08Z" level=debug msg=Normal message="created pgBackRest repository host StatefulSet/db1-repo-host" object="{PostgresCluster demo-database db1 82161687-e0cd-4fca-9387-a60c99006af5 postgres-operator.crunchydata.com/v1beta1 2382010199 }" reason=RepoHostCreated version=5.5.1-0-amd64
time="2024-05-02T11:13:08Z" level=debug msg="reconciled cluster" controller=postgrescluster controllerGroup=postgres-operator.crunchydata.com controllerKind=PostgresCluster name=db1 namespace=demo-database postgresCluster=demo-database/db1 reconcileID=9630b809-771b-4df5-a8e1-5542c456f87f version=5.5.1-0-amd64
time="2024-05-02T11:13:08Z" level=debug msg="patched cluster status" controller=postgrescluster controllerGroup=postgres-operator.crunchydata.com controllerKind=PostgresCluster name=db1 namespace=demo-database postgresCluster=demo-database/db1 reconcileID=9630b809-771b-4df5-a8e1-5542c456f87f version=5.5.1-0-amd64
time="2024-05-02T11:13:08Z" level=debug msg="reconciled instance" controller=postgrescluster controllerGroup=postgres-operator.crunchydata.com controllerKind=PostgresCluster instance=db1-instance1-k5mj name=db1 namespace=demo-database postgresCluster=demo-database/db1 reconcileID=f2439856-0151-4406-b355-2b24fbc9e157 version=5.5.1-0-amd64
time="2024-05-02T11:13:08Z" level=debug msg="reconciled instance set" controller=postgrescluster controllerGroup=postgres-operator.crunchydata.com controllerKind=PostgresCluster instance-set=instance1 name=db1 namespace=demo-database postgresCluster=demo-database/db1 reconcileID=f2439856-0151-4406-b355-2b24fbc9e157 version=5.5.1-0-amd64
time="2024-05-02T11:13:08Z" level=debug msg="reconciled cluster" controller=postgrescluster controllerGroup=postgres-operator.crunchydata.com controllerKind=PostgresCluster name=db1 namespace=demo-database postgresCluster=demo-database/db1 reconcileID=f2439856-0151-4406-b355-2b24fbc9e157 version=5.5.1-0-amd64
time="2024-05-02T11:13:08Z" level=debug msg="patched cluster status" controller=postgrescluster controllerGroup=postgres-operator.crunchydata.com controllerKind=PostgresCluster name=db1 namespace=demo-database postgresCluster=demo-database/db1 reconcileID=f2439856-0151-4406-b355-2b24fbc9e157 version=5.5.1-0-amd64
time="2024-05-02T11:13:09Z" level=debug msg="reconciled instance" controller=postgrescluster controllerGroup=postgres-operator.crunchydata.com controllerKind=PostgresCluster instance=db1-instance1-k5mj name=db1 namespace=demo-database postgresCluster=demo-database/db1 reconcileID=32d8abd4-2429-49c2-97cf-2da56736b3dc version=5.5.1-0-amd64
time="2024-05-02T11:13:09Z" level=debug msg="reconciled instance set" controller=postgrescluster controllerGroup=postgres-operator.crunchydata.com controllerKind=PostgresCluster instance-set=instance1 name=db1 namespace=demo-database postgresCluster=demo-database/db1 reconcileID=32d8abd4-2429-49c2-97cf-2da56736b3dc version=5.5.1-0-amd64
time="2024-05-02T11:13:09Z" level=debug msg="reconciled cluster" controller=postgrescluster controllerGroup=postgres-operator.crunchydata.com controllerKind=PostgresCluster name=db1 namespace=demo-database postgresCluster=demo-database/db1 reconcileID=32d8abd4-2429-49c2-97cf-2da56736b3dc version=5.5.1-0-amd64
time="2024-05-02T11:13:09Z" level=debug msg="patched cluster status" controller=postgrescluster controllerGroup=postgres-operator.crunchydata.com controllerKind=PostgresCluster name=db1 namespace=demo-database postgresCluster=demo-database/db1 reconcileID=32d8abd4-2429-49c2-97cf-2da56736b3dc version=5.5.1-0-amd64
time="2024-05-02T11:13:09Z" level=debug msg="reconciled instance" controller=postgrescluster controllerGroup=postgres-operator.crunchydata.com controllerKind=PostgresCluster instance=db1-instance1-k5mj name=db1 namespace=demo-database postgresCluster=demo-database/db1 reconcileID=26223bbf-9c26-4aeb-ad8b-3ec1da356c60 version=5.5.1-0-amd64
time="2024-05-02T11:13:09Z" level=debug msg="reconciled instance set" controller=postgrescluster controllerGroup=postgres-operator.crunchydata.com controllerKind=PostgresCluster instance-set=instance1 name=db1 namespace=demo-database postgresCluster=demo-database/db1 reconcileID=26223bbf-9c26-4aeb-ad8b-3ec1da356c60 version=5.5.1-0-amd64
time="2024-05-02T11:13:09Z" level=debug msg="reconciled cluster" controller=postgrescluster controllerGroup=postgres-operator.crunchydata.com controllerKind=PostgresCluster name=db1 namespace=demo-database postgresCluster=demo-database/db1 reconcileID=26223bbf-9c26-4aeb-ad8b-3ec1da356c60 version=5.5.1-0-amd64
time="2024-05-02T11:13:09Z" level=debug msg="patched cluster status" controller=postgrescluster controllerGroup=postgres-operator.crunchydata.com controllerKind=PostgresCluster name=db1 namespace=demo-database postgresCluster=demo-database/db1 reconcileID=26223bbf-9c26-4aeb-ad8b-3ec1da356c60 version=5.5.1-0-amd64
time="2024-05-02T11:13:09Z" level=debug msg="reconciled instance" controller=postgrescluster controllerGroup=postgres-operator.crunchydata.com controllerKind=PostgresCluster instance=db1-instance1-k5mj name=db1 namespace=demo-database postgresCluster=demo-database/db1 reconcileID=eb9f2028-8d27-4bb2-85e6-d7c64c4d5f2f version=5.5.1-0-amd64
time="2024-05-02T11:13:09Z" level=debug msg="reconciled instance set" controller=postgrescluster controllerGroup=postgres-operator.crunchydata.com controllerKind=PostgresCluster instance-set=instance1 name=db1 namespace=demo-database postgresCluster=demo-database/db1 reconcileID=eb9f2028-8d27-4bb2-85e6-d7c64c4d5f2f version=5.5.1-0-amd64
time="2024-05-02T11:13:10Z" level=debug msg="reconciled cluster" controller=postgrescluster controllerGroup=postgres-operator.crunchydata.com controllerKind=PostgresCluster name=db1 namespace=demo-database postgresCluster=demo-database/db1 reconcileID=eb9f2028-8d27-4bb2-85e6-d7c64c4d5f2f version=5.5.1-0-amd64
time="2024-05-02T11:13:10Z" level=debug msg="reconciled instance" controller=postgrescluster controllerGroup=postgres-operator.crunchydata.com controllerKind=PostgresCluster instance=db1-instance1-k5mj name=db1 namespace=demo-database postgresCluster=demo-database/db1 reconcileID=4aeffa11-2f6a-4dab-8e08-b312c1e04fcd version=5.5.1-0-amd64
time="2024-05-02T11:13:10Z" level=debug msg="reconciled instance set" controller=postgrescluster controllerGroup=postgres-operator.crunchydata.com controllerKind=PostgresCluster instance-set=instance1 name=db1 namespace=demo-database postgresCluster=demo-database/db1 reconcileID=4aeffa11-2f6a-4dab-8e08-b312c1e04fcd version=5.5.1-0-amd64
time="2024-05-02T11:13:10Z" level=debug msg="reconciled cluster" controller=postgrescluster controllerGroup=postgres-operator.crunchydata.com controllerKind=PostgresCluster name=db1 namespace=demo-database postgresCluster=demo-database/db1 reconcileID=4aeffa11-2f6a-4dab-8e08-b312c1e04fcd version=5.5.1-0-amd64
time="2024-05-02T11:13:30Z" level=debug msg="reconciled instance" controller=postgrescluster controllerGroup=postgres-operator.crunchydata.com controllerKind=PostgresCluster instance=db1-instance1-k5mj name=db1 namespace=demo-database postgresCluster=demo-database/db1 reconcileID=3cb4592c-5158-434f-8bc7-31d767525596 version=5.5.1-0-amd64
time="2024-05-02T11:13:30Z" level=debug msg="reconciled instance set" controller=postgrescluster controllerGroup=postgres-operator.crunchydata.com controllerKind=PostgresCluster instance-set=instance1 name=db1 namespace=demo-database postgresCluster=demo-database/db1 reconcileID=3cb4592c-5158-434f-8bc7-31d767525596 version=5.5.1-0-amd64
time="2024-05-02T11:13:30Z" level=debug msg="enabled pgAudit" controller=postgrescluster controllerGroup=postgres-operator.crunchydata.com controllerKind=PostgresCluster name=db1 namespace=demo-database pod=db1-instance1-k5mj-0 postgresCluster=demo-database/db1 reconcileID=3cb4592c-5158-434f-8bc7-31d767525596 revision=559678bf8f stderr= stdout= version=5.5.1-0-amd64
time="2024-05-02T11:13:31Z" level=debug msg="created PostgreSQL databases" controller=postgrescluster controllerGroup=postgres-operator.crunchydata.com controllerKind=PostgresCluster name=db1 namespace=demo-database pod=db1-instance1-k5mj-0 postgresCluster=demo-database/db1 reconcileID=3cb4592c-5158-434f-8bc7-31d767525596 revision=559678bf8f stderr= stdout= version=5.5.1-0-amd64
time="2024-05-02T11:13:31Z" level=debug msg="wrote PostgreSQL users" controller=postgrescluster controllerGroup=postgres-operator.crunchydata.com controllerKind=PostgresCluster name=db1 namespace=demo-database pod=db1-instance1-k5mj-0 postgresCluster=demo-database/db1 reconcileID=3cb4592c-5158-434f-8bc7-31d767525596 revision=78b5654fb8 stderr= stdout= version=5.5.1-0-amd64
time="2024-05-02T11:14:31Z" level=error msg="unable to create stanza" controller=postgrescluster controllerGroup=postgres-operator.crunchydata.com controllerKind=PostgresCluster error="command terminated with exit code 49: ERROR: [049]: unable to get address for 'db1-repo-host-0.db1-pods.demo-database.svc.kubernetes.default.': [-2] Name or service not known\n" file="internal/controller/postgrescluster/pgbackrest.go:2618" func="postgrescluster.(*Reconciler).reconcileStanzaCreate" name=db1 namespace=demo-database postgresCluster=demo-database/db1 reconcileID=3cb4592c-5158-434f-8bc7-31d767525596 reconciler=pgBackRest version=5.5.1-0-amd64
time="2024-05-02T11:14:31Z" level=debug msg=Warning message="command terminated with exit code 49: ERROR: [049]: unable to get address for 'db1-repo-host-0.db1-pods.demo-database.svc.kubernetes.default.': [-2] Name or service not known\n" object="{PostgresCluster demo-database db1 82161687-e0cd-4fca-9387-a60c99006af5 postgres-operator.crunchydata.com/v1beta1 2382010354 }" reason=UnableToCreateStanzas version=5.5.1-0-amd64
time="2024-05-02T11:14:31Z" level=debug msg="removed PgBouncer objects" controller=postgrescluster controllerGroup=postgres-operator.crunchydata.com controllerKind=PostgresCluster name=db1 namespace=demo-database postgresCluster=demo-database/db1 reconcileID=3cb4592c-5158-434f-8bc7-31d767525596 revision=5c9966f6bc stderr= stdout= version=5.5.1-0-amd64
time="2024-05-02T11:14:31Z" level=debug msg="removed PgBouncer user" controller=postgrescluster controllerGroup=postgres-operator.crunchydata.com controllerKind=PostgresCluster name=db1 namespace=demo-database postgresCluster=demo-database/db1 reconcileID=3cb4592c-5158-434f-8bc7-31d767525596 revision=5c9966f6bc stderr= stdout= version=5.5.1-0-amd64
time="2024-05-02T11:14:31Z" level=debug msg="monitoring user disabled" controller=postgrescluster controllerGroup=postgres-operator.crunchydata.com controllerKind=PostgresCluster name=db1 namespace=demo-database postgresCluster=demo-database/db1 reconcileID=3cb4592c-5158-434f-8bc7-31d767525596 revision=559c4c97d6 stderr= stdout= version=5.5.1-0-amd64
time="2024-05-02T11:14:31Z" level=debug msg="reconciled cluster" controller=postgrescluster controllerGroup=postgres-operator.crunchydata.com controllerKind=PostgresCluster name=db1 namespace=demo-database postgresCluster=demo-database/db1 reconcileID=3cb4592c-5158-434f-8bc7-31d767525596 version=5.5.1-0-amd64
time="2024-05-02T11:14:31Z" level=debug msg="patched cluster status" controller=postgrescluster controllerGroup=postgres-operator.crunchydata.com controllerKind=PostgresCluster name=db1 namespace=demo-database postgresCluster=demo-database/db1 reconcileID=3cb4592c-5158-434f-8bc7-31d767525596 version=5.5.1-0-amd64
time="2024-05-02T11:14:32Z" level=debug msg="replaced configuration" controller=postgrescluster controllerGroup=postgres-operator.crunchydata.com controllerKind=PostgresCluster name=db1 namespace=demo-database postgresCluster=demo-database/db1 reconcileID=c253da23-5066-4d14-92da-022641486662 stderr= stdout="Not changed\n" version=5.5.1-0-amd64
time="2024-05-02T11:14:32Z" level=debug msg="reconciled instance" controller=postgrescluster controllerGroup=postgres-operator.crunchydata.com controllerKind=PostgresCluster instance=db1-instance1-k5mj name=db1 namespace=demo-database postgresCluster=demo-database/db1 reconcileID=c253da23-5066-4d14-92da-022641486662 version=5.5.1-0-amd64
time="2024-05-02T11:14:32Z" level=debug msg="reconciled instance set" controller=postgrescluster controllerGroup=postgres-operator.crunchydata.com controllerKind=PostgresCluster instance-set=instance1 name=db1 namespace=demo-database postgresCluster=demo-database/db1 reconcileID=c253da23-5066-4d14-92da-022641486662 version=5.5.1-0-amd64
time="2024-05-02T11:15:33Z" level=error msg="unable to create stanza" controller=postgrescluster controllerGroup=postgres-operator.crunchydata.com controllerKind=PostgresCluster error="command terminated with exit code 49: ERROR: [049]: unable to get address for 'db1-repo-host-0.db1-pods.demo-database.svc.kubernetes.default.': [-2] Name or service not known\n" file="internal/controller/postgrescluster/pgbackrest.go:2618" func="postgrescluster.(*Reconciler).reconcileStanzaCreate" name=db1 namespace=demo-database postgresCluster=demo-database/db1 reconcileID=c253da23-5066-4d14-92da-022641486662 reconciler=pgBackRest version=5.5.1-0-amd64
time="2024-05-02T11:15:33Z" level=debug msg=Warning message="command terminated with exit code 49: ERROR: [049]: unable to get address for 'db1-repo-host-0.db1-pods.demo-database.svc.kubernetes.default.': [-2] Name or service not known\n" object="{PostgresCluster demo-database db1 82161687-e0cd-4fca-9387-a60c99006af5 postgres-operator.crunchydata.com/v1beta1 2382014423 }" reason=UnableToCreateStanzas version=5.5.1-0-amd64
time="2024-05-02T11:15:33Z" level=debug msg="reconciled cluster" controller=postgrescluster controllerGroup=postgres-operator.crunchydata.com controllerKind=PostgresCluster name=db1 namespace=demo-database postgresCluster=demo-database/db1 reconcileID=c253da23-5066-4d14-92da-022641486662 version=5.5.1-0-amd64
time="2024-05-02T11:15:33Z" level=debug msg="patched cluster status" controller=postgrescluster controllerGroup=postgres-operator.crunchydata.com controllerKind=PostgresCluster name=db1 namespace=demo-database postgresCluster=demo-database/db1 reconcileID=c253da23-5066-4d14-92da-022641486662 version=5.5.1-0-amd64
time="2024-05-02T11:15:34Z" level=debug msg="replaced configuration" controller=postgrescluster controllerGroup=postgres-operator.crunchydata.com controllerKind=PostgresCluster name=db1 namespace=demo-database postgresCluster=demo-database/db1 reconcileID=54b678ea-2f9f-409e-864a-5f7832de615d stderr= stdout="Not changed\n" version=5.5.1-0-amd64
time="2024-05-02T11:15:34Z" level=debug msg="reconciled instance" controller=postgrescluster controllerGroup=postgres-operator.crunchydata.com controllerKind=PostgresCluster instance=db1-instance1-k5mj name=db1 namespace=demo-database postgresCluster=demo-database/db1 reconcileID=54b678ea-2f9f-409e-864a-5f7832de615d version=5.5.1-0-amd64
time="2024-05-02T11:15:34Z" level=debug msg="reconciled instance set" controller=postgrescluster controllerGroup=postgres-operator.crunchydata.com controllerKind=PostgresCluster instance-set=instance1 name=db1 namespace=demo-database postgresCluster=demo-database/db1 reconcileID=54b678ea-2f9f-409e-864a-5f7832de615d version=5.5.1-0-amd64
time="2024-05-02T11:16:35Z" level=error msg="unable to create stanza" controller=postgrescluster controllerGroup=postgres-operator.crunchydata.com controllerKind=PostgresCluster error="command terminated with exit code 49: ERROR: [049]: unable to get address for 'db1-repo-host-0.db1-pods.demo-database.svc.kubernetes.default.': [-2] Name or service not known\n" file="internal/controller/postgrescluster/pgbackrest.go:2618" func="postgrescluster.(*Reconciler).reconcileStanzaCreate" name=db1 namespace=demo-database postgresCluster=demo-database/db1 reconcileID=54b678ea-2f9f-409e-864a-5f7832de615d reconciler=pgBackRest version=5.5.1-0-amd64
time="2024-05-02T11:16:35Z" level=debug msg=Warning message="command terminated with exit code 49: ERROR: [049]: unable to get address for 'db1-repo-host-0.db1-pods.demo-database.svc.kubernetes.default.': [-2] Name or service not known\n" object="{PostgresCluster demo-database db1 82161687-e0cd-4fca-9387-a60c99006af5 postgres-operator.crunchydata.com/v1beta1 2382014423 }" reason=UnableToCreateStanzas version=5.5.1-0-amd64
time="2024-05-02T11:16:35Z" level=debug msg="reconciled cluster" controller=postgrescluster controllerGroup=postgres-operator.crunchydata.com controllerKind=PostgresCluster name=db1 namespace=demo-database postgresCluster=demo-database/db1 reconcileID=54b678ea-2f9f-409e-864a-5f7832de615d version=5.5.1-0-amd64
time="2024-05-02T11:16:35Z" level=debug msg="patched cluster status" controller=postgrescluster controllerGroup=postgres-operator.crunchydata.com controllerKind=PostgresCluster name=db1 namespace=demo-database postgresCluster=demo-database/db1 reconcileID=54b678ea-2f9f-409e-864a-5f7832de615d version=5.5.1-0-amd64
time="2024-05-02T11:16:35Z" level=debug msg="replaced configuration" controller=postgrescluster controllerGroup=postgres-operator.crunchydata.com controllerKind=PostgresCluster name=db1 namespace=demo-database postgresCluster=demo-database/db1 reconcileID=d31d4913-b91e-41f6-a144-6ffc2a3a2444 stderr= stdout="Not changed\n" version=5.5.1-0-amd64
time="2024-05-02T11:16:35Z" level=debug msg="reconciled instance" controller=postgrescluster controllerGroup=postgres-operator.crunchydata.com controllerKind=PostgresCluster instance=db1-instance1-k5mj name=db1 namespace=demo-database postgresCluster=demo-database/db1 reconcileID=d31d4913-b91e-41f6-a144-6ffc2a3a2444 version=5.5.1-0-amd64
time="2024-05-02T11:16:35Z" level=debug msg="reconciled instance set" controller=postgrescluster controllerGroup=postgres-operator.crunchydata.com controllerKind=PostgresCluster instance-set=instance1 name=db1 namespace=demo-database postgresCluster=demo-database/db1 reconcileID=d31d4913-b91e-41f6-a144-6ffc2a3a2444 version=5.5.1-0-amd64
time="2024-05-02T11:17:36Z" level=error msg="unable to create stanza" controller=postgrescluster controllerGroup=postgres-operator.crunchydata.com controllerKind=PostgresCluster error="command terminated with exit code 49: ERROR: [049]: unable to get address for 'db1-repo-host-0.db1-pods.demo-database.svc.kubernetes.default.': [-2] Name or service not known\n" file="internal/controller/postgrescluster/pgbackrest.go:2618" func="postgrescluster.(*Reconciler).reconcileStanzaCreate" name=db1 namespace=demo-database postgresCluster=demo-database/db1 reconcileID=d31d4913-b91e-41f6-a144-6ffc2a3a2444 reconciler=pgBackRest version=5.5.1-0-amd64
time="2024-05-02T11:17:36Z" level=debug msg="reconciled cluster" controller=postgrescluster controllerGroup=postgres-operator.crunchydata.com controllerKind=PostgresCluster name=db1 namespace=demo-database postgresCluster=demo-database/db1 reconcileID=d31d4913-b91e-41f6-a144-6ffc2a3a2444 version=5.5.1-0-amd64
time="2024-05-02T11:17:36Z" level=debug msg=Warning message="command terminated with exit code 49: ERROR: [049]: unable to get address for 'db1-repo-host-0.db1-pods.demo-database.svc.kubernetes.default.': [-2] Name or service not known\n" object="{PostgresCluster demo-database db1 82161687-e0cd-4fca-9387-a60c99006af5 postgres-operator.crunchydata.com/v1beta1 2382020535 }" reason=UnableToCreateStanzas version=5.5.1-0-amd64

Additional information

Please, see below the description of the postgres cluster:

Name:         db1
Namespace:    demo-database
Labels:       <none>
Annotations:  <none>
API Version:  postgres-operator.crunchydata.com/v1beta1
Kind:         PostgresCluster
Metadata:
  Creation Timestamp:  2024-05-02T11:13:07Z
  Finalizers:
    postgres-operator.crunchydata.com/finalizer
  Generation:  1
  Managed Fields:
    API Version:  postgres-operator.crunchydata.com/v1beta1
    Fields Type:  FieldsV1
    fieldsV1:
      f:metadata:
        f:annotations:
          .:
          f:kubectl.kubernetes.io/last-applied-configuration:
      f:spec:
        .:
        f:backups:
          .:
          f:pgbackrest:
            .:
            f:global:
              .:
              f:repo1-retention-full:
              f:repo1-retention-full-type:
            f:repos:
              .:
              k:{"name":"repo1"}:
                .:
                f:name:
                f:schedules:
                  .:
                  f:full:
                f:volume:
                  .:
                  f:volumeClaimSpec:
                    .:
                    f:accessModes:
                    f:resources:
                      .:
                      f:requests:
                        .:
                        f:storage:
        f:instances:
          .:
          k:{"name":"instance1"}:
            .:
            f:dataVolumeClaimSpec:
              .:
              f:accessModes:
              f:resources:
                .:
                f:requests:
                  .:
                  f:storage:
            f:name:
            f:replicas:
        f:patroni:
          .:
          f:dynamicConfiguration:
            .:
            f:postgresql:
              .:
              f:parameters:
                .:
                f:archive_mode:
                f:archive_timeout:
                f:max_connections:
                f:max_parallel_workers:
                f:max_worker_processes:
                f:shared_buffers:
                f:shared_preload_libraries:
                f:work_mem:
          f:leaderLeaseDurationSeconds:
          f:port:
          f:syncPeriodSeconds:
        f:port:
        f:postgresVersion:
        f:users:
          .:
          k:{"name":"postgres"}:
            .:
            f:name:
    Manager:      kubectl-client-side-apply
    Operation:    Update
    Time:         2024-05-02T11:13:07Z
    API Version:  postgres-operator.crunchydata.com/v1beta1
    Fields Type:  FieldsV1
    fieldsV1:
      f:metadata:
        f:finalizers:
          .:
          v:"postgres-operator.crunchydata.com/finalizer":
    Manager:      postgrescluster-controller
    Operation:    Update
    Time:         2024-05-02T11:13:07Z
    API Version:  postgres-operator.crunchydata.com/v1beta1
    Fields Type:  FieldsV1
    fieldsV1:
      f:status:
        .:
        f:conditions:
          .:
          k:{"type":"PGBackRestReplicaCreate"}:
            .:
            f:lastTransitionTime:
            f:message:
            f:observedGeneration:
            f:reason:
            f:status:
            f:type:
          k:{"type":"PGBackRestReplicaRepoReady"}:
            .:
            f:lastTransitionTime:
            f:message:
            f:observedGeneration:
            f:reason:
            f:status:
            f:type:
          k:{"type":"PGBackRestRepoHostReady"}:
            .:
            f:lastTransitionTime:
            f:message:
            f:observedGeneration:
            f:reason:
            f:status:
            f:type:
        f:databaseRevision:
        f:instances:
          .:
          k:{"name":"instance1"}:
            .:
            f:name:
            f:readyReplicas:
            f:replicas:
            f:updatedReplicas:
        f:monitoring:
          .:
          f:exporterConfiguration:
        f:observedGeneration:
        f:patroni:
          .:
          f:systemIdentifier:
        f:pgbackrest:
          .:
          f:repoHost:
            .:
            f:apiVersion:
            f:kind:
            f:ready:
          f:repos:
            .:
            k:{"name":"repo1"}:
              .:
              f:bound:
              f:name:
              f:stanzaCreated:
              f:volume:
        f:proxy:
          .:
          f:pgBouncer:
            .:
            f:postgresRevision:
        f:usersRevision:
    Manager:         postgrescluster-controller
    Operation:       Update
    Subresource:     status
    Time:            2024-05-02T11:16:35Z
  Resource Version:  2382020535
  UID:               82161687-e0cd-4fca-9387-a60c99006af5
Spec:
  Backups:
    Pgbackrest:
      Global:
        repo1-retention-full:       4
        repo1-retention-full-type:  time
      Repos:
        Name:  repo1
        Schedules:
          Full:  10 1 * * *
        Volume:
          Volume Claim Spec:
            Access Modes:
              ReadWriteOnce
            Resources:
              Requests:
                Storage:  50Gi
  Instances:
    Data Volume Claim Spec:
      Access Modes:
        ReadWriteOnce
      Resources:
        Requests:
          Storage:  100Gi
    Name:           instance1
    Replicas:       1
  Patroni:
    Dynamic Configuration:
      Postgresql:
        Parameters:
          archive_mode:              on
          archive_timeout:           60
          max_connections:           1000
          max_parallel_workers:      2
          max_worker_processes:      2
          shared_buffers:            1GB
          shared_preload_libraries:  timescaledb
          work_mem:                  2MB
    Leader Lease Duration Seconds:   30
    Port:                            8008
    Sync Period Seconds:             10
  Port:                              5432
  Postgres Version:                  16
  Users:
    Name:  postgres
Status:
  Conditions:
    Last Transition Time:  2024-05-02T11:15:34Z
    Message:               pgBackRest dedicated repository host is ready
    Observed Generation:   1
    Reason:                RepoHostReady
    Status:                True
    Type:                  PGBackRestRepoHostReady
    Last Transition Time:  2024-05-02T11:13:08Z
    Message:               pgBackRest replica create repo is not ready for backups
    Observed Generation:   1
    Reason:                StanzaNotCreated
    Status:                False
    Type:                  PGBackRestReplicaRepoReady
    Last Transition Time:  2024-05-02T11:13:08Z
    Message:               pgBackRest replica creation is not currently possible
    Observed Generation:   1
    Reason:                RepoBackupNotComplete
    Status:                False
    Type:                  PGBackRestReplicaCreate
  Database Revision:       559678bf8f
  Instances:
    Name:              instance1
    Ready Replicas:    1
    Replicas:          1
    Updated Replicas:  1
  Monitoring:
    Exporter Configuration:  559c4c97d6
  Observed Generation:       1
  Patroni:
    System Identifier:  7364358837055205488
  Pgbackrest:
    Repo Host:
      API Version:  apps/v1
      Kind:         StatefulSet
      Ready:        true
    Repos:
      Bound:           true
      Name:            repo1
      Stanza Created:  false
      Volume:          d-uf67s7p478o4zd3goidr
  Proxy:
    Pg Bouncer:
      Postgres Revision:  5c9966f6bc
  Users Revision:         78b5654fb8
Events:
  Type     Reason                 Age                 From                        Message
  ----     ------                 ----                ----                        -------
  Normal   RepoHostCreated        28m                 postgrescluster-controller  created pgBackRest repository host StatefulSet/db1-repo-host
  Warning  UnableToCreateStanzas  53s (x27 over 27m)  postgrescluster-controller  command terminated with exit code 49: ERROR: [049]: unable to get address for 'db1-repo-host-0.db1-pods.demo-database.svc.kubernetes.default.': [-2] Name or service not known

burigolucas avatar May 02 '24 11:05 burigolucas

Hi @burigolucas! This error appears to be due to a network issue within your Kubernetes cluster.

More specifically, the address for accessing the pgBackRest repo host Service per the logs and kubectl outputs you provided (specifically db1-repo-host-0.db1-pods.demo-database.svc.kubernetes.default.) appears to be valid. I specifically tried the spec your provided in a few different Kubernetes envs, and I was unable to reproduce this same issue. I.e, db1-repo-host-0.db1-pods.demo-database.svc.kubernetes.default. resolved just fine, and the pgBackRest stanza was created successfully.

I therefore suggest checking out the Kubernetes docs for troubleshooting DNS to get the bottom of the network issue within your env:

https://kubernetes.io/docs/tasks/administer-cluster/dns-debugging-resolution/

This does not appear to be an issue with CPK itself, since CPK is forming the proper DNS name for the Service per the DNS for Services and Pod docs.

Hope this helps!

andrewlecuyer avatar Jun 03 '24 22:06 andrewlecuyer

Hi @burigolucas, I just wanted to check in on this. Have you been able to solve this issue with the help of the above comment or is this still an ongoing concern?

benjaminjb avatar Jun 14 '24 22:06 benjaminjb

Since we haven't heard back on this issue for some time, I am closing this issue. If you need further assistance, feel free to re-open this issue or ask a question in our Discord server.

dsessler7 avatar Jul 22 '24 22:07 dsessler7