Pgbackrest exec fails intermittently

[dcaputo-harmoni]

Report

Exec commands failing due to lack of retry, similar to an issue encountered with the mysql xtradb operator. Looks like the same code in clientcmd.go is present here. Submitting a PR shortly with the same fix.

More about the problem

: exec: : error sending request: Post \"https://10.0.64.1:443/api/v1/namespaces/core-postgres/pods/pg-db-postgres-jms4-0/exec?command=pgbackrest&command=info&command=--output%3Djson&command=--repo%3D1&container=database&stderr=true&stdout=true\": unexpected EOF", "errorVerbose": "error sending request: Post \"https://10.0.64.1:443/api/v1/namespaces/core-postgres/pods/pg-db-postgres-jms4-0/exec?command=pgbackrest&command=info&command=--output%3Djson&command=--repo%3D1&container=database&stderr=true&stdout=true\": unexpected EOF\nexec: \ngithub.com/percona/percona-postgresql-operator/percona/pgbackrest.GetInfo\n\t/go/src/github.com/percona/percona-postgresql-operator/percona/pgbackrest/pgbackrest.go:61\ngithub.com/percona/percona-postgresql-operator/percona/controller/pgcluster.(*PGClusterReconciler).cleanupOutdatedBackups\n\t/go/src/github.com/percona/percona-postgresql-operator/percona/controller/pgcluster/backup.go:60\ngithub.com/percona/percona-postgresql-operator/percona/controller/pgcluster.(*PGClusterReconciler).reconcileBackups\n\t/go/src/github.com/percona/percona-postgresql-operator/percona/controller/pgcluster/backup.go:29\ngithub.com/percona/percona-postgresql-operator/percona/controller/pgcluster.(*PGClusterReconciler).Reconcile\n\t/go/src/github.com/percona/percona-postgresql-operator/percona/controller/pgcluster/controller.go:226\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:114\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:311\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:261\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:222\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1695\nget pgBackRest info\ngithub.com/percona/percona-postgresql-operator/percona/controller/pgcluster.(*PGClusterReconciler).cleanupOutdatedBackups\n\t/go/src/github.com/percona/percona-postgresql-operator/percona/controller/pgcluster/backup.go:66\ngithub.com/percona/percona-postgresql-operator/percona/controller/pgcluster.(*PGClusterReconciler).reconcileBackups\n\t/go/src/github.com/percona/percona-postgresql-operator/percona/controller/pgcluster/backup.go:29\ngithub.com/percona/percona-postgresql-operator/percona/controller/pgcluster.(*PGClusterReconciler).Reconcile\n\t/go/src/github.com/percona/percona-postgresql-operator/percona/controller/pgcluster/controller.go:226\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:114\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:311\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:261\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:222\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1695"}
github.com/percona/percona-postgresql-operator/percona/controller/pgcluster.(*PGClusterReconciler).Reconcile
    /go/src/github.com/percona/percona-postgresql-operator/percona/controller/pgcluster/controller.go:226
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
    /go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:114
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
    /go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:311
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
    /go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:261
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
    /go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:222

Steps to reproduce

Run cluster with pgbackup enabled, observe intermittent errors (in my case on AKS)

Versions

1. Kubernetes 1.30.4
2. Operator 2.4.0
3. Database 16

Anything else?

No response