percona-xtradb-cluster-operator Cannot Restore a Backup from another pcx-db cluster on v1.16.1

Report

There is an issue when restoring a backup from one cluster to another cluster with a different name.

More about the problem

I have 2 pxc-db clusters named: mysql2-pxc-db and mysql3-pxc-db - both with the same specification. A backup cron-mysql2-pxc-db-s3-backup-bucket-2025220104054-1eb8b was created for mysql2-pxc-db cluster.

When I attempt to restore this backup to the mysql3-pxc-db cluster using the following definition:

apiVersion: pxc.percona.com/v1 kind: PerconaXtraDBClusterRestore metadata: name: restore-from-mysql2-to-mysql3 namespace: persistence spec: pxcCluster: mysql3-pxc-db backupSource: verifyTLS: true destination: s3://bkd1-database-mysql2-backup/mysql2-pxc-db-2025-02-20-10:40:54-full s3: bucket: bkd1-database-mysql2-backup credentialsSecret: mysql1-s3-backup-source endpointUrl: ....

The restore object is created, but the operator reports the following error:

2025-02-21T11:53:38.824Z INFO backup restore request {"controller": "pxcrestore-controller", "namespace": "persistence", "name": "restore-from-mysql2-to-mysql3", "reconcileID": "b9167f84-0581-4c60-8e86-69f34eed761d"} 2025-02-21T11:53:38.844Z ERROR Observed a panic {"controller": "pxcrestore-controller", "namespace": "persistence", "name": "restore-from-mysql2-to-mysql3", "reconcileID": "b9167f84-0581-4c60-8e86-69f34eed761d", "panic": "runtime error: invalid memory address or nil pointer dereference", "panicGoValue": "\"invalid memory address or nil pointer dereference\"", "stacktrace": "goroutine 175 [running]:\nk8s.io/apimachinery/pkg/util/runtime.logPanic({0x23a92c8, 0xc0015ea1b0}, {0x1d5a6a0, 0x33bc7c0})\n\t/go/pkg/mod/k8s.io/[email protected]/pkg/util/runtime/runtime.go:107 +0xbc\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile.func1()\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:105 +0x112\npanic({0x1d5a6a0?, 0x33bc7c0?})\n\t/usr/local/go/src/runtime/panic.go:785 +0x132\ngithub.com/percona/percona-xtradb-cluster-operator/pkg/naming.LabelsRestoreJob(0xc000f6ea08, {0xc001638e40, 0x37}, {0x0, 0x0})\n\t/go/src/github.com/percona/percona-xtradb-cluster-operator/pkg/naming/labels.go:98 +0x79\ngithub.com/percona/percona-xtradb-cluster-operator/pkg/pxc/backup.RestoreJob(0xc001336d00, 0xc00180b448, 0xc000f6ea08, {0xc000800e10, 0x2e}, {0xc000f250e0, 0x47}, 0x0)\n\t/go/src/github.com/percona/percona-xtradb-cluster-operator/pkg/pxc/backup/restore.go:271 +0x1565\ngithub.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxcrestore.(*s3).Job(0xc000a98cc0?)\n\t/go/src/github.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxcrestore/restorer.go:39 +0x3a\ngithub.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxcrestore.(*ReconcilePerconaXtraDBClusterRestore).validate(0xc000f6ea08?, {0x23a92c8, 0xc0015ea1b0}, 0xc001336d00, 0x23a93a8?, 0x341cb20?)\n\t/go/src/github.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxcrestore/restore.go:80 +0x42\ngithub.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxcrestore.(*ReconcilePerconaXtraDBClusterRestore).Reconcile(0xc000a98cc0, {0x23a92c8, 0xc0015ea1b0}, {{{0xc001492250?, 0x202909a?}, {0xc001664600?, 0x100?}}})\n\t/go/src/github.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxcrestore/controller.go:189 +0xe8e\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile(0xc0015ea120?, {0x23a92c8?, 0xc0015ea1b0?}, {{{0xc001492250?, 0x0?}, {0xc001664600?, 0x0?}}})\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:116 +0xbf\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler(0x23b9b20, {0x23a9300, 0xc000878140}, {{{0xc001492250, 0xb}, {0xc001664600, 0x1d}}})\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:303 +0x3a5\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem(0x23b9b20, {0x23a9300, 0xc000878140})\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:263 +0x20e\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2()\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:224 +0x85\ncreated by sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2 in goroutine 135\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:220 +0x490\n"} runtime.sigpanic /usr/local/go/src/runtime/signal_unix.go:917 github.com/percona/percona-xtradb-cluster-operator/pkg/naming.LabelsRestoreJob /go/src/github.com/percona/percona-xtradb-cluster-operator/pkg/naming/labels.go:98 github.com/percona/percona-xtradb-cluster-operator/pkg/pxc/backup.RestoreJob /go/src/github.com/percona/percona-xtradb-cluster-operator/pkg/pxc/backup/restore.go:271 github.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxcrestore.(*s3).Job /go/src/github.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxcrestore/restorer.go:39 github.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxcrestore.(*ReconcilePerconaXtraDBClusterRestore).validate /go/src/github.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxcrestore/restore.go:80 github.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxcrestore.(*ReconcilePerconaXtraDBClusterRestore).Reconcile /go/src/github.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxcrestore/controller.go:189 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:116 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:303 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:263 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2 /go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:224 2025-02-21T11:53:38.844Z ERROR Reconciler error {"controller": "pxcrestore-controller", "namespace": "persistence", "name": "restore-from-mysql2-to-mysql3", "reconcileID": "b9167f84-0581-4c60-8e86-69f34eed761d", "error": "panic: runtime error: invalid memory address or nil pointer dereference [recovered]"} sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:316 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:263 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2 /go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:224

As a result, the restore object is being updated with status:

status: comments: | You can view xtrabackup log: $ kubectl logs job/restore-job-restore-from-mysql2-to-mysql3-mysql3-pxc-db If everything is fine, you can cleanup the job: $ kubectl delete pxc-restore/restore-from-mysql2-to-mysql3 completed: '2025-02-21T11:53:38Z' state: Succeeded

Steps to reproduce

Create clusters mysql2 and mysql3
Create a backup of mysql2
Try to restore the backup to mysql3

Versions

Operator: v1.16.1

Anything else?

The restore works when targeting a cluster with the same name as the original, even on a different Kubernetes cluster.

Feb 21 '25 12:02 bkd231

@bkd231 Do you have a storage defined for the additional cluster? https://github.com/percona/percona-xtradb-cluster-operator/issues/1749#issuecomment-2214282438

Mar 05 '25 16:03 ydixken

The "fix" is indeed to define the same storage for the cluster where you want to restore towards, with the same info as the originating cluster. Kind of silly, certainly as you have to supply everything for the restore (bucket, path, name, credentials, ...), so there's is no reason as to why it needs to read the storage section again...

Mar 11 '25 13:03 wonko

maybe this issue can be solved together with this: https://github.com/percona/percona-xtradb-cluster-operator/issues/1620

Sep 19 '25 08:09 gkech

Hey again, we prepated a ticket so that we can see priorities: https://perconadev.atlassian.net/browse/K8SPXC-1715

Sep 19 '25 14:09 gkech