kubeblocks icon indicating copy to clipboard operation
kubeblocks copied to clipboard
verified publisher icon apecloud

[BUG] mongodb-4.4 cluster restore volume-snapshot or datafile backup failed

Open JashBook opened this issue 1 year ago • 1 comments

Describe the bug A clear and concise description of what the bug is.

kbcli version
Kubernetes: v1.26.6-gke.1700
KubeBlocks: 0.8.1-beta.4
kbcli: 0.8.0

To Reproduce Steps to reproduce the behavior:

  1. create cluster with mongodb-4.4
kbcli cluster create  mongo-cluster --termination-policy=WipeOut --cluster-definition=mongodb --cluster-version=mongodb-4.4 --set cpu=100m,memory=0.5Gi,replicas=1,storage=3Gi
  1. hscale cluster
kbcli cluster hscale mongo-cluster --replicas 3 --components mongodb --auto-approve
  1. do volume-snapshot or datafile backup
kbcli cluster backup  mongo-cluster --method volume-snapshot
kbcli cluster backup  mongo-cluster --method datafile
  1. restore cluster
kubectl get backup
NAME                                            POLICY                                  METHOD            REPO                    STATUS      TOTAL-SIZE   DURATION   CREATION-TIME          COMPLETION-TIME        EXPIRATION-TIME
backup-default-mongo-cluster-20240119123625     mongo-cluster-mongodb-backup-policy     volume-snapshot                           Completed   3Gi          25s        2024-01-19T04:36:29Z   2024-01-19T04:36:54Z   

kbcli cluster restore mongo-cluster-bk1 --backup backup-default-mongo-cluster-20240119123625
  1. See error
kubectl get pod -l app.kubernetes.io/instance=mongo-cluster-bk1
NAME                          READY   STATUS    RESTARTS        AGE
mongo-cluster-bk1-mongodb-0   3/3     Running   1 (3m41s ago)   5m36s
mongo-cluster-bk1-mongodb-1   3/3     Running   0               5m36s
mongo-cluster-bk1-mongodb-2   3/3     Running   0               5m36s
➜  ~ 
➜  ~ kubectl get cluster mongo-cluster-bk1
NAME                CLUSTER-DEFINITION   VERSION       TERMINATION-POLICY   STATUS     AGE
mongo-cluster-bk1   mongodb              mongodb-4.4   WipeOut              Creating   7m32s

logs pod

kubectl logs mongo-cluster-bk1-mongodb-0 mongodb --previous
Error: couldn't connect to server 127.0.0.1:27027, connection attempt failed: SocketException: Error connecting to 127.0.0.1:27027 :: caused by :: Connection refused :
connect@src/mongo/shell/mongo.js:374:17
@(connect):3:6
exception: connect failed
exiting with code 1
Error: couldn't connect to server 127.0.0.1:27027, connection attempt failed: SocketException: Error connecting to 127.0.0.1:27027 :: caused by :: Connection refused :
connect@src/mongo/shell/mongo.js:374:17
@(connect):3:6
exception: connect failed
exiting with code 1
Error: couldn't connect to server 127.0.0.1:27027, connection attempt failed: SocketException: Error connecting to 127.0.0.1:27027 :: caused by :: Connection refused :
connect@src/mongo/shell/mongo.js:374:17
@(connect):3:6
exception: connect failed
exiting with code 1
Error: couldn't connect to server 127.0.0.1:27027, connection attempt failed: SocketException: Error connecting to 127.0.0.1:27027 :: caused by :: Connection refused :
connect@src/mongo/shell/mongo.js:374:17
@(connect):3:6
exception: connect failed
exiting with code 1
Error: couldn't connect to server 127.0.0.1:27027, connection attempt failed: SocketException: Error connecting to 127.0.0.1:27027 :: caused by :: Connection refused :
connect@src/mongo/shell/mongo.js:374:17
@(connect):3:6
exception: connect failed
exiting with code 1
Error: couldn't connect to server 127.0.0.1:27027, connection attempt failed: SocketException: Error connecting to 127.0.0.1:27027 :: caused by :: Connection refused :
connect@src/mongo/shell/mongo.js:374:17
@(connect):3:6
exception: connect failed
exiting with code 1
restore process is ready
{ "acknowledged" : true, "deletedCount" : 1 }
true
INFO: restore set-up configuration successfully.

kubectl logs mongo-cluster-bk1-mongodb-0 kb-checkrole
2024-01-19T04:48:07Z	INFO	HA	Create Root.
2024-01-19T04:48:07Z	INFO	MongoDB	Create user: root, passwd: pn7979km, roles: map[db:admin role:root]
2024-01-19T04:48:07Z	INFO	MongoDB	Create Root failed	{"error": "failed to create user: (NotWritablePrimary) not primary", "errorVerbose": "(NotWritablePrimary) not primary\nfailed to create user\ngithub.com/apecloud/kubeblocks/pkg/lorry/engines/mongodb.CreateUser\n\t/src/pkg/lorry/engines/mongodb/users.go:39\ngithub.com/apecloud/kubeblocks/pkg/lorry/engines/mongodb.(*Manager).CreateRoot\n\t/src/pkg/lorry/engines/mongodb/manager.go:246\ngithub.com/apecloud/kubeblocks/pkg/lorry/highavailability.(*Ha).Start\n\t/src/pkg/lorry/highavailability/ha.go:258\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1650"}
2024-01-19T04:48:07Z	ERROR	HA	Cluster initialize failed	{"error": "failed to create user: (NotWritablePrimary) not primary", "errorVerbose": "(NotWritablePrimary) not primary\nfailed to create user\ngithub.com/apecloud/kubeblocks/pkg/lorry/engines/mongodb.CreateUser\n\t/src/pkg/lorry/engines/mongodb/users.go:39\ngithub.com/apecloud/kubeblocks/pkg/lorry/engines/mongodb.(*Manager).CreateRoot\n\t/src/pkg/lorry/engines/mongodb/manager.go:246\ngithub.com/apecloud/kubeblocks/pkg/lorry/highavailability.(*Ha).Start\n\t/src/pkg/lorry/highavailability/ha.go:258\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1650"}
github.com/apecloud/kubeblocks/pkg/lorry/highavailability.(*Ha).Start
	/src/pkg/lorry/highavailability/ha.go:260
2024-01-19T04:48:08Z	INFO	MongoDB	DB is not ready	{"error": "connection() error occurred during connection handshake: auth error: sasl conversation error: unable to authenticate using mechanism \"SCRAM-SHA-1\": (AuthenticationFailed) Authentication failed."}
2024-01-19T04:48:10Z	INFO	MongoDB	DB is not ready	{"error": "connection() error occurred during connection handshake: auth error: sasl conversation error: unable to authenticate using mechanism \"SCRAM-SHA-1\": (AuthenticationFailed) Authentication failed."}
2024-01-19T04:48:12Z	INFO	MongoDB	DB is not ready	{"error": "connection() error occurred during connection handshake: auth error: sasl conversation error: unable to authenticate using mechanism \"SCRAM-SHA-1\": (AuthenticationFailed) Authentication failed."}
2024-01-19T04:48:12Z	INFO	HA	Create Root.
2024-01-19T04:48:12Z	INFO	MongoDB	Create user: root, passwd: pn7979km, roles: map[db:admin role:root]
2024-01-19T04:48:12Z	INFO	MongoDB	Create Root failed	{"error": "failed to create user: (NotWritablePrimary) not primary", "errorVerbose": "(NotWritablePrimary) not primary\nfailed to create user\ngithub.com/apecloud/kubeblocks/pkg/lorry/engines/mongodb.CreateUser\n\t/src/pkg/lorry/engines/mongodb/users.go:39\ngithub.com/apecloud/kubeblocks/pkg/lorry/engines/mongodb.(*Manager).CreateRoot\n\t/src/pkg/lorry/engines/mongodb/manager.go:246\ngithub.com/apecloud/kubeblocks/pkg/lorry/highavailability.(*Ha).Start\n\t/src/pkg/lorry/highavailability/ha.go:258\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1650"}
2024-01-19T04:48:12Z	ERROR	HA	Cluster initialize failed	{"error": "failed to create user: (NotWritablePrimary) not primary", "errorVerbose": "(NotWritablePrimary) not primary\nfailed to create user\ngithub.com/apecloud/kubeblocks/pkg/lorry/engines/mongodb.CreateUser\n\t/src/pkg/lorry/engines/mongodb/users.go:39\ngithub.com/apecloud/kubeblocks/pkg/lorry/engines/mongodb.(*Manager).CreateRoot\n\t/src/pkg/lorry/engines/mongodb/manager.go:246\ngithub.com/apecloud/kubeblocks/pkg/lorry/highavailability.(*Ha).Start\n\t/src/pkg/lorry/highavailability/ha.go:258\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1650"}
github.com/apecloud/kubeblocks/pkg/lorry/highavailability.(*Ha).Start
	/src/pkg/lorry/highavailability/ha.go:260
2024-01-19T04:48:14Z	INFO	MongoDB	DB is not ready	{"error": "connection() error occurred during connection handshake: auth error: sasl conversation error: unable to authenticate using mechanism \"SCRAM-SHA-1\": (AuthenticationFailed) Authentication failed."}
2024-01-19T04:48:16Z	INFO	MongoDB	DB is not ready	{"error": "connection() error occurred during connection handshake: auth error: sasl conversation error: unable to authenticate using mechanism \"SCRAM-SHA-1\": (AuthenticationFailed) Authentication failed."}
2024-01-19T04:48:17Z	INFO	HA	Create Root.
2024-01-19T04:48:17Z	INFO	MongoDB	Create user: root, passwd: pn7979km, roles: map[db:admin role:root]
2024-01-19T04:48:17Z	INFO	MongoDB	Create Root failed	{"error": "failed to create user: (NotWritablePrimary) not primary", "errorVerbose": "(NotWritablePrimary) not primary\nfailed to create user\ngithub.com/apecloud/kubeblocks/pkg/lorry/engines/mongodb.CreateUser\n\t/src/pkg/lorry/engines/mongodb/users.go:39\ngithub.com/apecloud/kubeblocks/pkg/lorry/engines/mongodb.(*Manager).CreateRoot\n\t/src/pkg/lorry/engines/mongodb/manager.go:246\ngithub.com/apecloud/kubeblocks/pkg/lorry/highavailability.(*Ha).Start\n\t/src/pkg/lorry/highavailability/ha.go:258\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1650"}
2024-01-19T04:48:17Z	ERROR	HA	Cluster initialize failed	{"error": "failed to create user: (NotWritablePrimary) not primary", "errorVerbose": "(NotWritablePrimary) not primary\nfailed to create user\ngithub.com/apecloud/kubeblocks/pkg/lorry/engines/mongodb.CreateUser\n\t/src/pkg/lorry/engines/mongodb/users.go:39\ngithub.com/apecloud/kubeblocks/pkg/lorry/engines/mongodb.(*Manager).CreateRoot\n\t/src/pkg/lorry/engines/mongodb/manager.go:246\ngithub.com/apecloud/kubeblocks/pkg/lorry/highavailability.(*Ha).Start\n\t/src/pkg/lorry/highavailability/ha.go:258\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1650"}
github.com/apecloud/kubeblocks/pkg/lorry/highavailability.(*Ha).Start
	/src/pkg/lorry/highavailability/ha.go:260
2024-01-19T04:48:18Z	INFO	MongoDB	DB is not ready	{"error": "connection() error occurred during connection handshake: auth error: sasl conversation error: unable to authenticate using mechanism \"SCRAM-SHA-1\": (AuthenticationFailed) Authentication failed."}
2024-01-19T04:48:20Z	INFO	MongoDB	DB is not ready	{"error": "connection() error occurred during connection handshake: auth error: sasl conversation error: unable to authenticate using mechanism \"SCRAM-SHA-1\": (AuthenticationFailed) Authentication failed."}

Expected behavior mongodb-4.4 cluster restore volume-snapshot or datafile backup success .

Screenshots If applicable, add screenshots to help explain your problem.

Desktop (please complete the following information):

  • OS: [e.g. iOS]
  • Browser [e.g. chrome, safari]
  • Version [e.g. 22]

Additional context Add any other context about the problem here.

JashBook avatar Jan 19 '24 04:01 JashBook

This issue has been marked as stale because it has been open for 30 days with no activity

github-actions[bot] avatar Feb 19 '24 00:02 github-actions[bot]

fix via https://github.com/apecloud/kubeblocks-addons/pull/1996

JashBook avatar Sep 18 '25 07:09 JashBook