kubeblocks
kubeblocks copied to clipboard
Published
20 hours ago •
apecloud
apecloud
[BUG] mongodb-4.4 cluster restore volume-snapshot or datafile backup failed
Describe the bug A clear and concise description of what the bug is.
kbcli version
Kubernetes: v1.26.6-gke.1700
KubeBlocks: 0.8.1-beta.4
kbcli: 0.8.0
To Reproduce Steps to reproduce the behavior:
- create cluster with mongodb-4.4
kbcli cluster create mongo-cluster --termination-policy=WipeOut --cluster-definition=mongodb --cluster-version=mongodb-4.4 --set cpu=100m,memory=0.5Gi,replicas=1,storage=3Gi
- hscale cluster
kbcli cluster hscale mongo-cluster --replicas 3 --components mongodb --auto-approve
- do volume-snapshot or datafile backup
kbcli cluster backup mongo-cluster --method volume-snapshot
kbcli cluster backup mongo-cluster --method datafile
- restore cluster
kubectl get backup
NAME POLICY METHOD REPO STATUS TOTAL-SIZE DURATION CREATION-TIME COMPLETION-TIME EXPIRATION-TIME
backup-default-mongo-cluster-20240119123625 mongo-cluster-mongodb-backup-policy volume-snapshot Completed 3Gi 25s 2024-01-19T04:36:29Z 2024-01-19T04:36:54Z
kbcli cluster restore mongo-cluster-bk1 --backup backup-default-mongo-cluster-20240119123625
- See error
kubectl get pod -l app.kubernetes.io/instance=mongo-cluster-bk1
NAME READY STATUS RESTARTS AGE
mongo-cluster-bk1-mongodb-0 3/3 Running 1 (3m41s ago) 5m36s
mongo-cluster-bk1-mongodb-1 3/3 Running 0 5m36s
mongo-cluster-bk1-mongodb-2 3/3 Running 0 5m36s
➜ ~
➜ ~ kubectl get cluster mongo-cluster-bk1
NAME CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS AGE
mongo-cluster-bk1 mongodb mongodb-4.4 WipeOut Creating 7m32s
logs pod
kubectl logs mongo-cluster-bk1-mongodb-0 mongodb --previous
Error: couldn't connect to server 127.0.0.1:27027, connection attempt failed: SocketException: Error connecting to 127.0.0.1:27027 :: caused by :: Connection refused :
connect@src/mongo/shell/mongo.js:374:17
@(connect):3:6
exception: connect failed
exiting with code 1
Error: couldn't connect to server 127.0.0.1:27027, connection attempt failed: SocketException: Error connecting to 127.0.0.1:27027 :: caused by :: Connection refused :
connect@src/mongo/shell/mongo.js:374:17
@(connect):3:6
exception: connect failed
exiting with code 1
Error: couldn't connect to server 127.0.0.1:27027, connection attempt failed: SocketException: Error connecting to 127.0.0.1:27027 :: caused by :: Connection refused :
connect@src/mongo/shell/mongo.js:374:17
@(connect):3:6
exception: connect failed
exiting with code 1
Error: couldn't connect to server 127.0.0.1:27027, connection attempt failed: SocketException: Error connecting to 127.0.0.1:27027 :: caused by :: Connection refused :
connect@src/mongo/shell/mongo.js:374:17
@(connect):3:6
exception: connect failed
exiting with code 1
Error: couldn't connect to server 127.0.0.1:27027, connection attempt failed: SocketException: Error connecting to 127.0.0.1:27027 :: caused by :: Connection refused :
connect@src/mongo/shell/mongo.js:374:17
@(connect):3:6
exception: connect failed
exiting with code 1
Error: couldn't connect to server 127.0.0.1:27027, connection attempt failed: SocketException: Error connecting to 127.0.0.1:27027 :: caused by :: Connection refused :
connect@src/mongo/shell/mongo.js:374:17
@(connect):3:6
exception: connect failed
exiting with code 1
restore process is ready
{ "acknowledged" : true, "deletedCount" : 1 }
true
INFO: restore set-up configuration successfully.
kubectl logs mongo-cluster-bk1-mongodb-0 kb-checkrole
2024-01-19T04:48:07Z INFO HA Create Root.
2024-01-19T04:48:07Z INFO MongoDB Create user: root, passwd: pn7979km, roles: map[db:admin role:root]
2024-01-19T04:48:07Z INFO MongoDB Create Root failed {"error": "failed to create user: (NotWritablePrimary) not primary", "errorVerbose": "(NotWritablePrimary) not primary\nfailed to create user\ngithub.com/apecloud/kubeblocks/pkg/lorry/engines/mongodb.CreateUser\n\t/src/pkg/lorry/engines/mongodb/users.go:39\ngithub.com/apecloud/kubeblocks/pkg/lorry/engines/mongodb.(*Manager).CreateRoot\n\t/src/pkg/lorry/engines/mongodb/manager.go:246\ngithub.com/apecloud/kubeblocks/pkg/lorry/highavailability.(*Ha).Start\n\t/src/pkg/lorry/highavailability/ha.go:258\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1650"}
2024-01-19T04:48:07Z ERROR HA Cluster initialize failed {"error": "failed to create user: (NotWritablePrimary) not primary", "errorVerbose": "(NotWritablePrimary) not primary\nfailed to create user\ngithub.com/apecloud/kubeblocks/pkg/lorry/engines/mongodb.CreateUser\n\t/src/pkg/lorry/engines/mongodb/users.go:39\ngithub.com/apecloud/kubeblocks/pkg/lorry/engines/mongodb.(*Manager).CreateRoot\n\t/src/pkg/lorry/engines/mongodb/manager.go:246\ngithub.com/apecloud/kubeblocks/pkg/lorry/highavailability.(*Ha).Start\n\t/src/pkg/lorry/highavailability/ha.go:258\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1650"}
github.com/apecloud/kubeblocks/pkg/lorry/highavailability.(*Ha).Start
/src/pkg/lorry/highavailability/ha.go:260
2024-01-19T04:48:08Z INFO MongoDB DB is not ready {"error": "connection() error occurred during connection handshake: auth error: sasl conversation error: unable to authenticate using mechanism \"SCRAM-SHA-1\": (AuthenticationFailed) Authentication failed."}
2024-01-19T04:48:10Z INFO MongoDB DB is not ready {"error": "connection() error occurred during connection handshake: auth error: sasl conversation error: unable to authenticate using mechanism \"SCRAM-SHA-1\": (AuthenticationFailed) Authentication failed."}
2024-01-19T04:48:12Z INFO MongoDB DB is not ready {"error": "connection() error occurred during connection handshake: auth error: sasl conversation error: unable to authenticate using mechanism \"SCRAM-SHA-1\": (AuthenticationFailed) Authentication failed."}
2024-01-19T04:48:12Z INFO HA Create Root.
2024-01-19T04:48:12Z INFO MongoDB Create user: root, passwd: pn7979km, roles: map[db:admin role:root]
2024-01-19T04:48:12Z INFO MongoDB Create Root failed {"error": "failed to create user: (NotWritablePrimary) not primary", "errorVerbose": "(NotWritablePrimary) not primary\nfailed to create user\ngithub.com/apecloud/kubeblocks/pkg/lorry/engines/mongodb.CreateUser\n\t/src/pkg/lorry/engines/mongodb/users.go:39\ngithub.com/apecloud/kubeblocks/pkg/lorry/engines/mongodb.(*Manager).CreateRoot\n\t/src/pkg/lorry/engines/mongodb/manager.go:246\ngithub.com/apecloud/kubeblocks/pkg/lorry/highavailability.(*Ha).Start\n\t/src/pkg/lorry/highavailability/ha.go:258\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1650"}
2024-01-19T04:48:12Z ERROR HA Cluster initialize failed {"error": "failed to create user: (NotWritablePrimary) not primary", "errorVerbose": "(NotWritablePrimary) not primary\nfailed to create user\ngithub.com/apecloud/kubeblocks/pkg/lorry/engines/mongodb.CreateUser\n\t/src/pkg/lorry/engines/mongodb/users.go:39\ngithub.com/apecloud/kubeblocks/pkg/lorry/engines/mongodb.(*Manager).CreateRoot\n\t/src/pkg/lorry/engines/mongodb/manager.go:246\ngithub.com/apecloud/kubeblocks/pkg/lorry/highavailability.(*Ha).Start\n\t/src/pkg/lorry/highavailability/ha.go:258\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1650"}
github.com/apecloud/kubeblocks/pkg/lorry/highavailability.(*Ha).Start
/src/pkg/lorry/highavailability/ha.go:260
2024-01-19T04:48:14Z INFO MongoDB DB is not ready {"error": "connection() error occurred during connection handshake: auth error: sasl conversation error: unable to authenticate using mechanism \"SCRAM-SHA-1\": (AuthenticationFailed) Authentication failed."}
2024-01-19T04:48:16Z INFO MongoDB DB is not ready {"error": "connection() error occurred during connection handshake: auth error: sasl conversation error: unable to authenticate using mechanism \"SCRAM-SHA-1\": (AuthenticationFailed) Authentication failed."}
2024-01-19T04:48:17Z INFO HA Create Root.
2024-01-19T04:48:17Z INFO MongoDB Create user: root, passwd: pn7979km, roles: map[db:admin role:root]
2024-01-19T04:48:17Z INFO MongoDB Create Root failed {"error": "failed to create user: (NotWritablePrimary) not primary", "errorVerbose": "(NotWritablePrimary) not primary\nfailed to create user\ngithub.com/apecloud/kubeblocks/pkg/lorry/engines/mongodb.CreateUser\n\t/src/pkg/lorry/engines/mongodb/users.go:39\ngithub.com/apecloud/kubeblocks/pkg/lorry/engines/mongodb.(*Manager).CreateRoot\n\t/src/pkg/lorry/engines/mongodb/manager.go:246\ngithub.com/apecloud/kubeblocks/pkg/lorry/highavailability.(*Ha).Start\n\t/src/pkg/lorry/highavailability/ha.go:258\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1650"}
2024-01-19T04:48:17Z ERROR HA Cluster initialize failed {"error": "failed to create user: (NotWritablePrimary) not primary", "errorVerbose": "(NotWritablePrimary) not primary\nfailed to create user\ngithub.com/apecloud/kubeblocks/pkg/lorry/engines/mongodb.CreateUser\n\t/src/pkg/lorry/engines/mongodb/users.go:39\ngithub.com/apecloud/kubeblocks/pkg/lorry/engines/mongodb.(*Manager).CreateRoot\n\t/src/pkg/lorry/engines/mongodb/manager.go:246\ngithub.com/apecloud/kubeblocks/pkg/lorry/highavailability.(*Ha).Start\n\t/src/pkg/lorry/highavailability/ha.go:258\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1650"}
github.com/apecloud/kubeblocks/pkg/lorry/highavailability.(*Ha).Start
/src/pkg/lorry/highavailability/ha.go:260
2024-01-19T04:48:18Z INFO MongoDB DB is not ready {"error": "connection() error occurred during connection handshake: auth error: sasl conversation error: unable to authenticate using mechanism \"SCRAM-SHA-1\": (AuthenticationFailed) Authentication failed."}
2024-01-19T04:48:20Z INFO MongoDB DB is not ready {"error": "connection() error occurred during connection handshake: auth error: sasl conversation error: unable to authenticate using mechanism \"SCRAM-SHA-1\": (AuthenticationFailed) Authentication failed."}
Expected behavior mongodb-4.4 cluster restore volume-snapshot or datafile backup success .
Screenshots If applicable, add screenshots to help explain your problem.
Desktop (please complete the following information):
- OS: [e.g. iOS]
- Browser [e.g. chrome, safari]
- Version [e.g. 22]
Additional context Add any other context about the problem here.
