kubeblocks icon indicating copy to clipboard operation
kubeblocks copied to clipboard

Published 20 hours ago • verified publisher icon apecloud

[BUG]Upgraded from 0.6.4 to 0.7 Mongo restored cluster role are none

Open ahjing99 opened this issue 1 year ago • 2 comments

  1. Install kb 0.6.4
  2. create mongo
  3. upgrade kb from 0.6.4 to 0.7
  4. create backup and restore

      `kbcli cluster backup mongo-jrsita --method datafile --namespace default `

Backup backup-default-mongo-jrsita-20231103101059 created successfully, you can view the progress:
	kbcli cluster list-backups --name=backup-default-mongo-jrsita-20231103101059 -n default


      `kbcli cluster describe-backup backup-default-mongo-jrsita-20231103101059 --namespace default `

Name: backup-default-mongo-jrsita-20231103101059	Cluster: mongo-jrsita	Namespace: default

Spec:
  Method:             datafile
  Policy Name:        mongo-jrsita-mongodb-backup-policy

Status:
  Phase:              Completed
  Total Size:         8413718
  ActionSet Name:     mongodb-physical-backup
  Repository:         backuprepo-kbcli-test
  Duration:           16s
  Start Time:         Nov 03,2023 10:11 UTC+0800
  Completion Time:    Nov 03,2023 10:11 UTC+0800
  Path:               /default/mongo-jrsita-19173ef5-746a-4e67-8d97-ea7e8e728103/mongodb/backup-default-mongo-jrsita-20231103101059
  Time Range Start:   Nov 03,2023 10:10 UTC+0800
  Time Range End:     Nov 03,2023 10:11 UTC+0800

Warning Events: <none>

      `kbcli cluster restore mongo-jrsita-backup --backup backup-default-mongo-jrsita-20231103101059 --namespace default `

Cluster mongo-jrsita-backup created
  1. cluster is always creating
➜  ~ kbcli cluster describe mongo-jrsita-backup
Name: mongo-jrsita-backup	 Created Time: Nov 03,2023 10:11 UTC+0800
NAMESPACE   CLUSTER-DEFINITION   VERSION       STATUS     TERMINATION-POLICY
default     mongodb              mongodb-4.4   Creating   WipeOut

Endpoints:
COMPONENT   MODE        INTERNAL                                                      EXTERNAL
mongodb     ReadWrite   mongo-jrsita-backup-mongodb.default.svc.cluster.local:27017   <none>

Topology:
COMPONENT   INSTANCE                        ROLE     STATUS    AZ              NODE                                                CREATED-TIME
mongodb     mongo-jrsita-backup-mongodb-0   <none>   Running   us-central1-c   gke-yijing-default-pool-3e14ea35-hqtr/10.128.0.30   Nov 03,2023 10:11 UTC+0800
mongodb     mongo-jrsita-backup-mongodb-1   <none>   Running   us-central1-c   gke-yijing-default-pool-3e14ea35-wg54/10.128.0.35   Nov 03,2023 10:11 UTC+0800
mongodb     mongo-jrsita-backup-mongodb-2   <none>   Running   us-central1-c   gke-yijing-default-pool-3e14ea35-hxpl/10.128.0.28   Nov 03,2023 10:11 UTC+0800

Resources Allocation:
COMPONENT   DEDICATED   CPU(REQUEST/LIMIT)   MEMORY(REQUEST/LIMIT)           STORAGE-SIZE   STORAGE-CLASS
mongodb     false       200m / 200m          644245094400m / 644245094400m   data:5Gi       kb-default-sc

Images:
COMPONENT   TYPE      IMAGE
mongodb     mongodb   infracreate-registry.cn-zhangjiakou.cr.aliyuncs.com/apecloud/mongo:4.4

Data Protection:
BACKUP-REPO   AUTO-BACKUP   BACKUP-SCHEDULE   BACKUP-METHOD   BACKUP-RETENTION

Show cluster events: kbcli cluster list-events -n default mongo-jrsita-backup


2023-11-03T02:54:13Z	INFO	Mongo	operation called	{"operation": "checkRole", "result": {"event":"Failed","message":"db not ready"}}
2023-11-03T02:54:13Z	INFO	grpc	No event detected	{"meta": "{\"duration\":\"4.605489ms\",\"end-time\":\"2023-11-03T02:54:13.138057337Z\",\"operation\":\"checkRole\",\"start-time\":\"2023-11-03T02:54:13.133451848Z\"}", "data": "{\"event\":\"Failed\",\"message\":\"db not ready\"}"}
2023-11-03T02:54:14Z	INFO	HA	Create Root.
2023-11-03T02:54:14Z	INFO	Mongo	Create user: root, passwd: srb4lx47, roles: map[db:admin role:root]
2023-11-03T02:54:14Z	ERROR	Mongo	Create Root failed	{"error": "failed to create user: (NotWritablePrimary) not primary", "errorVerbose": "(NotWritablePrimary) not primary\nfailed to create user\ngithub.com/apecloud/kubeblocks/lorry/component/mongodb.CreateUser\n\t/src/lorry/component/mongodb/users.go:39\ngithub.com/apecloud/kubeblocks/lorry/component/mongodb.(*Manager).CreateRoot\n\t/src/lorry/component/mongodb/manager.go:243\ngithub.com/apecloud/kubeblocks/lorry/highavailability.(*Ha).Start\n\t/src/lorry/highavailability/ha.go:250\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1650"}
github.com/apecloud/kubeblocks/lorry/component/mongodb.(*Manager).CreateRoot
	/src/lorry/component/mongodb/manager.go:245
github.com/apecloud/kubeblocks/lorry/highavailability.(*Ha).Start
	/src/lorry/highavailability/ha.go:250
2023-11-03T02:54:14Z	ERROR	HA	Cluster initialize failed	{"error": "failed to create user: (NotWritablePrimary) not primary", "errorVerbose": "(NotWritablePrimary) not primary\nfailed to create user\ngithub.com/apecloud/kubeblocks/lorry/component/mongodb.CreateUser\n\t/src/lorry/component/mongodb/users.go:39\ngithub.com/apecloud/kubeblocks/lorry/component/mongodb.(*Manager).CreateRoot\n\t/src/lorry/component/mongodb/manager.go:243\ngithub.com/apecloud/kubeblocks/lorry/highavailability.(*Ha).Start\n\t/src/lorry/highavailability/ha.go:250\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1650"}
github.com/apecloud/kubeblocks/lorry/highavailability.(*Ha).Start
	/src/lorry/highavailability/ha.go:252
2023-11-03T02:54:15Z	INFO	grpc	role probe request	{"type": "mongodb"}
2023-11-03T02:54:15Z	ERROR	Mongo	DB is not ready	{"error": "connection() error occurred during connection handshake: auth error: sasl conversation error: unable to authenticate using mechanism \"SCRAM-SHA-1\": (AuthenticationFailed) Authentication failed."}
github.com/apecloud/kubeblocks/lorry/component/mongodb.(*Manager).IsDBStartupReady
	/src/lorry/component/mongodb/manager.go:273
github.com/apecloud/kubeblocks/lorry/binding/mongodb.(*MongoDBOperations).Init.(*BaseOperations).RegisterOperationOnDBReady.StartupCheckWraper.func2
	/src/lorry/binding/utils.go:333
github.com/apecloud/kubeblocks/lorry/binding.(*BaseOperations).Invoke
	/src/lorry/binding/base.go:194
github.com/apecloud/kubeblocks/lorry/middleware/probe.route
	/src/lorry/middleware/probe/router.go:170
github.com/apecloud/kubeblocks/lorry/middleware/grpc.NewGRPCServer.GetGrpcRouter.func1
	/src/lorry/middleware/probe/router.go:176
github.com/apecloud/kubeblocks/lorry/middleware/grpc.(*GRPCServer).Check
	/src/lorry/middleware/grpc/gprc_server.go:47
google.golang.org/grpc/health/grpc_health_v1._Health_Check_Handler
	/go/pkg/mod/google.golang.org/[email protected]/health/grpc_health_v1/health_grpc.pb.go:170
google.golang.org/grpc.(*Server).processUnaryRPC
	/go/pkg/mod/google.golang.org/[email protected]/server.go:1335
google.golang.org/grpc.(*Server).handleStream
	/go/pkg/mod/google.golang.org/[email protected]/server.go:1712
google.golang.org/grpc.(*Server).serveStreams.func1.1
	/go/pkg/mod/google.golang.org/[email protected]/server.go:947
2023-11-03T02:54:15Z	INFO	Mongo	operation called	{"operation": "checkRole", "result": {"event":"Failed","message":"db not ready"}}
2023-11-03T02:54:15Z	INFO	grpc	No event detected	{"meta": "{\"duration\":\"4.900275ms\",\"end-time\":\"2023-11-03T02:54:15.142554987Z\",\"operation\":\"checkRole\",\"start-time\":\"2023-11-03T02:54:15.137654711Z\"}", "data": "{\"event\":\"Failed\",\"message\":\"db not ready\"}"}
2023-11-03T02:54:17Z	INFO	grpc	role probe request	{"type": "mongodb"}
2023-11-03T02:54:17Z	ERROR	Mongo	DB is not ready	{"error": "connection() error occurred during connection handshake: auth error: sasl conversation error: unable to authenticate using mechanism \"SCRAM-SHA-1\": (AuthenticationFailed) Authentication failed."}
github.com/apecloud/kubeblocks/lorry/component/mongodb.(*Manager).IsDBStartupReady
	/src/lorry/component/mongodb/manager.go:273
github.com/apecloud/kubeblocks/lorry/binding/mongodb.(*MongoDBOperations).Init.(*BaseOperations).RegisterOperationOnDBReady.StartupCheckWraper.func2
	/src/lorry/binding/utils.go:333
github.com/apecloud/kubeblocks/lorry/binding.(*BaseOperations).Invoke
	/src/lorry/binding/base.go:194
github.com/apecloud/kubeblocks/lorry/middleware/probe.route
	/src/lorry/middleware/probe/router.go:170
github.com/apecloud/kubeblocks/lorry/middleware/grpc.NewGRPCServer.GetGrpcRouter.func1
	/src/lorry/middleware/probe/router.go:176
github.com/apecloud/kubeblocks/lorry/middleware/grpc.(*GRPCServer).Check
	/src/lorry/middleware/grpc/gprc_server.go:47
google.golang.org/grpc/health/grpc_health_v1._Health_Check_Handler
	/go/pkg/mod/google.golang.org/[email protected]/health/grpc_health_v1/health_grpc.pb.go:170
google.golang.org/grpc.(*Server).processUnaryRPC
	/go/pkg/mod/google.golang.org/[email protected]/server.go:1335
google.golang.org/grpc.(*Server).handleStream
	/go/pkg/mod/google.golang.org/[email protected]/server.go:1712
google.golang.org/grpc.(*Server).serveStreams.func1.1
	/go/pkg/mod/google.golang.org/[email protected]/server.go:947
2023-11-03T02:54:17Z	INFO	Mongo	operation called	{"operation": "checkRole", "result": {"event":"Failed","message":"db not ready"}}
2023-11-03T02:54:17Z	INFO	grpc	No event detected	{"meta": "{\"duration\":\"5.617041ms\",\"end-time\":\"2023-11-03T02:54:17.160095688Z\",\"operation\":\"checkRole\",\"start-time\":\"2023-11-03T02:54:17.154478644Z\"}", "data": "{\"event\":\"Failed\",\"message\":\"db not ready\"}"}

full checkrole log: ➜ ~ k logs mongo-jrsita-back Uploading kbcheckrole.txt… up-mongodb-0 kb-checkrole >kbcheckrole.txt

ahjing99 avatar Nov 03 '23 02:11 ahjing99