create cluster failed!can you help me?

[linqingping]

[linqingping]

E0322 03:11:58.035480 1 cluster_manager.go:126] Failed to get the cluster status: signal: killed getting cluster status from group seeds github.com/oracle/mysql-operator/pkg/controllers/cluster/manager.(*ClusterManager).getClusterStatus /go/src/github.com/oracle/mysql-operator/pkg/controllers/cluster/manager/cluster_manager.go:106 github.com/oracle/mysql-operator/pkg/controllers/cluster/manager.(*ClusterManager).Sync /go/src/github.com/oracle/mysql-operator/pkg/controllers/cluster/manager/cluster_manager.go:122 github.com/oracle/mysql-operator/cmd/mysql-agent/app.Run /go/src/github.com/oracle/mysql-operator/cmd/mysql-agent/app/mysql_agent.go:108 main.main /go/src/github.com/oracle/mysql-operator/cmd/mysql-agent/main.go:51 runtime.main /usr/local/go/src/runtime/proc.go:195 runtime.goexit /usr/local/go/src/runtime/asm_amd64.s:2337

[linqingping]

ServiceAccount and RoleBinding mysql_cluster.yaml helm list kubectl get po kubectl describe po

[prydie]

@linqingping The minimum number of members you should have in a cluster is three for MySQL InnoDB clustering.

[linqingping]

@prydie Thank you for you help！ members=3 How can we solve the problem?

[prydie]

@linqingping Can you grab the logs from the mysql-agent sidecar in the mysql-0 Pod please?

Other thoughts: do you have a NetworkPolicy defined that's not allowing the MySQL server instances to communicate?

[linqingping]

[root@localhost mysql_test]# kubectl logs mysql-0 -n mysql-operator -c mysql-agent Starting mysql-agent version 0.2.0 I0322 08:08:53.229613 1 main.go:48] FLAG: --address="0.0.0.0" I0322 08:08:53.229666 1 main.go:48] FLAG: --alsologtostderr="false" I0322 08:08:53.229673 1 main.go:48] FLAG: --cluster-name="mysql" I0322 08:08:53.229678 1 main.go:48] FLAG: --healthcheck-port="10512" I0322 08:08:53.229684 1 main.go:48] FLAG: --hostname="mysql-0" I0322 08:08:53.229688 1 main.go:48] FLAG: --log-backtrace-at=":0" I0322 08:08:53.229695 1 main.go:48] FLAG: --log-dir="" I0322 08:08:53.229700 1 main.go:48] FLAG: --log-flush-frequency="5s" I0322 08:08:53.229706 1 main.go:48] FLAG: --logtostderr="true" I0322 08:08:53.229710 1 main.go:48] FLAG: --min-resync-period="12h0m0s" I0322 08:08:53.229715 1 main.go:48] FLAG: --namespace="mysql-operator" I0322 08:08:53.229719 1 main.go:48] FLAG: --stderrthreshold="2" I0322 08:08:53.229723 1 main.go:48] FLAG: --v="4" I0322 08:08:53.229726 1 main.go:48] FLAG: --vmodule="" I0322 08:09:06.270386 1 cluster_manager.go:155] MySQL instance is online I0322 08:09:06.723925 1 cluster_labeler.go:241] Starting the ClusterLabelerController I0322 08:09:06.723947 1 cluster_labeler.go:244] Waiting for ClusterLabelerController informer caches to sync I0322 08:09:06.723961 1 cache.go:30] Waiting for caches to sync for innodb-cluster-labeler controller I0322 08:09:06.724240 1 reflector.go:202] Starting reflector *v1.Pod (19h15m21.324440719s) from github.com/oracle/mysql-operator/vendor/k8s.io/client-go/informers/factory.go:87 I0322 08:09:06.735192 1 reflector.go:240] Listing and watching *v1.Pod from github.com/oracle/mysql-operator/vendor/k8s.io/client-go/informers/factory.go:87 I0322 08:09:06.761284 1 agent_controller.go:103] Creating event broadcaster I0322 08:09:06.761771 1 agent_controller.go:111] Creating event broadcaster I0322 08:09:06.796578 1 agent_controller.go:171] Starting AgentController I0322 08:09:06.796606 1 agent_controller.go:174] Waiting for caches to sync I0322 08:09:06.796614 1 cache.go:30] Waiting for caches to sync for operator-backup-controller controller I0322 08:09:06.796662 1 agent_controller.go:180] Starting AgentController I0322 08:09:06.796668 1 agent_controller.go:183] Waiting for caches to sync I0322 08:09:06.796673 1 cache.go:30] Waiting for caches to sync for operator-restore-controller controller I0322 08:09:06.797109 1 reflector.go:202] Starting reflector *v1alpha1.Restore (0s) from github.com/oracle/mysql-operator/pkg/generated/informers/externalversions/factory.go:70 I0322 08:09:06.797126 1 reflector.go:240] Listing and watching *v1alpha1.Restore from github.com/oracle/mysql-operator/pkg/generated/informers/externalversions/factory.go:70 I0322 08:09:06.801028 1 reflector.go:202] Starting reflector *v1alpha1.Cluster (0s) from github.com/oracle/mysql-operator/pkg/generated/informers/externalversions/factory.go:70 I0322 08:09:06.801049 1 reflector.go:240] Listing and watching *v1alpha1.Cluster from github.com/oracle/mysql-operator/pkg/generated/informers/externalversions/factory.go:70 I0322 08:09:06.801504 1 reflector.go:202] Starting reflector *v1alpha1.Backup (0s) from github.com/oracle/mysql-operator/pkg/generated/informers/externalversions/factory.go:70 I0322 08:09:06.801518 1 reflector.go:240] Listing and watching *v1alpha1.Backup from github.com/oracle/mysql-operator/pkg/generated/informers/externalversions/factory.go:70 I0322 08:09:11.891462 1 cluster_manager.go:155] MySQL instance is online I0322 08:09:12.534131 1 shared_informer.go:123] caches populated I0322 08:09:12.534171 1 cache.go:37] Caches are synced for innodb-cluster-labeler controller I0322 08:09:12.534184 1 cluster_labeler.go:249] Starting ClusterLabelerController controller worker I0322 08:09:12.534196 1 cluster_labeler.go:252] Started ClusterLabelerController controller worker I0322 08:09:12.597269 1 shared_informer.go:123] caches populated I0322 08:09:12.597298 1 cache.go:37] Caches are synced for operator-backup-controller controller I0322 08:09:12.597307 1 agent_controller.go:181] Caches are synced I0322 08:09:12.611208 1 shared_informer.go:123] caches populated I0322 08:09:12.611238 1 cache.go:37] Caches are synced for operator-restore-controller controller I0322 08:09:12.611249 1 agent_controller.go:191] Caches are synced E0322 08:10:30.044536 1 cluster_manager.go:126] Failed to get the cluster status: signal: killed getting cluster status from group seeds github.com/oracle/mysql-operator/pkg/controllers/cluster/manager.(*ClusterManager).getClusterStatus /go/src/github.com/oracle/mysql-operator/pkg/controllers/cluster/manager/cluster_manager.go:106 github.com/oracle/mysql-operator/pkg/controllers/cluster/manager.(*ClusterManager).Sync /go/src/github.com/oracle/mysql-operator/pkg/controllers/cluster/manager/cluster_manager.go:122 github.com/oracle/mysql-operator/pkg/controllers/cluster/manager.(*ClusterManager).Run.func1 /go/src/github.com/oracle/mysql-operator/pkg/controllers/cluster/manager/cluster_manager.go:350 github.com/oracle/mysql-operator/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1 /go/src/github.com/oracle/mysql-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133 github.com/oracle/mysql-operator/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil /go/src/github.com/oracle/mysql-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:134 github.com/oracle/mysql-operator/vendor/k8s.io/apimachinery/pkg/util/wait.Until /go/src/github.com/oracle/mysql-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88 github.com/oracle/mysql-operator/pkg/controllers/cluster/manager.(*ClusterManager).Run /go/src/github.com/oracle/mysql-operator/pkg/controllers/cluster/manager/cluster_manager.go:350 github.com/oracle/mysql-operator/cmd/mysql-agent/app.Run.func2 /go/src/github.com/oracle/mysql-operator/cmd/mysql-agent/app/mysql_agent.go:114 runtime.goexit /usr/local/go/src/runtime/asm_amd64.s:2337 I0322 08:10:55.724118 1 cluster_manager.go:116] Database not yet running. Waiting... I0322 08:11:21.863000 1 cluster_manager.go:116] Database not yet running. Waiting... E0322 08:13:05.762813 1 cluster_manager.go:126] Failed to get the cluster status: signal: killed getting cluster status from group seeds github.com/oracle/mysql-operator/pkg/controllers/cluster/manager.(*ClusterManager).getClusterStatus /go/src/github.com/oracle/mysql-operator/pkg/controllers/cluster/manager/cluster_manager.go:106 github.com/oracle/mysql-operator/pkg/controllers/cluster/manager.(*ClusterManager).Sync /go/src/github.com/oracle/mysql-operator/pkg/controllers/cluster/manager/cluster_manager.go:122 github.com/oracle/mysql-operator/pkg/controllers/cluster/manager.(*ClusterManager).Run.func1 /go/src/github.com/oracle/mysql-operator/pkg/controllers/cluster/manager/cluster_manager.go:350 github.com/oracle/mysql-operator/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1 /go/src/github.com/oracle/mysql-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133 github.com/oracle/mysql-operator/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil /go/src/github.com/oracle/mysql-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:134 github.com/oracle/mysql-operator/vendor/k8s.io/apimachinery/pkg/util/wait.Until /go/src/github.com/oracle/mysql-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88 github.com/oracle/mysql-operator/pkg/controllers/cluster/manager.(*ClusterManager).Run /go/src/github.com/oracle/mysql-operator/pkg/controllers/cluster/manager/cluster_manager.go:350 github.com/oracle/mysql-operator/cmd/mysql-agent/app.Run.func2 /go/src/github.com/oracle/mysql-operator/cmd/mysql-agent/app/mysql_agent.go:114 runtime.goexit /usr/local/go/src/runtime/asm_amd64.s:2337 I0322 08:13:31.439974 1 cluster_manager.go:116] Database not yet running. Waiting... I0322 08:13:38.399737 1 agent_controller.go:203] Shutting down AgentController I0322 08:13:38.399767 1 agent_controller.go:166] Waiting for workers to finish their work I0322 08:13:38.399790 1 agent_controller.go:193] Shutting down AgentController I0322 08:13:38.399796 1 agent_controller.go:157] Waiting for workers to finish their work I0322 08:13:38.400267 1 mysql_agent.go:152] Waiting for all controllers to shut down gracefully I0322 08:13:38.400332 1 cluster_labeler.go:255] Shutting down ClusterLabelerController controller worker I0322 08:13:38.400362 1 agent_controller.go:176] All workers have finished I0322 08:13:38.400380 1 agent_controller.go:167] All workers have finished

[linqingping]

I don‘t have NetworkPolicy @prydie

[prydie]

@linqingping Ah, you're running 0.2.0. Can you try the latest release (0.3.0)?

[linqingping]

[root@localhost mysql_test]# kubectl logs mysql-0 -n mysql-operator -c mysql-agent Starting mysql-agent version 0.3.0 I0322 08:36:21.807203 1 main.go:48] FLAG: --address="0.0.0.0" I0322 08:36:21.807248 1 main.go:48] FLAG: --alsologtostderr="false" I0322 08:36:21.807254 1 main.go:48] FLAG: --cluster-name="mysql" I0322 08:36:21.807259 1 main.go:48] FLAG: --healthcheck-port="10512" I0322 08:36:21.807266 1 main.go:48] FLAG: --hostname="mysql-0" I0322 08:36:21.807270 1 main.go:48] FLAG: --log-backtrace-at=":0" I0322 08:36:21.807277 1 main.go:48] FLAG: --log-dir="" I0322 08:36:21.807282 1 main.go:48] FLAG: --log-flush-frequency="5s" I0322 08:36:21.807289 1 main.go:48] FLAG: --logtostderr="true" I0322 08:36:21.807293 1 main.go:48] FLAG: --min-resync-period="12h0m0s" I0322 08:36:21.807297 1 main.go:48] FLAG: --namespace="mysql-operator" I0322 08:36:21.807302 1 main.go:48] FLAG: --stderrthreshold="2" I0322 08:36:21.807306 1 main.go:48] FLAG: --v="4" I0322 08:36:21.807309 1 main.go:48] FLAG: --vmodule="" I0322 08:36:25.569124 1 cluster_manager.go:160] MySQL instance is online I0322 08:36:25.586764 1 agent_controller.go:103] Creating event broadcaster I0322 08:36:25.586941 1 agent_controller.go:111] Creating event broadcaster I0322 08:36:25.621140 1 cluster_labeler.go:241] Starting the ClusterLabelerController I0322 08:36:25.621168 1 cluster_labeler.go:244] Waiting for ClusterLabelerController informer caches to sync I0322 08:36:25.621179 1 cache.go:30] Waiting for caches to sync for innodb-cluster-labeler controller I0322 08:36:25.638683 1 agent_controller.go:183] Starting AgentController I0322 08:36:25.638721 1 agent_controller.go:186] Waiting for caches to sync I0322 08:36:25.638728 1 cache.go:30] Waiting for caches to sync for operator-backup-controller controller I0322 08:36:25.638781 1 agent_controller.go:193] Starting AgentController I0322 08:36:25.638786 1 agent_controller.go:196] Waiting for caches to sync I0322 08:36:25.638790 1 cache.go:30] Waiting for caches to sync for operator-restore-controller controller I0322 08:36:25.639006 1 reflector.go:202] Starting reflector *v1.Pod (19h15m21.324440719s) from github.com/oracle/mysql-operator/vendor/k8s.io/client-go/informers/factory.go:87 I0322 08:36:25.639033 1 reflector.go:240] Listing and watching *v1.Pod from github.com/oracle/mysql-operator/vendor/k8s.io/client-go/informers/factory.go:87 I0322 08:36:25.639788 1 reflector.go:202] Starting reflector *v1alpha1.Restore (0s) from github.com/oracle/mysql-operator/pkg/generated/informers/externalversions/factory.go:70 I0322 08:36:25.639801 1 reflector.go:240] Listing and watching *v1alpha1.Restore from github.com/oracle/mysql-operator/pkg/generated/informers/externalversions/factory.go:70 I0322 08:36:25.640140 1 reflector.go:202] Starting reflector *v1alpha1.Cluster (0s) from github.com/oracle/mysql-operator/pkg/generated/informers/externalversions/factory.go:70 I0322 08:36:25.640152 1 reflector.go:240] Listing and watching *v1alpha1.Cluster from github.com/oracle/mysql-operator/pkg/generated/informers/externalversions/factory.go:70 I0322 08:36:25.648714 1 reflector.go:202] Starting reflector *v1alpha1.Backup (0s) from github.com/oracle/mysql-operator/pkg/generated/informers/externalversions/factory.go:70 I0322 08:36:25.648736 1 reflector.go:240] Listing and watching *v1alpha1.Backup from github.com/oracle/mysql-operator/pkg/generated/informers/externalversions/factory.go:70 I0322 08:36:27.178114 1 shared_informer.go:123] caches populated I0322 08:36:27.182283 1 cache.go:37] Caches are synced for operator-restore-controller controller I0322 08:36:27.182301 1 agent_controller.go:204] Caches are synced I0322 08:36:27.182353 1 shared_informer.go:123] caches populated I0322 08:36:27.182359 1 cache.go:37] Caches are synced for innodb-cluster-labeler controller I0322 08:36:27.329872 1 cluster_labeler.go:249] Starting ClusterLabelerController controller worker I0322 08:36:27.329903 1 cluster_labeler.go:252] Started ClusterLabelerController controller worker I0322 08:36:27.330149 1 shared_informer.go:123] caches populated I0322 08:36:27.330159 1 cache.go:37] Caches are synced for operator-backup-controller controller I0322 08:36:27.330165 1 agent_controller.go:193] Caches are synced I0322 08:36:32.835150 1 cluster_manager.go:160] MySQL instance is online

[prydie]

That looks more promising 😄

[linqingping]

[linqingping]

@prydie Still wrong

[prydie]

Can you grab the mysql-agent logs from mysql-1?

[linqingping]

[root@localhost mysql_test]# kubectl logs mysql-1 -n mysql-operator -c mysql-agent Starting mysql-agent version 0.3.0 I0322 08:39:00.530681 1 main.go:48] FLAG: --address="0.0.0.0" I0322 08:39:00.530730 1 main.go:48] FLAG: --alsologtostderr="false" I0322 08:39:00.530737 1 main.go:48] FLAG: --cluster-name="mysql" I0322 08:39:00.530742 1 main.go:48] FLAG: --healthcheck-port="10512" I0322 08:39:00.530750 1 main.go:48] FLAG: --hostname="mysql-1" I0322 08:39:00.530754 1 main.go:48] FLAG: --log-backtrace-at=":0" I0322 08:39:00.530762 1 main.go:48] FLAG: --log-dir="" I0322 08:39:00.530768 1 main.go:48] FLAG: --log-flush-frequency="5s" I0322 08:39:00.530775 1 main.go:48] FLAG: --logtostderr="true" I0322 08:39:00.530779 1 main.go:48] FLAG: --min-resync-period="12h0m0s" I0322 08:39:00.530784 1 main.go:48] FLAG: --namespace="mysql-operator" I0322 08:39:00.530789 1 main.go:48] FLAG: --stderrthreshold="2" I0322 08:39:00.530793 1 main.go:48] FLAG: --v="4" I0322 08:39:00.530797 1 main.go:48] FLAG: --vmodule="" I0322 08:39:01.563462 1 cluster_manager.go:160] MySQL instance is online I0322 08:39:01.563508 1 agent_controller.go:103] Creating event broadcaster I0322 08:39:01.563708 1 agent_controller.go:111] Creating event broadcaster I0322 08:39:01.564304 1 reflector.go:202] Starting reflector *v1.Pod (19h15m21.324440719s) from github.com/oracle/mysql-operator/vendor/k8s.io/client-go/informers/factory.go:87 I0322 08:39:01.564344 1 reflector.go:240] Listing and watching *v1.Pod from github.com/oracle/mysql-operator/vendor/k8s.io/client-go/informers/factory.go:87 I0322 08:39:01.587047 1 agent_controller.go:183] Starting AgentController I0322 08:39:01.587073 1 agent_controller.go:186] Waiting for caches to sync I0322 08:39:01.587080 1 cache.go:30] Waiting for caches to sync for operator-backup-controller controller I0322 08:39:01.587129 1 agent_controller.go:193] Starting AgentController I0322 08:39:01.587135 1 agent_controller.go:196] Waiting for caches to sync I0322 08:39:01.587140 1 cache.go:30] Waiting for caches to sync for operator-restore-controller controller I0322 08:39:01.587530 1 reflector.go:202] Starting reflector *v1alpha1.Restore (0s) from github.com/oracle/mysql-operator/pkg/generated/informers/externalversions/factory.go:70 I0322 08:39:01.587548 1 reflector.go:240] Listing and watching *v1alpha1.Restore from github.com/oracle/mysql-operator/pkg/generated/informers/externalversions/factory.go:70 I0322 08:39:01.588083 1 reflector.go:202] Starting reflector *v1alpha1.Cluster (0s) from github.com/oracle/mysql-operator/pkg/generated/informers/externalversions/factory.go:70 I0322 08:39:01.588098 1 reflector.go:240] Listing and watching *v1alpha1.Cluster from github.com/oracle/mysql-operator/pkg/generated/informers/externalversions/factory.go:70 I0322 08:39:01.589051 1 reflector.go:202] Starting reflector *v1alpha1.Backup (0s) from github.com/oracle/mysql-operator/pkg/generated/informers/externalversions/factory.go:70 I0322 08:39:01.589069 1 reflector.go:240] Listing and watching *v1alpha1.Backup from github.com/oracle/mysql-operator/pkg/generated/informers/externalversions/factory.go:70 I0322 08:39:02.796896 1 shared_informer.go:123] caches populated I0322 08:39:02.796936 1 cache.go:37] Caches are synced for operator-restore-controller controller I0322 08:39:02.796948 1 agent_controller.go:204] Caches are synced I0322 08:39:02.797013 1 shared_informer.go:123] caches populated I0322 08:39:02.797020 1 cache.go:37] Caches are synced for operator-backup-controller controller I0322 08:39:02.797027 1 agent_controller.go:193] Caches are synced I0322 08:39:02.892626 1 cluster_manager.go:160] MySQL instance is online I0322 08:39:19.404509 1 cluster_manager.go:160] MySQL instance is online I0322 08:39:35.570575 1 cluster_manager.go:160] MySQL instance is online I0322 08:39:50.921820 1 cluster_manager.go:160] MySQL instance is online I0322 08:40:06.872118 1 cluster_manager.go:160] MySQL instance is online I0322 08:40:22.754593 1 cluster_manager.go:160] MySQL instance is online I0322 08:40:38.154708 1 cluster_manager.go:160] MySQL instance is online I0322 08:40:53.427846 1 cluster_manager.go:160] MySQL instance is online I0322 08:41:08.848268 1 cluster_manager.go:160] MySQL instance is online I0322 08:41:24.270733 1 cluster_manager.go:160] MySQL instance is online I0322 08:41:39.510834 1 cluster_manager.go:160] MySQL instance is online I0322 08:41:54.766589 1 cluster_manager.go:160] MySQL instance is online I0322 08:42:10.028766 1 cluster_manager.go:160] MySQL instance is online I0322 08:42:25.311532 1 cluster_manager.go:160] MySQL instance is online I0322 08:42:40.865616 1 cluster_manager.go:160] MySQL instance is online I0322 08:42:56.141458 1 cluster_manager.go:160] MySQL instance is online

[prydie]

That looks good too. The Pods are still not showing as Ready then?

[linqingping]

Has been constantly restart @prydie