charts
charts copied to clipboard
Published
20 hours ago •
bitnami
bitnami
[bitnami/mysql] replication error nmysqladmin: connect to server at 'localhost' failed, status 0/1 runing
Name and Version
bitnami/mysql:9.3.0
What steps will reproduce the bug?
helm install bucai-mysql bitnami/mysql --set global.storageClass=nfs-sc-default --set architecture=replication --set secondary.replicaCount=1 -n xxx
Are you using any custom parameters or values?
--set global.storageClass=nfs-sc-default --set architecture=replication --set secondary.replicaCount=1
What is the expected behavior?
Expect normal operation
What do you see instead?
[root@master ~]# kubectl get po -n bucai
NAME READY STATUS RESTARTS AGE
bucai-mysql-primary-0 0/1 Running 4 (113s ago) 8m15s
bucai-mysql-secondary-0 0/1 Running 2 (110s ago) 8m15s
[root@master ~]# kubectl describe pod bucai-mysql-primary-0 -n bucai
Name: bucai-mysql-primary-0
Namespace: bucai
Priority: 0
Node: node1/10.65.42.70
Start Time: Wed, 24 Aug 2022 14:46:03 +0800
Labels: app.kubernetes.io/component=primary
app.kubernetes.io/instance=bucai-mysql
app.kubernetes.io/managed-by=Helm
app.kubernetes.io/name=mysql
controller-revision-hash=bucai-mysql-primary-756b4745cb
helm.sh/chart=mysql-9.3.0
statefulset.kubernetes.io/pod-name=bucai-mysql-primary-0
Annotations: checksum/configuration: 3f419de93f1797216769b4b8b2323a3b8bde9ea309ec7263da90dfc015f8b10f
Status: Running
IP: 10.244.1.56
IPs:
IP: 10.244.1.56
Controlled By: StatefulSet/bucai-mysql-primary
Containers:
mysql:
Container ID: docker://93540a984a1bc113d88cb053426a458f6219ca22fc3f98f142999c5197d1ec04
Image: docker.io/bitnami/mysql:8.0.23-debian-10-r0
Image ID: docker-pullable://bitnami/mysql@sha256:726854187b8f5d232ebd400e6fe8a259afa42b81793efc7aadd6843b2a853897
Port: 3306/TCP
Host Port: 0/TCP
State: Running
Started: Wed, 24 Aug 2022 14:48:44 +0800
Last State: Terminated
Reason: Error
Exit Code: 137
Started: Wed, 24 Aug 2022 14:47:20 +0800
Finished: Wed, 24 Aug 2022 14:48:44 +0800
Ready: False
Restart Count: 2
Liveness: exec [/bin/bash -ec password_aux="${MYSQL_ROOT_PASSWORD:-}"
if [[ -f "${MYSQL_ROOT_PASSWORD_FILE:-}" ]]; then
password_aux=$(cat "$MYSQL_ROOT_PASSWORD_FILE")
fi
mysqladmin status -uroot -p"${password_aux}"
] delay=5s timeout=1s period=10s #success=1 #failure=3
Readiness: exec [/bin/bash -ec password_aux="${MYSQL_ROOT_PASSWORD:-}"
if [[ -f "${MYSQL_ROOT_PASSWORD_FILE:-}" ]]; then
password_aux=$(cat "$MYSQL_ROOT_PASSWORD_FILE")
fi
mysqladmin status -uroot -p"${password_aux}"
] delay=5s timeout=1s period=10s #success=1 #failure=3
Startup: exec [/bin/bash -ec password_aux="${MYSQL_ROOT_PASSWORD:-}"
if [[ -f "${MYSQL_ROOT_PASSWORD_FILE:-}" ]]; then
password_aux=$(cat "$MYSQL_ROOT_PASSWORD_FILE")
fi
mysqladmin status -uroot -p"${password_aux}"
] delay=15s timeout=1s period=10s #success=1 #failure=10
Environment:
BITNAMI_DEBUG: false
MYSQL_ROOT_PASSWORD: <set to the key 'mysql-root-password' in secret 'bucai-mysql'> Optional: false
MYSQL_DATABASE: my_database
MYSQL_REPLICATION_MODE: master
MYSQL_REPLICATION_USER: replicator
MYSQL_REPLICATION_PASSWORD: <set to the key 'mysql-replication-password' in secret 'bucai-mysql'> Optional: false
Mounts:
/bitnami/mysql from data (rw)
/opt/bitnami/mysql/conf/my.cnf from config (rw,path="my.cnf")
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-8l4jc (ro)
Conditions:
Type Status
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
data:
Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
ClaimName: data-bucai-mysql-primary-0
ReadOnly: false
config:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: bucai-mysql-primary
Optional: false
kube-api-access-8l4jc:
Type: Projected (a volume that contains injected data from multiple sources)
TokenExpirationSeconds: 3607
ConfigMapName: kube-root-ca.crt
ConfigMapOptional: <nil>
DownwardAPI: true
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 4m default-scheduler Successfully assigned bucai/bucai-mysql-primary-0 to node1
Normal Killing 110s kubelet Container mysql failed startup probe, will be restarted
Normal Pulled 79s (x3 over 3m59s) kubelet Container image "docker.io/bitnami/mysql:8.0.23-debian-10-r0" already present on machine
Normal Created 79s (x3 over 3m58s) kubelet Created container mysql
Normal Started 79s (x3 over 3m58s) kubelet Started container mysql
Warning Unhealthy 50s (x11 over 3m40s) kubelet Startup probe failed: mysqladmin: [Warning] Using a password on the command line interface can be insecure.
mysqladmin: connect to server at 'localhost' failed
error: 'Can't connect to local MySQL server through socket '/opt/bitnami/mysql/tmp/mysql.sock' (2)'
Check that mysqld is running and that the socket: '/opt/bitnami/mysql/tmp/mysql.sock' exists!
Warning Unhealthy 20s (x4 over 2m) kubelet Startup probe failed: mysqladmin: [Warning] Using a password on the command line interface can be insecure.
mysqladmin: connect to server at 'localhost' failed
error: 'Access denied for user 'root'@'localhost' (using password: YES)'
[root@master ~]# kubectl describe pod bucai-mysql-secondary-0 -n bucai
Name: bucai-mysql-secondary-0
Namespace: bucai
Priority: 0
Node: node2/10.65.42.94
Start Time: Wed, 24 Aug 2022 14:46:03 +0800
Labels: app.kubernetes.io/component=secondary
app.kubernetes.io/instance=bucai-mysql
app.kubernetes.io/managed-by=Helm
app.kubernetes.io/name=mysql
controller-revision-hash=bucai-mysql-secondary-69db84f5f4
helm.sh/chart=mysql-9.3.0
statefulset.kubernetes.io/pod-name=bucai-mysql-secondary-0
Annotations: checksum/configuration: e3c30ab630a4b3cb4da672dd34ada6c779fe9b75043faa3751f8abde032000ec
Status: Running
IP: 10.244.2.65
IPs:
IP: 10.244.2.65
Controlled By: StatefulSet/bucai-mysql-secondary
Containers:
mysql:
Container ID: docker://be77e443910507995c7f72b390644d8243f0e1644a27e656bddb7c42828505ec
Image: docker.io/bitnami/mysql:8.0.23-debian-10-r0
Image ID: docker-pullable://bitnami/mysql@sha256:726854187b8f5d232ebd400e6fe8a259afa42b81793efc7aadd6843b2a853897
Port: 3306/TCP
Host Port: 0/TCP
State: Running
Started: Wed, 24 Aug 2022 14:52:28 +0800
Last State: Terminated
Reason: Error
Exit Code: 137
Started: Wed, 24 Aug 2022 14:49:15 +0800
Finished: Wed, 24 Aug 2022 14:52:28 +0800
Ready: False
Restart Count: 2
Liveness: exec [/bin/bash -ec password_aux="${MYSQL_MASTER_ROOT_PASSWORD:-}"
if [[ -f "${MYSQL_MASTER_ROOT_PASSWORD_FILE:-}" ]]; then
password_aux=$(cat "$MYSQL_MASTER_ROOT_PASSWORD_FILE")
fi
mysqladmin status -uroot -p"${password_aux}"
] delay=5s timeout=1s period=10s #success=1 #failure=3
Readiness: exec [/bin/bash -ec password_aux="${MYSQL_MASTER_ROOT_PASSWORD:-}"
if [[ -f "${MYSQL_MASTER_ROOT_PASSWORD_FILE:-}" ]]; then
password_aux=$(cat "$MYSQL_MASTER_ROOT_PASSWORD_FILE")
fi
mysqladmin status -uroot -p"${password_aux}"
] delay=5s timeout=1s period=10s #success=1 #failure=3
Startup: exec [/bin/bash -ec password_aux="${MYSQL_MASTER_ROOT_PASSWORD:-}"
if [[ -f "${MYSQL_MASTER_ROOT_PASSWORD_FILE:-}" ]]; then
password_aux=$(cat "$MYSQL_MASTER_ROOT_PASSWORD_FILE")
fi
mysqladmin status -uroot -p"${password_aux}"
] delay=15s timeout=1s period=10s #success=1 #failure=15
Environment:
BITNAMI_DEBUG: false
MYSQL_REPLICATION_MODE: slave
MYSQL_MASTER_HOST: bucai-mysql-primary
MYSQL_MASTER_PORT_NUMBER: 3306
MYSQL_MASTER_ROOT_USER: root
MYSQL_REPLICATION_USER: replicator
MYSQL_MASTER_ROOT_PASSWORD: <set to the key 'mysql-root-password' in secret 'bucai-mysql'> Optional: false
MYSQL_REPLICATION_PASSWORD: <set to the key 'mysql-replication-password' in secret 'bucai-mysql'> Optional: false
Mounts:
/bitnami/mysql from data (rw)
/opt/bitnami/mysql/conf/my.cnf from config (rw,path="my.cnf")
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-lncjq (ro)
Conditions:
Type Status
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
data:
Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
ClaimName: data-bucai-mysql-secondary-0
ReadOnly: false
config:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: bucai-mysql-secondary
Optional: false
kube-api-access-lncjq:
Type: Projected (a volume that contains injected data from multiple sources)
TokenExpirationSeconds: 3607
ConfigMapName: kube-root-ca.crt
ConfigMapOptional: <nil>
DownwardAPI: true
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 8m2s default-scheduler Successfully assigned bucai/bucai-mysql-secondary-0 to node2
Normal Killing 5m22s kubelet Container mysql failed startup probe, will be restarted
Normal Pulled 4m51s (x2 over 8m) kubelet Container image "docker.io/bitnami/mysql:8.0.23-debian-10-r0" already present on machine
Normal Created 4m51s (x2 over 8m) kubelet Created container mysql
Normal Started 4m50s (x2 over 8m) kubelet Started container mysql
Warning Unhealthy 2m52s (x26 over 7m42s) kubelet Startup probe failed: mysqladmin: [Warning] Using a password on the command line interface can be insecure.
mysqladmin: connect to server at 'localhost' failed
error: 'Can't connect to local MySQL server through socket '/opt/bitnami/mysql/tmp/mysql.sock' (2)'
Check that mysqld is running and that the socket: '/opt/bitnami/mysql/tmp/mysql.sock' exists!
Additional information
[root@master ~]# helm version
version.BuildInfo{Version:"v3.9.3", GitCommit:"414ff28d4029ae8c8b05d62aa06c7fe3dee2bc58", GitTreeState:"clean", GoVersion:"go1.17.13"}
[root@master ~]# kubelet --version
Kubernetes v1.22.4
Hi @notbucai
Could you share the following info?:
- Please describe the
PersistenVolumeClaims:
$ kubectl describe -n xxx pvc data-bucai-mysql-primary-0
$ kubectl describe -n xxx pvc data-bucai-mysql-secondary-0
- Get the logs from the pods
$ kubectl logs -n xxx bucai-mysql-primary-0
$ kubectl logs -n xxx bucai-mysql-secondary-0
@fmulero PersistenVolumeClaims:
[root@master ~]# kubectl describe -n xxx pvc data-bucai-mysql-primary-0
Name: data-bucai-mysql-primary-0
Namespace: bucai
StorageClass: nfs-sc-default
Status: Bound
Volume: pvc-2965ae52-5742-4519-88c7-0c769818e240
Labels: app.kubernetes.io/component=primary
app.kubernetes.io/instance=bucai-mysql
app.kubernetes.io/name=mysql
Annotations: pv.kubernetes.io/bind-completed: yes
pv.kubernetes.io/bound-by-controller: yes
volume.beta.kubernetes.io/storage-provisioner: cluster.local/nfs-subdir-external-provisioner
Finalizers: [kubernetes.io/pvc-protection]
Capacity: 8Gi
Access Modes: RWO
VolumeMode: Filesystem
Used By: bucai-mysql-primary-0
Events: <none>
[root@master ~]# kubectl describe -n xxx pvc data-bucai-mysql-secondary-0
Name: data-bucai-mysql-secondary-0
Namespace: bucai
StorageClass: nfs-sc-default
Status: Bound
Volume: pvc-ca946cb1-df76-4a2e-aa5d-b90cc749eeca
Labels: app.kubernetes.io/component=secondary
app.kubernetes.io/instance=bucai-mysql
app.kubernetes.io/name=mysql
Annotations: pv.kubernetes.io/bind-completed: yes
pv.kubernetes.io/bound-by-controller: yes
volume.beta.kubernetes.io/storage-provisioner: cluster.local/nfs-subdir-external-provisioner
Finalizers: [kubernetes.io/pvc-protection]
Capacity: 8Gi
Access Modes: RWO
VolumeMode: Filesystem
Used By: bucai-mysql-secondary-0
Events: <none>
logs:
[root@master ~]# kubectl logs -n xxx bucai-mysql-primary-0
mysql 08:42:59.83
mysql 08:42:59.83 Welcome to the Bitnami mysql container
mysql 08:42:59.85 Subscribe to project updates by watching https://github.com/bitnami/bitnami-docker-mysql
mysql 08:42:59.85 Submit issues and feature requests at https://github.com/bitnami/bitnami-docker-mysql/issues
mysql 08:42:59.86
mysql 08:42:59.88 INFO ==> ** Starting MySQL setup **
mysql 08:42:59.98 INFO ==> Validating settings in MYSQL_*/MARIADB_* env vars
mysql 08:43:00.01 INFO ==> Initializing mysql database
mysql 08:43:00.11 WARN ==> The mysql configuration file '/opt/bitnami/mysql/conf/my.cnf' is not writable. Configurations based on environment variables will not be applied for this file.
mysql 08:43:00.12 INFO ==> Using persisted data
mysql 08:43:00.29 INFO ==> Running mysql_upgrade
mysql 08:43:00.35 INFO ==> Starting mysql in background
mysql 08:44:02.67 INFO ==> Stopping mysql
mysql 08:44:10.73 INFO ==> ** MySQL setup finished! **
mysql 08:44:10.76 INFO ==> ** Starting MySQL **
2022-08-24T08:44:11.274293Z 0 [System] [MY-010116] [Server] /opt/bitnami/mysql/bin/mysqld (mysqld 8.0.23) starting as process 1
2022-08-24T08:44:11.283537Z 0 [Warning] [MY-013242] [Server] --character-set-server: 'utf8' is currently an alias for the character set UTF8MB3, but will be an alias for UTF8MB4 in a future release. Please consider using UTF8MB4 in order to be unambiguous.
2022-08-24T08:44:11.283578Z 0 [Warning] [MY-013244] [Server] --collation-server: 'utf8_general_ci' is a collation of the deprecated character set UTF8MB3. Please consider using UTF8MB4 with an appropriate collation instead.
2022-08-24T08:44:11.333257Z 1 [System] [MY-013576] [InnoDB] InnoDB initialization has started.
2022-08-24T08:44:12.736474Z 1 [System] [MY-013577] [InnoDB] InnoDB initialization has ended.
2022-08-24T08:44:13.126023Z 0 [System] [MY-011323] [Server] X Plugin ready for connections. Bind-address: '::' port: 33060, socket: /tmp/mysqlx.sock
2022-08-24T08:44:13.252196Z 0 [Warning] [MY-010068] [Server] CA certificate ca.pem is self signed.
2022-08-24T08:44:13.252427Z 0 [System] [MY-013602] [Server] Channel mysql_main configured to support TLS. Encrypted connections are now supported for this channel.
2022-08-24T08:44:13.289675Z 0 [System] [MY-010931] [Server] /opt/bitnami/mysql/bin/mysqld: ready for connections. Version: '8.0.23' socket: '/opt/bitnami/mysql/tmp/mysql.sock' port: 3306 Source distribution.
[root@master ~]# kubectl logs -n xxx bucai-mysql-secondary-0
mysql 08:42:29.70
mysql 08:42:29.70 Welcome to the Bitnami mysql container
mysql 08:42:29.71 Subscribe to project updates by watching https://github.com/bitnami/bitnami-docker-mysql
mysql 08:42:29.72 Submit issues and feature requests at https://github.com/bitnami/bitnami-docker-mysql/issues
mysql 08:42:29.73
mysql 08:42:29.74 INFO ==> ** Starting MySQL setup **
mysql 08:42:29.79 INFO ==> Validating settings in MYSQL_*/MARIADB_* env vars
mysql 08:42:29.81 INFO ==> Initializing mysql database
mysql 08:43:18.20 WARN ==> The mysql configuration file '/opt/bitnami/mysql/conf/my.cnf' is not writable. Configurations based on environment variables will not be applied for this file.
mysql 08:44:17.29 INFO ==> Using persisted data
mysql 08:44:17.35 INFO ==> Running mysql_upgrade
mysql 08:44:17.36 INFO ==> Starting mysql in background
Hi @notbucai, could you try and deploy the chart with image.debug=true? That might show us some more information to understand what is really happening.
hi @marcosbc , here is the information after adding the parameters
pod/bucai-mysql-primary-0 1/1 Running 1 (15m ago) 19m
pod/bucai-mysql-secondary-0 0/1 Running 6 (61s ago) 19m
[root@master ~]# kubectl logs pod/bucai-mysql-secondary-0 -n bucai
mysql 04:41:25.72
mysql 04:41:25.72 Welcome to the Bitnami mysql container
mysql 04:41:25.75 Subscribe to project updates by watching https://github.com/bitnami/containers
mysql 04:41:25.75 Submit issues and feature requests at https://github.com/bitnami/containers/issues
mysql 04:41:25.76
mysql 04:41:25.77 INFO ==> ** Starting MySQL setup **
mysql 04:41:25.84 INFO ==> Validating settings in MYSQL_*/MARIADB_* env vars
mysql 04:41:25.87 INFO ==> Initializing mysql database
mysql 04:41:25.88 DEBUG ==> Ensuring expected directories/files exist
mysql 04:41:25.95 WARN ==> The mysql configuration file '/opt/bitnami/mysql/conf/my.cnf' is not writable. Configurations based on environment variables will not be applied for this file.
mysql 04:41:25.96 INFO ==> Using persisted data
mysql 04:41:26.11 INFO ==> Running mysql_upgrade
mysql 04:41:26.18 INFO ==> Starting mysql in background
2022-08-28T04:41:26.749054Z 0 [Warning] [MY-011068] [Server] The syntax 'log_slave_updates' is deprecated and will be removed in a future release. Please use log_replica_updates instead.
2022-08-28T04:41:26.749099Z 0 [Warning] [MY-011069] [Server] The syntax '--master-info-repository' is deprecated and will be removed in a future release.
2022-08-28T04:41:26.749122Z 0 [Warning] [MY-011069] [Server] The syntax '--relay-log-info-repository' is deprecated and will be removed in a future release.
2022-08-28T04:41:26.749158Z 0 [Warning] [MY-011068] [Server] The syntax 'skip_slave_start' is deprecated and will be removed in a future release. Please use skip_replica_start instead.
2022-08-28T04:41:26.749483Z 0 [Warning] [MY-010918] [Server] 'default_authentication_plugin' is deprecated and will be removed in a future release. Please use authentication_policy instead.
2022-08-28T04:41:26.749577Z 0 [System] [MY-010116] [Server] /opt/bitnami/mysql/bin/mysqld (mysqld 8.0.30) starting as process 54
2022-08-28T04:41:26.756621Z 0 [Warning] [MY-013242] [Server] --character-set-server: 'utf8' is currently an alias for the character set UTF8MB3, but will be an alias for UTF8MB4 in a future release. Please consider using UTF8MB4 in order to be unambiguous.
2022-08-28T04:41:26.756649Z 0 [Warning] [MY-013244] [Server] --collation-server: 'utf8mb3_general_ci' is a collation of the deprecated character set UTF8MB3. Please consider using UTF8MB4 with an appropriate collation instead.
2022-08-28T04:41:26.858956Z 1 [System] [MY-013576] [InnoDB] InnoDB initialization has started.
2022-08-28T04:41:30.169831Z 1 [System] [MY-013577] [InnoDB] InnoDB initialization has ended.
2022-08-28T04:41:31.606234Z 4 [System] [MY-013381] [Server] Server upgrade from '80030' to '80030' started.
2022-08-28T04:42:11.950523Z 4 [System] [MY-013381] [Server] Server upgrade from '80030' to '80030' completed.
2022-08-28T04:42:12.710604Z 0 [Warning] [MY-010068] [Server] CA certificate ca.pem is self signed.
2022-08-28T04:42:12.710734Z 0 [System] [MY-013602] [Server] Channel mysql_main configured to support TLS. Encrypted connections are now supported for this channel.
2022-08-28T04:42:12.796490Z 0 [System] [MY-011323] [Server] X Plugin ready for connections. Bind-address: '::' port: 33060, socket: /tmp/mysqlx.sock
2022-08-28T04:42:12.801709Z 0 [System] [MY-010931] [Server] /opt/bitnami/mysql/bin/mysqld: ready for connections. Version: '8.0.30' socket: '/opt/bitnami/mysql/tmp/mysql.sock' port: 3306 Source distribution.
find: '/docker-entrypoint-startdb.d/': No such file or directory
mysql 04:42:14.57 INFO ==> Stopping mysql
2022-08-28T04:42:14.586655Z 0 [System] [MY-013172] [Server] Received SHUTDOWN from user <via user signal>. Shutting down mysqld (Version: 8.0.30).
mysql 04:42:24.66 DEBUG ==> Waiting for mysql to unlock db files
/bitnami/mysql/data/ibdata1:
542022-08-28T04:42:25.337410Z 0 [System] [MY-010910] [Server] /opt/bitnami/mysql/bin/mysqld: Shutdown complete (mysqld 8.0.30) Source distribution.
mysql 04:42:29.78 INFO ==> ** MySQL setup finished! **
mysql 04:42:29.89 INFO ==> ** Starting MySQL **
2022-08-28T04:42:30.338842Z 0 [Warning] [MY-011068] [Server] The syntax 'log_slave_updates' is deprecated and will be removed in a future release. Please use log_replica_updates instead.
2022-08-28T04:42:30.338892Z 0 [Warning] [MY-011069] [Server] The syntax '--master-info-repository' is deprecated and will be removed in a future release.
2022-08-28T04:42:30.338920Z 0 [Warning] [MY-011069] [Server] The syntax '--relay-log-info-repository' is deprecated and will be removed in a future release.
2022-08-28T04:42:30.339278Z 0 [Warning] [MY-010918] [Server] 'default_authentication_plugin' is deprecated and will be removed in a future release. Please use authentication_policy instead.
2022-08-28T04:42:30.339343Z 0 [System] [MY-010116] [Server] /opt/bitnami/mysql/bin/mysqld (mysqld 8.0.30) starting as process 1
2022-08-28T04:42:30.345315Z 0 [Warning] [MY-013242] [Server] --character-set-server: 'utf8' is currently an alias for the character set UTF8MB3, but will be an alias for UTF8MB4 in a future release. Please consider using UTF8MB4 in order to be unambiguous.
2022-08-28T04:42:30.345347Z 0 [Warning] [MY-013244] [Server] --collation-server: 'utf8mb3_general_ci' is a collation of the deprecated character set UTF8MB3. Please consider using UTF8MB4 with an appropriate collation instead.
2022-08-28T04:42:30.407118Z 1 [System] [MY-013576] [InnoDB] InnoDB initialization has started.
2022-08-28T04:42:32.679077Z 1 [System] [MY-013577] [InnoDB] InnoDB initialization has ended.
2022-08-28T04:42:34.233747Z 0 [Warning] [MY-010068] [Server] CA certificate ca.pem is self signed.
2022-08-28T04:42:34.233881Z 0 [System] [MY-013602] [Server] Channel mysql_main configured to support TLS. Encrypted connections are now supported for this channel.
2022-08-28T04:42:34.420391Z 0 [System] [MY-011323] [Server] X Plugin ready for connections. Bind-address: '::' port: 33060, socket: /tmp/mysqlx.sock
2022-08-28T04:42:34.421390Z 0 [System] [MY-010931] [Server] /opt/bitnami/mysql/bin/mysqld: ready for connections. Version: '8.0.30' socket: '/opt/bitnami/mysql/tmp/mysql.sock' port: 3306 Source distribution.
[root@master ~]# kubectl describe pod/bucai-mysql-secondary-0 -n bucai
Name: bucai-mysql-secondary-0
Namespace: bucai
Priority: 0
Node: node1/10.65.42.70
Start Time: Sun, 28 Aug 2022 12:26:01 +0800
Labels: app.kubernetes.io/component=secondary
app.kubernetes.io/instance=bucai-mysql
app.kubernetes.io/managed-by=Helm
app.kubernetes.io/name=mysql
controller-revision-hash=bucai-mysql-secondary-9d7678dcf
helm.sh/chart=mysql-9.3.1
statefulset.kubernetes.io/pod-name=bucai-mysql-secondary-0
Annotations: checksum/configuration: 48e5bd6663bbefad3cbed2f0df3dacff28804dc2ecd50fe2a12bffd4ac7c96cd
Status: Running
IP: 10.244.1.11
IPs:
IP: 10.244.1.11
Controlled By: StatefulSet/bucai-mysql-secondary
Containers:
mysql:
Container ID: docker://cf81001ed544534ab0d92dde39978fc3081c841d2bcdd87033ea8d963e6c047c
Image: docker.io/bitnami/mysql:8.0.30-debian-11-r6
Image ID: docker-pullable://bitnami/mysql@sha256:f68141d76c1db11cac639beca4009bd381e734925c2a29b648d51d28136b7258
Port: 3306/TCP
Host Port: 0/TCP
State: Running
Started: Sun, 28 Aug 2022 12:41:25 +0800
Last State: Terminated
Reason: Completed
Exit Code: 0
Started: Sun, 28 Aug 2022 12:38:45 +0800
Finished: Sun, 28 Aug 2022 12:41:24 +0800
Ready: False
Restart Count: 5
Liveness: exec [/bin/bash -ec password_aux="${MYSQL_MASTER_ROOT_PASSWORD:-}"
if [[ -f "${MYSQL_MASTER_ROOT_PASSWORD_FILE:-}" ]]; then
password_aux=$(cat "$MYSQL_MASTER_ROOT_PASSWORD_FILE")
fi
mysqladmin status -uroot -p"${password_aux}"
] delay=5s timeout=1s period=10s #success=1 #failure=3
Readiness: exec [/bin/bash -ec password_aux="${MYSQL_MASTER_ROOT_PASSWORD:-}"
if [[ -f "${MYSQL_MASTER_ROOT_PASSWORD_FILE:-}" ]]; then
password_aux=$(cat "$MYSQL_MASTER_ROOT_PASSWORD_FILE")
fi
mysqladmin status -uroot -p"${password_aux}"
] delay=5s timeout=1s period=10s #success=1 #failure=3
Startup: exec [/bin/bash -ec password_aux="${MYSQL_MASTER_ROOT_PASSWORD:-}"
if [[ -f "${MYSQL_MASTER_ROOT_PASSWORD_FILE:-}" ]]; then
password_aux=$(cat "$MYSQL_MASTER_ROOT_PASSWORD_FILE")
fi
mysqladmin status -uroot -p"${password_aux}"
] delay=15s timeout=1s period=10s #success=1 #failure=15
Environment:
BITNAMI_DEBUG: true
MYSQL_REPLICATION_MODE: slave
MYSQL_MASTER_HOST: bucai-mysql-primary
MYSQL_MASTER_PORT_NUMBER: 3306
MYSQL_MASTER_ROOT_USER: root
MYSQL_REPLICATION_USER: replicator
MYSQL_MASTER_ROOT_PASSWORD: <set to the key 'mysql-root-password' in secret 'bucai-mysql'> Optional: false
MYSQL_REPLICATION_PASSWORD: <set to the key 'mysql-replication-password' in secret 'bucai-mysql'> Optional: false
Mounts:
/bitnami/mysql from data (rw)
/opt/bitnami/mysql/conf/my.cnf from config (rw,path="my.cnf")
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-wmzbk (ro)
Conditions:
Type Status
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
data:
Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
ClaimName: data-bucai-mysql-secondary-0
ReadOnly: false
config:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: bucai-mysql-secondary
Optional: false
kube-api-access-wmzbk:
Type: Projected (a volume that contains injected data from multiple sources)
TokenExpirationSeconds: 3607
ConfigMapName: kube-root-ca.crt
ConfigMapOptional: <nil>
DownwardAPI: true
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedScheduling 17m default-scheduler 0/3 nodes are available: 3 pod has unbound immediate PersistentVolumeClaims.
Normal Scheduled 17m default-scheduler Successfully assigned bucai/bucai-mysql-secondary-0 to node1
Normal Pulling 17m kubelet Pulling image "docker.io/bitnami/mysql:8.0.30-debian-11-r6"
Normal Pulled 15m kubelet Successfully pulled image "docker.io/bitnami/mysql:8.0.30-debian-11-r6" in 1m18.592380382s
Normal Killing 13m kubelet Container mysql failed startup probe, will be restarted
Normal Created 12m (x2 over 15m) kubelet Created container mysql
Normal Pulled 12m kubelet Container image "docker.io/bitnami/mysql:8.0.30-debian-11-r6" already present on machine
Warning Unhealthy 11m (x6 over 15m) kubelet Startup probe failed: mysqladmin: [Warning] Using a password on the command line interface can be insecure.
mysqladmin: connect to server at 'localhost' failed
error: 'Can't connect to local MySQL server through socket '/opt/bitnami/mysql/tmp/mysql.sock' (2)'
Check that mysqld is running and that the socket: '/opt/bitnami/mysql/tmp/mysql.sock' exists!
Normal Started 7m16s (x4 over 15m) kubelet Started container mysql
Warning Unhealthy 2m9s (x53 over 14m) kubelet Startup probe failed: mysqladmin: [Warning] Using a password on the command line interface can be insecure.
mysqladmin: connect to server at 'localhost' failed
error: 'Access denied for user 'root'@'localhost' (using password: YES)'
Hi @notbucai, I can see this error in the startup probe:
mysqladmin: connect to server at 'localhost' failed
error: 'Access denied for user 'root'@'localhost' (using password: YES)'
Could it be possible that you deployed MySQL in a cluster where a PVC already existed for the deployment, and the password had been randomly generated?
If so, it could be possible it is trying to use a different password. You can try and re-deploy to a different namespace and see if you get the same errors, with a different deployment name, or removing existing PVCs. Remember to specify a password when deploying to avoid this issue in the future.
This Issue has been automatically marked as "stale" because it has not had recent activity (for 15 days). It will be closed if no further activity occurs. Thanks for the feedback.
Due to the lack of activity in the last 5 days since it was marked as "stale", we proceed to close this Issue. Do not hesitate to reopen it later if necessary.