charts
charts copied to clipboard
Published
20 hours ago •
bitnami
bitnami
MariaDB Cluster Replication issue, Using architecture=replication, On Secondary pod ERROR 1045 (28000): Access denied for user 'root'@'localhost', Both master and slave status is EmptySet
Name and Version
bitnami/mariadb
What steps will reproduce the bug?
helm install mariadb bitnami/mariadb \
--set architecture=replication \
--set secondary.replicaCount=1 \
--set auth.database=test \
--set auth.username=admin \
--set auth.password=Dev@1234 \
--set auth.rootPassword=Dev@1234 \
--set auth.replicationPassword=Dev@1234
I have deployed MariaDB on IKS cluster and I am seeing this error
Are you using any custom parameters or values?
No
What is the expected behavior?
I wanted to run the Primary MariaDB pod and Secondary Pod on the same namespace and replicate the data from Primary to secondary pod and application access the data between the Primary and Secondary pods
What do you see instead?
mariadb 06:26:43.51
mariadb 06:26:43.51 Welcome to the Bitnami mariadb container
mariadb 06:26:43.51 Subscribe to project updates by watching https://github.com/bitnami/bitnami-docker-mariadb
mariadb 06:26:43.52 Submit issues and feature requests at https://github.com/bitnami/bitnami-docker-mariadb/issues
mariadb 06:26:43.52
mariadb 06:26:43.52 INFO ==> ** Starting MariaDB setup **
mariadb 06:26:43.58 INFO ==> Validating settings in MYSQL_*/MARIADB_* env vars
mariadb 06:26:43.59 INFO ==> Initializing mariadb database
mariadb 06:26:43.60 DEBUG ==> Ensuring expected directories/files exist
mariadb 06:26:43.62 WARN ==> The mariadb configuration file '/opt/bitnami/mariadb/conf/my.cnf' is not writable. Configurations based on environment variables will not be applied for this file.
mariadb 06:26:43.62 INFO ==> Using persisted data
mariadb 06:26:43.67 INFO ==> Running mysql_upgrade
mariadb 06:26:43.68 INFO ==> Starting mariadb in background
2022-07-25 6:26:43 0 [Note] /opt/bitnami/mariadb/sbin/mysqld (server 10.6.8-MariaDB-log) starting as process 55 ...
2022-07-25 6:26:43 0 [Note] InnoDB: Compressed tables use zlib 1.2.11
2022-07-25 6:26:43 0 [Note] InnoDB: Using transactional memory
2022-07-25 6:26:43 0 [Note] InnoDB: Number of pools: 1
2022-07-25 6:26:43 0 [Note] InnoDB: Using crc32 + pclmulqdq instructions
2022-07-25 6:26:43 0 [Note] mysqld: O_TMPFILE is not supported on /opt/bitnami/mariadb/tmp (disabling future attempts)
2022-07-25 6:26:43 0 [Note] InnoDB: Using Linux native AIO
2022-07-25 6:26:43 0 [Note] InnoDB: Initializing buffer pool, total size = 134217728, chunk size = 134217728
2022-07-25 6:26:43 0 [Note] InnoDB: Completed initialization of buffer pool
2022-07-25 6:26:47 0 [Note] InnoDB: 128 rollback segments are active.
2022-07-25 6:26:47 0 [Note] InnoDB: Creating shared tablespace for temporary tables
2022-07-25 6:26:47 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ...
2022-07-25 6:26:47 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB.
2022-07-25 6:26:47 0 [Note] InnoDB: 10.6.8 started; log sequence number 42365; transaction id 14
2022-07-25 6:26:47 0 [Note] InnoDB: Loading buffer pool(s) from /bitnami/mariadb/data/ib_buffer_pool
2022-07-25 6:26:47 0 [Note] Plugin 'FEEDBACK' is disabled.
2022-07-25 6:26:48 0 [Note] InnoDB: Buffer pool(s) load completed at 220725 6:26:48
2022-07-25 6:26:48 0 [Note] Server socket created on IP: '127.0.0.1'.
2022-07-25 6:26:48 0 [Warning] 'proxies_priv' entry '@% root@xxx-mariadb-secondary-0' ignored in --skip-name-resolve mode.
2022-07-25 6:26:48 0 [Note] /opt/bitnami/mariadb/sbin/mysqld: ready for connections.
Version: '10.6.8-MariaDB-log' socket: '/opt/bitnami/mariadb/tmp/mysql.sock' port: 3306 Source distribution
2022-07-25 6:26:49 4 [Warning] Access denied for user 'root'@'localhost' (using password: YES)
Reading datadir from the MariaDB server failed. Got the following error when executing the 'mysql' command line client
ERROR 1045 (28000): Access denied for user 'root'@'localhost' (using password: YES)
FATAL ERROR: Upgrade failed
mariadb 06:26:49.78 INFO ==> Stopping mariadb
2022-07-25 6:26:49 0 [Note] /opt/bitnami/mariadb/sbin/mysqld (initiated by: unknown): Normal shutdown
2022-07-25 6:26:49 0 [Note] InnoDB: FTS optimize thread exiting.
2022-07-25 6:26:49 0 [Note] InnoDB: Starting shutdown...
2022-07-25 6:26:49 0 [Note] InnoDB: Dumping buffer pool(s) to /bitnami/mariadb/data/ib_buffer_pool
2022-07-25 6:26:49 0 [Note] InnoDB: Buffer pool(s) dump completed at 220725 6:26:49
2022-07-25 6:26:50 0 [Note] InnoDB: Removed temporary tablespace data file: "./ibtmp1"
2022-07-25 6:26:50 0 [Note] InnoDB: Shutdown completed; log sequence number 42377; transaction id 15
2022-07-25 6:26:50 0 [Note] /opt/bitnami/mariadb/sbin/mysqld: Shutdown complete
Additional information
MariaDB Primary and Secondary replication is not working
logs -f xxx-mariadb-secondary-0
2022-07-26 06:02:15+00:00 [Note] [Entrypoint]: Entrypoint script for MariaDB Server 1:10.8.3+maria~jammy started.
2022-07-26 06:02:16+00:00 [Note] [Entrypoint]: MariaDB upgrade information missing, assuming required
2022-07-26 06:02:16+00:00 [Note] [Entrypoint]: MariaDB upgrade (mariadb-upgrade) required, but skipped due to $MARIADB_AUTO_UPGRADE setting
2022-07-26 6:02:16 0 [Note] mariadbd (server 10.8.3-MariaDB-1:10.8.3+maria~jammy) starting as process 1 ...
2022-07-26 6:02:16 0 [Note] InnoDB: Compressed tables use zlib 1.2.11
2022-07-26 6:02:16 0 [Note] InnoDB: Using transactional memory
2022-07-26 6:02:16 0 [Note] InnoDB: Number of transaction pools: 1
2022-07-26 6:02:16 0 [Note] InnoDB: Using crc32 + pclmulqdq instructions
2022-07-26 6:02:16 0 [Note] mariadbd: O_TMPFILE is not supported on /tmp (disabling future attempts)
2022-07-26 6:02:16 0 [Warning] mariadbd: io_uring_queue_init() failed with ENOSYS: check seccomp filters, and the kernel version (newer than 5.1 required)
2022-07-26 6:02:16 0 [Warning] InnoDB: liburing disabled: falling back to innodb_use_native_aio=OFF
2022-07-26 6:02:16 0 [Note] InnoDB: Initializing buffer pool, total size = 128.000MiB, chunk size = 2.000MiB
2022-07-26 6:02:16 0 [Note] InnoDB: Completed initialization of buffer pool
2022-07-26 6:02:16 0 [Note] InnoDB: File system buffers for log disabled (block size=4096 bytes)
2022-07-26 6:02:19 0 [Note] InnoDB: 128 rollback segments are active.
2022-07-26 6:02:19 0 [Note] InnoDB: Setting file './ibtmp1' size to 12.000MiB. Physically writing the file full; Please wait ...
2022-07-26 6:02:19 0 [Note] InnoDB: File './ibtmp1' size is now 12.000MiB.
2022-07-26 6:02:19 0 [Note] InnoDB: log sequence number 46693; transaction id 14
2022-07-26 6:02:19 0 [Note] InnoDB: Loading buffer pool(s) from /xxx/mariadb/ib_buffer_pool
2022-07-26 6:02:19 0 [Note] Plugin 'FEEDBACK' is disabled.
2022-07-26 6:02:19 0 [Warning] You need to use --log-bin to make --expire-logs-days or --binlog-expire-logs-seconds work.
2022-07-26 6:02:19 0 [Note] Server socket created on IP: '0.0.0.0'.
2022-07-26 6:02:19 0 [Note] Server socket created on IP: '::'.
2022-07-26 6:02:19 0 [Warning] 'user' entry 'root@xxx-mariadb-secondary-0' ignored in --skip-name-resolve mode.
2022-07-26 6:02:19 0 [Warning] 'proxies_priv' entry '@% root@xxx-mariadb-secondary-0' ignored in --skip-name-resolve mode.
2022-07-26 6:02:20 0 [Note] mariadbd: ready for connections.
Version: '10.8.3-MariaDB-1:10.8.3+maria~jammy' socket: '/run/mysqld/mysqld.sock' port: 3306 mariadb.org binary distribution
2022-07-26 6:02:20 0 [Note] InnoDB: Buffer pool(s) load completed at 220726 6:02:20
2022-07-26 6:04:27 3 [Warning] Access denied for user 'root'@'localhost' (using password: YES)
2022-07-26 6:04:42 4 [Warning] Access denied for user 'root'@'localhost' (using password: YES)
Primary Pod logs
2022-07-26 5:58:19 0 [Note] mariadbd (server 10.8.3-MariaDB-1:10.8.3+maria~jammy) starting as process 1 ...
2022-07-26 5:58:19 0 [Note] InnoDB: Compressed tables use zlib 1.2.11
2022-07-26 5:58:19 0 [Note] InnoDB: Using transactional memory
2022-07-26 5:58:19 0 [Note] InnoDB: Number of transaction pools: 1
2022-07-26 5:58:19 0 [Note] InnoDB: Using crc32 + pclmulqdq instructions
2022-07-26 5:58:19 0 [Note] mariadbd: O_TMPFILE is not supported on /tmp (disabling future attempts)
2022-07-26 5:58:19 0 [Warning] mariadbd: io_uring_queue_init() failed with ENOSYS: check seccomp filters, and the kernel version (newer than 5.1 required)
2022-07-26 5:58:19 0 [Warning] InnoDB: liburing disabled: falling back to innodb_use_native_aio=OFF
2022-07-26 5:58:19 0 [Note] InnoDB: Initializing buffer pool, total size = 128.000MiB, chunk size = 2.000MiB
2022-07-26 5:58:19 0 [Note] InnoDB: Completed initialization of buffer pool
2022-07-26 5:58:19 0 [Note] InnoDB: 128 rollback segments are active.
2022-07-26 5:58:19 0 [Note] InnoDB: Setting file './ibtmp1' size to 12.000MiB. Physically writing the file full; Please wait ...
2022-07-26 5:58:19 0 [Note] InnoDB: File './ibtmp1' size is now 12.000MiB.
2022-07-26 5:58:19 0 [Note] InnoDB: log sequence number 409169; transaction id 333
2022-07-26 5:58:19 0 [Note] Plugin 'FEEDBACK' is disabled.
2022-07-26 5:58:19 0 [Warning] You need to use --log-bin to make --expire-logs-days or --binlog-expire-logs-seconds work.
2022-07-26 5:58:19 0 [Note] Server socket created on IP: '0.0.0.0'.
2022-07-26 5:58:19 0 [Note] Server socket created on IP: '::'.
2022-07-26 5:58:19 0 [Note] InnoDB: Loading buffer pool(s) from /dcaw/mariadb/ib_buffer_pool
2022-07-26 5:58:19 0 [Note] InnoDB: Buffer pool(s) load completed at 220726 5:58:19
2022-07-26 5:58:19 0 [Note] mariadbd: ready for connections.
Version: '10.8.3-MariaDB-1:10.8.3+maria~jammy' socket: '/run/mysqld/mysqld.sock' port: 3306 mariadb.org binary distribution
Hi Team,
We are still waiting for a solution for this issue, Can you pls suggest us to fix the Cluster issue
Regards,
Manjunath VS
Hi @manjunath-vs
Does this error appear immediately after the installation? Have you tried to log into the database with the credentials provided in the helm install command? If you try to get the content of the secrets (kubectl get secret --namespace default mariadb -o jsonpath="{.data.mariadb-root-password}" | base64 -d), are you getting the same password that you set in the install command?
Hi Fmulero,
After MariaDB helm chart deployment I am seeing these updates ** Please be patient while the chart is being deployed ** The chart has been deployed in diagnostic mode. All probes have been disabled and the command has been overwritten with:
command:
- sleep
args:
- infinity
Get the list of pods by executing:
kubectl get pods --namespace dcaw-devops -l app.kubernetes.io/instance=mariadb
Access the pod you want to debug by executing
kubectl exec --namespace dcaw-devops -ti <NAME OF THE POD> -- bash
In order to replicate the container startup scripts execute this command:
/opt/bitnami/scripts/mariadb/entrypoint.sh /opt/bitnami/scripts/mariadb/run.sh
So based on these updates I have exec into primary pod and ran this command
/opt/bitnami/scripts/mariadb/entrypoint.sh /opt/bitnami/scripts/mariadb/run.sh
And I see this is stuck at this stage
sys.x$user_summary_by_file_io_type OK
sys.x$user_summary_by_stages OK
sys.x$user_summary_by_statement_latency OK
sys.x$user_summary_by_statement_type OK
sys.x$wait_classes_global_by_avg_latency OK
sys.x$wait_classes_global_by_latency OK
sys.x$waits_by_host_by_latency OK
sys.x$waits_by_user_by_latency OK
sys.x$waits_global_by_latency OK
Phase 4/7: Running 'mysql_fix_privilege_tables'
Phase 5/7: Fixing table and database names
Phase 6/7: Checking and upgrading tables
Processing databases
information_schema
performance_schema
sys
sys.sys_config OK
test
Phase 7/7: Running 'FLUSH PRIVILEGES'
OK
find: '/docker-entrypoint-startdb.d/': No such file or directory
mariadb 16:58:25.50 INFO ==> Stopping mariadb
mariadb 16:58:26.50 DEBUG ==> Waiting for mariadb to unlock db files
mariadb 16:58:26.52 INFO ==> ** MariaDB setup finished! **
mariadb 16:58:26.54 INFO ==> ** Starting MariaDB **
2022-08-17 16:58:26 0 [Note] /opt/bitnami/mariadb/sbin/mysqld (server 10.6.9-MariaDB-log) starting as process 19 ...
on Pod logs I see these issues
2022-08-17 16:58:26 0 [Note] InnoDB: Compressed tables use zlib 1.2.11
2022-08-17 16:58:26 0 [Note] InnoDB: Number of pools: 1
2022-08-17 16:58:26 0 [Note] InnoDB: Using crc32 + pclmulqdq instructions
2022-08-17 16:58:26 0 [Note] mysqld: O_TMPFILE is not supported on /opt/bitnami/mariadb/tmp (disabling future attempts)
2022-08-17 16:58:26 0 [Note] InnoDB: Using Linux native AIO
2022-08-17 16:58:26 0 [Note] InnoDB: Initializing buffer pool, total size = 134217728, chunk size = 134217728
2022-08-17 16:58:26 0 [Note] InnoDB: Completed initialization of buffer pool
2022-08-17 16:58:30 0 [Note] InnoDB: 128 rollback segments are active.
2022-08-17 16:58:30 0 [Note] InnoDB: Creating shared tablespace for temporary tables
2022-08-17 16:58:30 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ...
2022-08-17 16:58:30 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB.
2022-08-17 16:58:30 0 [Note] InnoDB: 10.6.9 started; log sequence number 57759; transaction id 42
2022-08-17 16:58:30 0 [Note] InnoDB: Loading buffer pool(s) from /bitnami/mariadb/data/ib_buffer_pool
2022-08-17 16:58:30 0 [Note] Plugin 'FEEDBACK' is disabled.
2022-08-17 16:58:31 0 [Note] InnoDB: Buffer pool(s) load completed at 220817 16:58:31
2022-08-17 16:58:31 0 [Note] Server socket created on IP: '0.0.0.0'.
2022-08-17 16:58:31 0 [Note] Server socket created on IP: '::'.
2022-08-17 16:58:31 0 [Warning] 'proxies_priv' entry '@% root@dcaw-mariadb-primary-0' ignored in --skip-name-resolve mode.
2022-08-17 16:58:31 0 [Warning] Neither --relay-log nor --relay-log-index were used; so replication may break when this MariaDB server acts as a replica and has its hostname changed. Please use '--log-basename=#' or '--relay-log=mysqld-relay-bin' to avoid this problem.
2022-08-17 16:58:31 5 [Note] Slave I/O thread: Start asynchronous replication to master '[email protected]:3310' in log '' at position 4
2022-08-17 16:58:31 0 [Note] /opt/bitnami/mariadb/sbin/mysqld: ready for connections.
Version: '10.6.9-MariaDB-log' socket: '/opt/bitnami/mariadb/tmp/mysql.sock' port: 3306 Source distribution
2022-08-17 16:58:31 6 [Note] Slave SQL thread initialized, starting replication in log 'FIRST' at position 4, relay log './mysqld-relay-bin.000001' position: 4; GTID position '0-1-2'
2022-08-17 16:59:31 5 [ERROR] Slave I/O: error connecting to master '[email protected]:3310' - retry-time: 60 maximum-retries: 100000 message: Can't connect to server on 'dcaw-mariadb-primary.dcaw-devops.svc.cluster.local' (110 "Connection timed out"), Internal MariaDB error code: 2003
On Secondary pod If I try to login to mariadb with root id I am seeing these errors
I have no name!@dcaw-mariadb-secondary-0:/bitnami/mariadb/data$ mysql -u root -p
Enter password:
ERROR 2002 (HY000): Can't connect to local server through socket '/opt/bitnami/mariadb/tmp/mysql.sock' (2)
I have no name!@dcaw-mariadb-secondary-0:/bitnami/mariadb/data$
On pod logs I do not see any logs
bash-3.2$ k logs -f dcaw-mariadb-secondary-0
Not sure if I am missing anything here
MariaDB [(none)]> CHANGE MASTER TO
-> MASTER_HOST="dcaw-mariadb-primary.dcaw-devops.svc.cluster.local",
-> MASTER_PORT=3310,
-> MASTER_USER="root",
-> MASTER_PASSWORD="Dev@1234",
-> MASTER_LOG_FILE='mysql-bin.000014',
-> MASTER_LOG_POS=600;
Query OK, 0 rows affected (0.009 sec)
----------------------------------------------
MariaDB [(none)]> show slave status\G
*************************** 1. row ***************************
Slave_IO_State: Connecting to master
Master_Host: dcaw-mariadb-primary.dcaw-devops.svc.cluster.local
Master_User: root
Master_Port: 3310
Connect_Retry: 10
Master_Log_File: mysql-bin.000014
Read_Master_Log_Pos: 600
Relay_Log_File: mysqld-relay-bin.000001
Relay_Log_Pos: 4
Relay_Master_Log_File: mysql-bin.000014
Slave_IO_Running: Connecting
Slave_SQL_Running: Yes
Replicate_Do_DB:
Replicate_Ignore_DB:
Replicate_Do_Table:
Replicate_Ignore_Table:
Replicate_Wild_Do_Table:
Replicate_Wild_Ignore_Table:
Last_Errno: 0
Last_Error:
Skip_Counter: 0
Exec_Master_Log_Pos: 600
Relay_Log_Space: 256
Until_Condition: None
Until_Log_File:
Until_Log_Pos: 0
Master_SSL_Allowed: No
Master_SSL_CA_File:
Master_SSL_CA_Path:
Master_SSL_Cert:
Master_SSL_Cipher:
Master_SSL_Key:
Seconds_Behind_Master: NULL
Master_SSL_Verify_Server_Cert: No
Last_IO_Errno: 2003
Last_IO_Error: error connecting to master '[email protected]:3310' - retry-time: 10 maximum-retries: 100000 message: Can't connect to server on 'dcaw-mariadb-primary.dcaw-devops.svc.cluster.local' (110 "Connection timed out")
Last_SQL_Errno: 0
Last_SQL_Error:
Replicate_Ignore_Server_Ids:
Master_Server_Id: 0
Master_SSL_Crl:
Master_SSL_Crlpath:
Using_Gtid: No
Gtid_IO_Pos: 0-1-2
Replicate_Do_Domain_Ids:
Replicate_Ignore_Domain_Ids:
Parallel_Mode: optimistic
SQL_Delay: 0
SQL_Remaining_Delay: NULL
Slave_SQL_Running_State: Slave has read all relay log; waiting for more updates
Slave_DDL_Groups: 0
Slave_Non_Transactional_Groups: 0
Slave_Transactional_Groups: 0
1 row in set (0.000 sec)
MariaDB [(none)]>
MariaDB [(none)]> show master status; +------------------+----------+--------------+------------------+ | File | Position | Binlog_Do_DB | Binlog_Ignore_DB | +------------------+----------+--------------+------------------+ | mysql-bin.000014 | 600 | | | +------------------+----------+--------------+------------------+ 1 row in set (0.001 sec)
Sorry @manjunath-vs. Could you edit the comments and apply the format? is very difficult to follow them and I am getting a little bit confused. Are you running it with diagnosticMode enabled?
Hi Fmulero, Yes I am running with diagnosticMode enabled to check on issues
On Primary Pod I am seeing these errors
mariadb 17:35:07.55
mariadb 17:35:07.55 Welcome to the Bitnami mariadb container
mariadb 17:35:07.55 Subscribe to project updates by watching https://github.com/bitnami/bitnami-docker-mariadb
mariadb 17:35:07.55 Submit issues and feature requests at https://github.com/bitnami/bitnami-docker-mariadb/issues
mariadb 17:35:07.55
mariadb 17:35:07.56 INFO ==> ** Starting MariaDB setup **
mariadb 17:35:07.58 INFO ==> Validating settings in MYSQL_*/MARIADB_* env vars
mariadb 17:35:07.59 INFO ==> Initializing mariadb database
mariadb 17:35:07.59 DEBUG ==> Ensuring expected directories/files exist
mariadb 17:35:07.63 WARN ==> The mariadb configuration file '/opt/bitnami/mariadb/conf/my.cnf' is not writable. Configurations based on environment variables will not be applied for this file.
mariadb 17:35:07.63 INFO ==> Using persisted data
mariadb 17:35:07.67 INFO ==> Running mysql_upgrade
mariadb 17:35:07.68 INFO ==> Starting mariadb in background
2022-08-22 17:35:07 0 [Note] /opt/bitnami/mariadb/sbin/mysqld (server 10.6.8-MariaDB-log) starting as process 54 ...
2022-08-22 17:35:07 0 [Note] InnoDB: Compressed tables use zlib 1.2.11
2022-08-22 17:35:07 0 [Note] InnoDB: Using transactional memory
2022-08-22 17:35:07 0 [Note] InnoDB: Number of pools: 1
2022-08-22 17:35:07 0 [Note] InnoDB: Using crc32 + pclmulqdq instructions
2022-08-22 17:35:07 0 [Note] mysqld: O_TMPFILE is not supported on /opt/bitnami/mariadb/tmp (disabling future attempts)
2022-08-22 17:35:07 0 [Note] InnoDB: Using Linux native AIO
2022-08-22 17:35:07 0 [Note] InnoDB: Initializing buffer pool, total size = 134217728, chunk size = 134217728
2022-08-22 17:35:07 0 [Note] InnoDB: Completed initialization of buffer pool
2022-08-22 17:35:11 0 [Note] InnoDB: 128 rollback segments are active.
2022-08-22 17:35:11 0 [Note] InnoDB: Creating shared tablespace for temporary tables
2022-08-22 17:35:11 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ...
2022-08-22 17:35:11 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB.
2022-08-22 17:35:11 0 [Note] InnoDB: 10.6.8 started; log sequence number 52465; transaction id 14
2022-08-22 17:35:11 0 [Note] Plugin 'FEEDBACK' is disabled.
2022-08-22 17:35:11 0 [Note] InnoDB: Loading buffer pool(s) from /bitnami/mariadb/data/ib_buffer_pool
2022-08-22 17:35:12 0 [Note] InnoDB: Buffer pool(s) load completed at 220822 17:35:12
2022-08-22 17:35:12 0 [Note] Server socket created on IP: '127.0.0.1'.
2022-08-22 17:35:12 0 [Warning] 'user' entry 'root@dcaw-mariadb-primary-0' ignored in --skip-name-resolve mode.
2022-08-22 17:35:12 0 [Warning] 'user' entry '@dcaw-mariadb-primary-0' ignored in --skip-name-resolve mode.
2022-08-22 17:35:12 0 [Warning] 'proxies_priv' entry '@% root@dcaw-mariadb-primary-0' ignored in --skip-name-resolve mode.
2022-08-22 17:35:12 0 [Note] /opt/bitnami/mariadb/sbin/mysqld: ready for connections.
Version: '10.6.8-MariaDB-log' socket: '/opt/bitnami/mariadb/tmp/mysql.sock' port: 3306 Source distribution
2022-08-22 17:35:13 4 [Warning] Access denied for user 'root'@'localhost' (using password: YES)
Reading datadir from the MariaDB server failed. Got the following error when executing the 'mysql' command line client
ERROR 1045 (28000): Access denied for user 'root'@'localhost' (using password: YES)
FATAL ERROR: Upgrade failed
mariadb 17:35:13.85 INFO ==> Stopping mariadb
2022-08-22 17:35:13 0 [Note] /opt/bitnami/mariadb/sbin/mysqld (initiated by: unknown): Normal shutdown
2022-08-22 17:35:13 0 [Note] InnoDB: FTS optimize thread exiting.
2022-08-22 17:35:13 0 [Note] InnoDB: Starting shutdown...
2022-08-22 17:35:13 0 [Note] InnoDB: Dumping buffer pool(s) to /bitnami/mariadb/data/ib_buffer_pool
2022-08-22 17:35:13 0 [Note] InnoDB: Buffer pool(s) dump completed at 220822 17:35:13
2022-08-22 17:35:14 0 [Note] InnoDB: Removed temporary tablespace data file: "./ibtmp1"
2022-08-22 17:35:14 0 [Note] InnoDB: Shutdown completed; log sequence number 52477; transaction id 15
2022-08-22 17:35:14 0 [Note] /opt/bitnami/mariadb/sbin/mysqld: Shutdown complete
mariadb 17:35:14.88 DEBUG ==> Waiting for mariadb to unlock db files
On Secondary Pod I am seeing these errors
mariadb 17:35:15.61
mariadb 17:35:15.62 Welcome to the Bitnami mariadb container
mariadb 17:35:15.62 Subscribe to project updates by watching https://github.com/bitnami/bitnami-docker-mariadb
mariadb 17:35:15.62 Submit issues and feature requests at https://github.com/bitnami/bitnami-docker-mariadb/issues
mariadb 17:35:15.63
mariadb 17:35:15.63 INFO ==> ** Starting MariaDB setup **
mariadb 17:35:15.66 INFO ==> Validating settings in MYSQL_*/MARIADB_* env vars
mariadb 17:35:15.67 INFO ==> Initializing mariadb database
mariadb 17:35:15.67 DEBUG ==> Ensuring expected directories/files exist
mariadb 17:35:15.69 WARN ==> The mariadb configuration file '/opt/bitnami/mariadb/conf/my.cnf' is not writable. Configurations based on environment variables will not be applied for this file.
mariadb 17:35:15.70 INFO ==> Using persisted data
mariadb 17:35:15.81 INFO ==> Running mysql_upgrade
mariadb 17:35:15.88 INFO ==> Starting mariadb in background
2022-08-22 17:35:15 0 [Note] /opt/bitnami/mariadb/sbin/mysqld (server 10.6.8-MariaDB-log) starting as process 55 ...
2022-08-22 17:35:16 0 [Note] InnoDB: Compressed tables use zlib 1.2.11
2022-08-22 17:35:16 0 [Note] InnoDB: Using transactional memory
2022-08-22 17:35:16 0 [Note] InnoDB: Number of pools: 1
2022-08-22 17:35:16 0 [Note] InnoDB: Using crc32 + pclmulqdq instructions
2022-08-22 17:35:16 0 [Note] mysqld: O_TMPFILE is not supported on /opt/bitnami/mariadb/tmp (disabling future attempts)
2022-08-22 17:35:16 0 [Note] InnoDB: Using Linux native AIO
2022-08-22 17:35:16 0 [Note] InnoDB: Initializing buffer pool, total size = 134217728, chunk size = 134217728
2022-08-22 17:35:16 0 [Note] InnoDB: Completed initialization of buffer pool
2022-08-22 17:35:19 0 [Note] InnoDB: 128 rollback segments are active.
2022-08-22 17:35:19 0 [Note] InnoDB: Creating shared tablespace for temporary tables
2022-08-22 17:35:19 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ...
2022-08-22 17:35:19 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB.
2022-08-22 17:35:19 0 [Note] InnoDB: 10.6.8 started; log sequence number 52841; transaction id 14
2022-08-22 17:35:19 0 [Note] InnoDB: Loading buffer pool(s) from /bitnami/mariadb/data/ib_buffer_pool
2022-08-22 17:35:19 0 [Note] Plugin 'FEEDBACK' is disabled.
2022-08-22 17:35:20 0 [Note] InnoDB: Buffer pool(s) load completed at 220822 17:35:20
2022-08-22 17:35:20 0 [Note] Server socket created on IP: '127.0.0.1'.
2022-08-22 17:35:20 0 [Warning] 'proxies_priv' entry '@% root@dcaw-mariadb-secondary-0' ignored in --skip-name-resolve mode.
2022-08-22 17:35:20 0 [Note] /opt/bitnami/mariadb/sbin/mysqld: ready for connections.
Version: '10.6.8-MariaDB-log' socket: '/opt/bitnami/mariadb/tmp/mysql.sock' port: 3306 Source distribution
2022-08-22 17:35:21 4 [Warning] Access denied for user 'root'@'localhost' (using password: YES)
Reading datadir from the MariaDB server failed. Got the following error when executing the 'mysql' command line client
ERROR 1045 (28000): Access denied for user 'root'@'localhost' (using password: YES)
FATAL ERROR: Upgrade failed
mariadb 17:35:22.04 INFO ==> Stopping mariadb
2022-08-22 17:35:22 0 [Note] /opt/bitnami/mariadb/sbin/mysqld (initiated by: unknown): Normal shutdown
2022-08-22 17:35:22 0 [Note] InnoDB: FTS optimize thread exiting.
2022-08-22 17:35:22 0 [Note] InnoDB: Starting shutdown...
2022-08-22 17:35:22 0 [Note] InnoDB: Dumping buffer pool(s) to /bitnami/mariadb/data/ib_buffer_pool
2022-08-22 17:35:22 0 [Note] InnoDB: Buffer pool(s) dump completed at 220822 17:35:22
2022-08-22 17:35:22 0 [Note] InnoDB: Removed temporary tablespace data file: "./ibtmp1"
2022-08-22 17:35:22 0 [Note] InnoDB: Shutdown completed; log sequence number 52853; transaction id 15
2022-08-22 17:35:22 0 [Note] /opt/bitnami/mariadb/sbin/mysqld: Shutdown complete
mariadb 17:35:23.13 DEBUG ==> Waiting for mariadb to unlock db files
Mainly I am seeing the major error has Not sure why ERROR 1045 (28000): Access denied for user 'root'@'localhost' (using password: YES) I am using the password for the deployment and but still I am seeing this error
To be perfectly blunt, I won't read any piece of code or log not properly formatted because it's very difficult to know what is happening. Please could you edit the comments and apply the right format? Here you can find information about how to do it.
This issue #9801 are pretty similar to this one, also you have this page with common issues Could you take them a glance?
The cause is that the secondary node was restarted while waiting for the primary node ready. Adjust livenessProbe may help.
livenessProbe:
enabled: true
initialDelaySeconds: 120
periodSeconds: 10
timeoutSeconds: 1
failureThreshold: 3
successThreshold: 1
