Docker-InnoDB-Cluster icon indicating copy to clipboard operation
Docker-InnoDB-Cluster copied to clipboard

Published 20 hours ago • verified publisher icon mattlord

child node not joining the cluster

Open anweshtupili opened this issue 6 years ago • 2 comments

After running compose file child nodes are not joining the cluster. child nodes are staying in starting stage for long time and restarting.

Log showing:

inno_mysql-2.1.u9eabv1l8sqm@node2 | info: attempting to join the C55153C1-1574-4972-BF06-7332D6AD46A7 group using mysql-1:6606,mysql-2:6606,mysql-3:6606 as seeds inno_mysql-2.1.u9eabv1l8sqm@node2 | Initializing database inno_mysql-2.1.u9eabv1l8sqm@node2 | Database initialized inno_mysql-2.1.u9eabv1l8sqm@node2 | MySQL init process in progress... inno_mysql-2.1.u9eabv1l8sqm@node2 | Warning: Unable to load '/usr/share/zoneinfo/iso3166.tab' as time zone. Skipping it. inno_mysql-2.1.u9eabv1l8sqm@node2 | Warning: Unable to load '/usr/share/zoneinfo/leapseconds' as time zone. Skipping it. inno_mysql-2.1.u9eabv1l8sqm@node2 | Warning: Unable to load '/usr/share/zoneinfo/tzdata.zi' as time zone. Skipping it. inno_mysql-2.1.u9eabv1l8sqm@node2 | Warning: Unable to load '/usr/share/zoneinfo/zone.tab' as time zone. Skipping it. inno_mysql-2.1.u9eabv1l8sqm@node2 | Warning: Unable to load '/usr/share/zoneinfo/zone1970.tab' as time zone. Skipping it. inno_mysql-2.1.u9eabv1l8sqm@node2 | Note (Code 1008): Can't drop database 'test'; database doesn't exist inno_mysql-2.1.u9eabv1l8sqm@node2 | inno_mysql-2.1.u9eabv1l8sqm@node2 | /entrypoint.sh: ignoring /docker-entrypoint-initdb.d/* inno_mysql-2.1.u9eabv1l8sqm@node2 | inno_mysql-2.1.u9eabv1l8sqm@node2 | mysql: [Warning] Using a password on the command line interface can be insecure. inno_mysql-2.1.u9eabv1l8sqm@node2 | mysql: [Warning] Using a password on the command line interface can be insecure. inno_mysql-2.1.u9eabv1l8sqm@node2 | Note (Code 1759): Sending passwords in plain text without SSL/TLS is extremely insecure. inno_mysql-2.1.u9eabv1l8sqm@node2 | Note (Code 1760): Storing MySQL user name or password information in the master info repository is not secure and is therefore not recommended. Please consider using the USER and PASSWORD connection options for START SLAVE; see the 'START SLAVE Syntax' in the MySQL Manual for more information. inno_mysql-2.1.u9eabv1l8sqm@node2 | inno_mysql-2.1.u9eabv1l8sqm@node2 | MySQL init process done. Ready for start up. inno_mysql-2.1.u9eabv1l8sqm@node2 | inno_mysql-2.1.vt7ebm1wknni@node2 | info: attempting to join the C55153C1-1574-4972-BF06-7332D6AD46A7 group using mysql-1:6606 as seeds inno_mysql-2.1.vt7ebm1wknni@node2 | Initializing database inno_mysql-2.1.vt7ebm1wknni@node2 | Database initialized inno_mysql-2.1.vt7ebm1wknni@node2 | MySQL init process in progress... inno_mysql-2.1.vt7ebm1wknni@node2 | Warning: Unable to load '/usr/share/zoneinfo/iso3166.tab' as time zone. Skipping it. inno_mysql-2.1.vt7ebm1wknni@node2 | Warning: Unable to load '/usr/share/zoneinfo/leapseconds' as time zone. Skipping it. inno_mysql-2.1.vt7ebm1wknni@node2 | Warning: Unable to load '/usr/share/zoneinfo/tzdata.zi' as time zone. Skipping it. inno_mysql-2.1.vt7ebm1wknni@node2 | Warning: Unable to load '/usr/share/zoneinfo/zone.tab' as time zone. Skipping it. inno_mysql-2.1.vt7ebm1wknni@node2 | Warning: Unable to load '/usr/share/zoneinfo/zone1970.tab' as time zone. Skipping it. inno_mysql-2.1.vt7ebm1wknni@node2 | Note (Code 1008): Can't drop database 'test'; database doesn't exist inno_mysql-2.1.vt7ebm1wknni@node2 | inno_mysql-2.1.vt7ebm1wknni@node2 | /entrypoint.sh: ignoring /docker-entrypoint-initdb.d/* inno_mysql-2.1.vt7ebm1wknni@node2 | inno_mysql-2.1.vt7ebm1wknni@node2 | mysql: [Warning] Using a password on the command line interface can be insecure. inno_mysql-2.1.vt7ebm1wknni@node2 | mysql: [Warning] Using a password on the command line interface can be insecure. inno_mysql-2.1.vt7ebm1wknni@node2 | Note (Code 1759): Sending passwords in plain text without SSL/TLS is extremely insecure. inno_mysql-2.1.vt7ebm1wknni@node2 | Note (Code 1760): Storing MySQL user name or password information in the master info repository is not secure and is therefore not recommended. Please consider using the USER and PASSWORD connection options for START SLAVE; see the 'START SLAVE Syntax' in the MySQL Manual for more information. inno_mysql-2.1.vt7ebm1wknni@node2 | inno_mysql-2.1.vt7ebm1wknni@node2 | MySQL init process done. Ready for start up. inno_mysql-2.1.vt7ebm1wknni@node2 | inno_mysql-2.1.3cpv7igk0vnk@node2 | info: attempting to join the C55153C1-1574-4972-BF06-7332D6AD46A7 group using mysql-1:6606,mysql-2:6606,mysql-3:6606 as seeds inno_mysql-2.1.3cpv7igk0vnk@node2 | Initializing database inno_mysql-2.1.3cpv7igk0vnk@node2 | Database initialized inno_mysql-2.1.3cpv7igk0vnk@node2 | MySQL init process in progress... inno_mysql-2.1.3cpv7igk0vnk@node2 | Warning: Unable to load '/usr/share/zoneinfo/iso3166.tab' as time zone. Skipping it. inno_mysql-2.1.3cpv7igk0vnk@node2 | Warning: Unable to load '/usr/share/zoneinfo/leapseconds' as time zone. Skipping it. inno_mysql-2.1.3cpv7igk0vnk@node2 | Warning: Unable to load '/usr/share/zoneinfo/tzdata.zi' as time zone. Skipping it. inno_mysql-2.1.3cpv7igk0vnk@node2 | Warning: Unable to load '/usr/share/zoneinfo/zone.tab' as time zone. Skipping it. inno_mysql-2.1.3cpv7igk0vnk@node2 | Warning: Unable to load '/usr/share/zoneinfo/zone1970.tab' as time zone. Skipping it. inno_mysql-2.1.3cpv7igk0vnk@node2 | Note (Code 1008): Can't drop database 'test'; database doesn't exist inno_mysql-2.1.3cpv7igk0vnk@node2 | inno_mysql-2.1.3cpv7igk0vnk@node2 | /entrypoint.sh: ignoring /docker-entrypoint-initdb.d/* inno_mysql-2.1.3cpv7igk0vnk@node2 | inno_mysql-2.1.3cpv7igk0vnk@node2 | mysql: [Warning] Using a password on the command line interface can be insecure. inno_mysql-2.1.3cpv7igk0vnk@node2 | mysql: [Warning] Using a password on the command line interface can be insecure. inno_mysql-2.1.3cpv7igk0vnk@node2 | Note (Code 1759): Sending passwords in plain text without SSL/TLS is extremely insecure. inno_mysql-2.1.3cpv7igk0vnk@node2 | Note (Code 1760): Storing MySQL user name or password information in the master info repository is not secure and is therefore not recommended. Please consider using the USER and PASSWORD connection options for START SLAVE; see the 'START SLAVE Syntax' in the MySQL Manual for more information. inno_mysql-2.1.3cpv7igk0vnk@node2 | inno_mysql-2.1.3cpv7igk0vnk@node2 | MySQL init process done. Ready for start up. inno_mysql-2.1.3cpv7igk0vnk@node2 | inno_mysql-2.1.ub63qcvutmo3@node2 | info: attempting to join the C55153C1-1574-4972-BF06-7332D6AD46A7 group using mysql-1:6606,mysql-2:6606,mysql-3:6606 as seeds inno_mysql-2.1.ub63qcvutmo3@node2 | Initializing database inno_mysql-2.1.7qd7vz1mrj8q@node2 | info: attempting to join the C55153C1-1574-4972-BF06-7332D6AD46A7 group using mysql-1:6606 as seeds inno_mysql-2.1.7qd7vz1mrj8q@node2 | Initializing database inno_mysql-2.1.7qd7vz1mrj8q@node2 | Database initialized inno_mysql-2.1.7qd7vz1mrj8q@node2 | MySQL init process in progress... inno_mysql-2.1.7qd7vz1mrj8q@node2 | Warning: Unable to load '/usr/share/zoneinfo/iso3166.tab' as time zone. Skipping it. inno_mysql-2.1.7qd7vz1mrj8q@node2 | Warning: Unable to load '/usr/share/zoneinfo/leapseconds' as time zone. Skipping it. inno_mysql-2.1.7qd7vz1mrj8q@node2 | Warning: Unable to load '/usr/share/zoneinfo/tzdata.zi' as time zone. Skipping it. inno_mysql-2.1.7qd7vz1mrj8q@node2 | Warning: Unable to load '/usr/share/zoneinfo/zone.tab' as time zone. Skipping it. inno_mysql-2.1.7qd7vz1mrj8q@node2 | Warning: Unable to load '/usr/share/zoneinfo/zone1970.tab' as time zone. Skipping it. inno_mysql-2.1.7qd7vz1mrj8q@node2 | Note (Code 1008): Can't drop database 'test'; database doesn't exist inno_mysql-2.1.7qd7vz1mrj8q@node2 | inno_mysql-2.1.7qd7vz1mrj8q@node2 | /entrypoint.sh: ignoring /docker-entrypoint-initdb.d/* inno_mysql-2.1.7qd7vz1mrj8q@node2 | inno_mysql-2.1.7qd7vz1mrj8q@node2 | mysql: [Warning] Using a password on the command line interface can be insecure. inno_mysql-2.1.7qd7vz1mrj8q@node2 | mysql: [Warning] Using a password on the command line interface can be insecure. inno_mysql-2.1.7qd7vz1mrj8q@node2 | Note (Code 1759): Sending passwords in plain text without SSL/TLS is extremely insecure. inno_mysql-2.1.7qd7vz1mrj8q@node2 | Note (Code 1760): Storing MySQL user name or password information in the master info repository is not secure and is therefore not recommended. Please consider using the USER and PASSWORD connection options for START SLAVE; see the 'START SLAVE Syntax' in the MySQL Manual for more information. inno_mysql-2.1.7qd7vz1mrj8q@node2 | inno_mysql-2.1.7qd7vz1mrj8q@node2 | MySQL init process done. Ready for start up. inno_mysql-2.1.7qd7vz1mrj8q@node2 |

anweshtupili avatar Dec 05 '18 03:12 anweshtupili

My Compose File:

version: "3" services: mysql-1: image: "mattalord/innodb-cluster" restart: always hostname: mysql-1 deploy: mode: "replicated" replicas: 1 placement: constraints: [node.hostname == node1] environment: - "MYSQL_ROOT_PASSWORD=root" - "GROUP_NAME=C55153C1-1574-4972-BF06-7332D6AD46A7" - "BOOTSTRAP=1" volumes: - mysql-1db:/var/lib/mysqlrouter networks: - inno_ydel_net

mysql-2: image: "mattalord/innodb-cluster" restart: always hostname: mysql-2 deploy: mode: "replicated" replicas: 1 placement: constraints: [node.hostname == node2] environment: - "MYSQL_ROOT_PASSWORD=root" - "GROUP_NAME=C55153C1-1574-4972-BF06-7332D6AD46A7" - "GROUP_SEEDS=mysql-1:6606" volumes: - mysql-2db:/var/lib/mysqlrouter networks: - inno_ydel_net depends_on: - mysql-1

mysql-3: image: "mattalord/innodb-cluster" restart: always hostname: mysql-3 deploy: mode: "replicated" replicas: 1 placement: constraints: [node.hostname == node3] environment: - "MYSQL_ROOT_PASSWORD=root" - "GROUP_NAME=C55153C1-1574-4972-BF06-7332D6AD46A7" - "GROUP_SEEDS=mysql-1:6606" volumes: - mysql-3db:/var/lib/mysqlrouter networks: - inno_ydel_net depends_on: - mysql-1

router: image: "mattalord/innodb-cluster" restart: always hostname: router deploy: mode: "replicated" replicas: 1 placement: constraints: [node.hostname == node4] ports: - "3306:3306" - "6606:6606" - "6446-6447:6446-6447" volumes: - mysql-routerdb:/var/lib/mysqlrouter networks: - inno_ydel_net depends_on: - mysql-1 - mysql-2 - mysql-3 environment: - "NODE_TYPE=router" - "MYSQL_ROOT_PASSWORD=root" - "MYSQL_HOST=mysql-1"

volumes: mysql-1db: driver: local mysql-2db: driver: local mysql-3db: driver: local mysql-routerdb: driver: local

networks: inno_ydel_net:

anweshtupili avatar Dec 05 '18 05:12 anweshtupili

I don't notice anything obvious in the log. What did the performance_schema.group* tables show?

This was my hobby project in the past and I'm not really working on it anymore (no longer at MySQL). I would recommend trying to use the more official MySQL work happening in this space:

  • https://mysqlhighavailability.com/setting-up-mysql-group-replication-with-mysql-docker-images/
  • https://mysqlrelease.com/2018/03/docker-compose-setup-for-innodb-cluster/

If you do find an issue though, I'd be more than happy with a pull request. :-)

mattlord avatar Dec 05 '18 17:12 mattlord