dtle icon indicating copy to clipboard operation
dtle copied to clipboard

全量传输时,遇到带有bit类型的表,job出现dead状态

Open AstonPudding opened this issue 5 years ago • 0 comments

Description

在使用dtle从MySQL到MySQL的单库全量传输数据时,遇到表中带有bit类型的字段时会在日志中输出mysql.extractor. error,导致job为dead状态。

Steps to reproduce the issue

  1. job.json

{
	"Name":"src_to_dest",
	"Tasks":[
		{
			"Type":"Src",
			"Config":{
				"Gtid":"",
				"ReplicateDoDb":[
					{
						"TableSchema":"mcc",
						"Tables":[
						]
					}
				],
				"ConnectionConfig":{
					"Host":"192.168.86.101",
					"Port":"3306",
					"User":"dba",
					"Password":"123456",
					"Charset":"utf8mb4"
				}
			}
		},
		{
			"Type":"Dest",
			"Config":{
				"ConnectionConfig":{
					"Host":"192.168.86.102",
					"Port":"3307",
					"User":"dba",
					"Password":"123456",
					"Charset":"utf8mb4"
				}
			}
		}
	]
}

2.表结构

CREATE TABLE `app_a` (
  `id` int(11) NOT NULL AUTO_INCREMENT COMMENT ,
  `p_id` int(11) NOT NULL COMMENT ,
  `p_name` varchar(20) CHARACTER SET utf8 COLLATE utf8_unicode_ci NOT NULL ,
  `s_name` varchar(20) CHARACTER SET utf8 COLLATE utf8_unicode_ci NOT NULL ,
  `s_identify` varchar(20) CHARACTER SET utf8 COLLATE utf8_unicode_ci NOT NULL ,
  `is_enabled` bit(1) NOT NULL DEFAULT b'0' ,
  `update_by` varchar(64) CHARACTER SET utf8 COLLATE utf8_unicode_ci DEFAULT NULL ,
  `update_time` datetime DEFAULT NULL ,
  `create_by` varchar(64) CHARACTER SET utf8 COLLATE utf8_unicode_ci NOT NULL ,
  `create_time` datetime NOT NULL ,
  PRIMARY KEY (`id`),
  UNIQUE KEY `uniq_index_scene_identify` (`s_identify`) USING BTREE
) ENGINE=InnoDB AUTO_INCREMENT=4 DEFAULT CHARSET=utf8 ROW_FORMAT=DYNAMIC
  1. 创建任务后,发现日志中有如下错误信息

ERRO[0053] mysql.extractor. error:                       job=55af1e10-5ab1-fc61-8795-cc530128bb41
DEBU[0053] *** Extractor.Shutdown                        job=55af1e10-5ab1-fc61-8795-cc530128bb41
INFO[0053] mysql.extractor: Shutting down                job=55af1e10-5ab1-fc61-8795-cc530128bb41
INFO[0053] mysql.extractor: Step 3: committing transaction  job=55af1e10-5ab1-fc61-8795-cc530128bb41
DEBU[0053] setState 4                                   
DEBU[0053] setState.SaveState                           
DEBU[0053] getChunkData. n_row: 0                        job=55af1e10-5ab1-fc61-8795-cc530128bb41
INFO[0053] mysql.dumper: nRows < d.chunkSize. 0 2000     job=55af1e10-5ab1-fc61-8795-cc530128bb41
INFO[0053] mysql.dumper: nRows == 0. dump finished. 0 2000  job=55af1e10-5ab1-fc61-8795-cc530128bb41
DEBU[0053] mysql.applier: Exec [CREATE DATABASE IF NOT EXISTS `mcc`]  job=55af1e10-5ab1-fc61-8795-cc530128bb41
ERRO[0053] mysql.extractor. error:                       job=55af1e10-5ab1-fc61-8795-cc530128bb41
DEBU[0053] mysql.applier: Exec [USE `mcc`]            job=55af1e10-5ab1-fc61-8795-cc530128bb41
DEBU[0053] mysql.applier: Exec [CREATE TABLE `rspl` (
  `id` bigint(20) NOT NULL AUTO_INCREMENT ,
  `log_type` tinyint(4) NOT NULL  ,
  `operate_con]  job=55af1e10-5ab1-fc61-8795-cc530128bb41
DEBU[0053] mysql.applier: Exec [CREATE DATABASE IF NOT EXISTS `mcc`]  job=55af1e10-5ab1-fc61-8795-cc530128bb41
DEBU[0053] mysql.applier: Exec [USE `mcc`]            job=55af1e10-5ab1-fc61-8795-cc530128bb41
DEBU[0053] mysql.applier: Exec [CREATE TABLE `rsp` (
  `id` int(11) NOT NULL AUTO_INCREMENT,
  `project_name` varchar(64) NOT NULL,
  `source_channel_id` int(11) NOT NULL DEFAULT '999',
  `create_time` datetime ]  job=55af1e10-5ab1-fc61-8795-cc530128bb41
DEBU[0053] mysql.applier: Exec [CREATE DATABASE IF NOT EXISTS `mcc`]  job=55af1e10-5ab1-fc61-8795-cc530128bb41
DEBU[0053] mysql.applier: Exec [USE `mcc`]            job=55af1e10-5ab1-fc61-8795-cc530128bb41
DEBU[0053] mysql.applier: Exec [CREATE TABLE `rsrsc` (
  `id` int(11) NOT NULL,
  `scene_identify` varchar(64) NOT NULL,
  `source_channel` int(11) NOT NULL  job=55af1e10-5ab1-fc61-8795-cc530128bb41
DEBU[0053] Worker.SaveState: lock                        configLock="&{{0 0} 0 0 0 0}" task="&{Src   MySQL map[ConnectionConfig:map[Host:192.168.86.101 Port:3306 User:dba01 Password:3512932504 Charset:utf8mb4] TrafficAgainstLimits:0 NatsAddr:127.0.0.1:8193 Gtid: ReplicateDoDb:[map[TableSchema:mcc Tables:[]]]] 0xc000a944a0 false []}"
DEBU[0053] Worker.SaveState: after unlock                task="&{Src   MySQL map[NatsAddr:127.0.0.1:8193 Gtid: ReplicateDoDb:[map[TableSchema:mcc Tables:[]]] ConnectionConfig:map[Host:192.168.86.101 Port:3306 User:dba01 Password:3512932504 Charset:utf8mb4] TrafficAgainstLimits:0] 0xc000a944a0 false []}"
DEBU[0053] updater                                      
ERRO[0053] agent: Task "Src" for alloc "6348621e-e849-f093-00ee-ad659f1b5c64" failed: Wait returned exit code 2, and error   allocId=6348621e-e849-f093-00ee-ad659f1b5c64 taskTuype=Src waitRes="Wait returned exit code 2, and error "
INFO[0053] agent: Not restarting task: Src for alloc: 6348621e-e849-f093-00ee-ad659f1b5c64   allocId=6348621e-e849-f093-00ee-ad659f1b5c64 taskType=Src
DEBU[0053] setState restart 1                           
DEBU[0053] setState.SaveState                            

Describe the results you received

src实例中已经不存在binlog dump线程,且job状态为dead

shell > dtle job-status
ID                                    Type         Status
55af1e10-5ab1-fc61-8795-cc530128bb41  synchronous  dead

Describe the results you expected

job启动后能够正常传输带有bit类型的数据

Output of ./dtle version:**

Dtle 2.19.11.0 (git: 2.19.11.0 292c35814fac4e213524446cd8285968087eb098)

Additional information

曾尝试过将bit类型的字段删掉或修改为int类型可以正常传输,只有为bit时会导致job死亡。

Additional details (log, config, job config etc):

  1. mysql version:5.7.19
  2. dtle.conf

name = "test"
data_dir = "/data/dtle/data/"
bind_addr = "127.0.0.1"
log_level = "DEBUG"
log_file = "/var/log/dtle/dtle.log"
log_to_stdout = true

# Modify our port to avoid a collision with server
ports {
	http = 8190
}

# Enable the manager
manager {
	enabled = true

	# Self-elect, should be 3 or 5 for production,
	# Addresses to attempt to join when the server starts.
	join = ["127.0.0.1"]
}

advertise {
	http = "127.0.0.1"
	rpc  = "127.0.0.1"
	serf = "127.0.0.1"
	nats = "127.0.0.1"
}

# Enable the agent
agent {
	enabled = true
	managers = ["127.0.0.1:8191"]
}

AstonPudding avatar Apr 19 '20 15:04 AstonPudding