dtle
dtle copied to clipboard
全量传输时,遇到带有bit类型的表,job出现dead状态
Description
在使用dtle从MySQL到MySQL的单库全量传输数据时,遇到表中带有bit类型的字段时会在日志中输出mysql.extractor. error,导致job为dead状态。
Steps to reproduce the issue
- job.json
{
"Name":"src_to_dest",
"Tasks":[
{
"Type":"Src",
"Config":{
"Gtid":"",
"ReplicateDoDb":[
{
"TableSchema":"mcc",
"Tables":[
]
}
],
"ConnectionConfig":{
"Host":"192.168.86.101",
"Port":"3306",
"User":"dba",
"Password":"123456",
"Charset":"utf8mb4"
}
}
},
{
"Type":"Dest",
"Config":{
"ConnectionConfig":{
"Host":"192.168.86.102",
"Port":"3307",
"User":"dba",
"Password":"123456",
"Charset":"utf8mb4"
}
}
}
]
}
2.表结构
CREATE TABLE `app_a` (
`id` int(11) NOT NULL AUTO_INCREMENT COMMENT ,
`p_id` int(11) NOT NULL COMMENT ,
`p_name` varchar(20) CHARACTER SET utf8 COLLATE utf8_unicode_ci NOT NULL ,
`s_name` varchar(20) CHARACTER SET utf8 COLLATE utf8_unicode_ci NOT NULL ,
`s_identify` varchar(20) CHARACTER SET utf8 COLLATE utf8_unicode_ci NOT NULL ,
`is_enabled` bit(1) NOT NULL DEFAULT b'0' ,
`update_by` varchar(64) CHARACTER SET utf8 COLLATE utf8_unicode_ci DEFAULT NULL ,
`update_time` datetime DEFAULT NULL ,
`create_by` varchar(64) CHARACTER SET utf8 COLLATE utf8_unicode_ci NOT NULL ,
`create_time` datetime NOT NULL ,
PRIMARY KEY (`id`),
UNIQUE KEY `uniq_index_scene_identify` (`s_identify`) USING BTREE
) ENGINE=InnoDB AUTO_INCREMENT=4 DEFAULT CHARSET=utf8 ROW_FORMAT=DYNAMIC
- 创建任务后,发现日志中有如下错误信息
ERRO[0053] mysql.extractor. error: job=55af1e10-5ab1-fc61-8795-cc530128bb41
DEBU[0053] *** Extractor.Shutdown job=55af1e10-5ab1-fc61-8795-cc530128bb41
INFO[0053] mysql.extractor: Shutting down job=55af1e10-5ab1-fc61-8795-cc530128bb41
INFO[0053] mysql.extractor: Step 3: committing transaction job=55af1e10-5ab1-fc61-8795-cc530128bb41
DEBU[0053] setState 4
DEBU[0053] setState.SaveState
DEBU[0053] getChunkData. n_row: 0 job=55af1e10-5ab1-fc61-8795-cc530128bb41
INFO[0053] mysql.dumper: nRows < d.chunkSize. 0 2000 job=55af1e10-5ab1-fc61-8795-cc530128bb41
INFO[0053] mysql.dumper: nRows == 0. dump finished. 0 2000 job=55af1e10-5ab1-fc61-8795-cc530128bb41
DEBU[0053] mysql.applier: Exec [CREATE DATABASE IF NOT EXISTS `mcc`] job=55af1e10-5ab1-fc61-8795-cc530128bb41
ERRO[0053] mysql.extractor. error: job=55af1e10-5ab1-fc61-8795-cc530128bb41
DEBU[0053] mysql.applier: Exec [USE `mcc`] job=55af1e10-5ab1-fc61-8795-cc530128bb41
DEBU[0053] mysql.applier: Exec [CREATE TABLE `rspl` (
`id` bigint(20) NOT NULL AUTO_INCREMENT ,
`log_type` tinyint(4) NOT NULL ,
`operate_con] job=55af1e10-5ab1-fc61-8795-cc530128bb41
DEBU[0053] mysql.applier: Exec [CREATE DATABASE IF NOT EXISTS `mcc`] job=55af1e10-5ab1-fc61-8795-cc530128bb41
DEBU[0053] mysql.applier: Exec [USE `mcc`] job=55af1e10-5ab1-fc61-8795-cc530128bb41
DEBU[0053] mysql.applier: Exec [CREATE TABLE `rsp` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`project_name` varchar(64) NOT NULL,
`source_channel_id` int(11) NOT NULL DEFAULT '999',
`create_time` datetime ] job=55af1e10-5ab1-fc61-8795-cc530128bb41
DEBU[0053] mysql.applier: Exec [CREATE DATABASE IF NOT EXISTS `mcc`] job=55af1e10-5ab1-fc61-8795-cc530128bb41
DEBU[0053] mysql.applier: Exec [USE `mcc`] job=55af1e10-5ab1-fc61-8795-cc530128bb41
DEBU[0053] mysql.applier: Exec [CREATE TABLE `rsrsc` (
`id` int(11) NOT NULL,
`scene_identify` varchar(64) NOT NULL,
`source_channel` int(11) NOT NULL job=55af1e10-5ab1-fc61-8795-cc530128bb41
DEBU[0053] Worker.SaveState: lock configLock="&{{0 0} 0 0 0 0}" task="&{Src MySQL map[ConnectionConfig:map[Host:192.168.86.101 Port:3306 User:dba01 Password:3512932504 Charset:utf8mb4] TrafficAgainstLimits:0 NatsAddr:127.0.0.1:8193 Gtid: ReplicateDoDb:[map[TableSchema:mcc Tables:[]]]] 0xc000a944a0 false []}"
DEBU[0053] Worker.SaveState: after unlock task="&{Src MySQL map[NatsAddr:127.0.0.1:8193 Gtid: ReplicateDoDb:[map[TableSchema:mcc Tables:[]]] ConnectionConfig:map[Host:192.168.86.101 Port:3306 User:dba01 Password:3512932504 Charset:utf8mb4] TrafficAgainstLimits:0] 0xc000a944a0 false []}"
DEBU[0053] updater
ERRO[0053] agent: Task "Src" for alloc "6348621e-e849-f093-00ee-ad659f1b5c64" failed: Wait returned exit code 2, and error allocId=6348621e-e849-f093-00ee-ad659f1b5c64 taskTuype=Src waitRes="Wait returned exit code 2, and error "
INFO[0053] agent: Not restarting task: Src for alloc: 6348621e-e849-f093-00ee-ad659f1b5c64 allocId=6348621e-e849-f093-00ee-ad659f1b5c64 taskType=Src
DEBU[0053] setState restart 1
DEBU[0053] setState.SaveState
Describe the results you received
src实例中已经不存在binlog dump线程,且job状态为dead
shell > dtle job-status
ID Type Status
55af1e10-5ab1-fc61-8795-cc530128bb41 synchronous dead
Describe the results you expected
job启动后能够正常传输带有bit类型的数据
Output of ./dtle version:**
Dtle 2.19.11.0 (git: 2.19.11.0 292c35814fac4e213524446cd8285968087eb098)
Additional information
曾尝试过将bit类型的字段删掉或修改为int类型可以正常传输,只有为bit时会导致job死亡。
Additional details (log, config, job config etc):
- mysql version:5.7.19
- dtle.conf
name = "test"
data_dir = "/data/dtle/data/"
bind_addr = "127.0.0.1"
log_level = "DEBUG"
log_file = "/var/log/dtle/dtle.log"
log_to_stdout = true
# Modify our port to avoid a collision with server
ports {
http = 8190
}
# Enable the manager
manager {
enabled = true
# Self-elect, should be 3 or 5 for production,
# Addresses to attempt to join when the server starts.
join = ["127.0.0.1"]
}
advertise {
http = "127.0.0.1"
rpc = "127.0.0.1"
serf = "127.0.0.1"
nats = "127.0.0.1"
}
# Enable the agent
agent {
enabled = true
managers = ["127.0.0.1:8191"]
}