amazonriver
amazonriver copied to clipboard
hellobike
一个表几分钟不更新，第一次更新的数据会丢失
我之前的描述有问题，不是必现，是出现的概率比较高，下面配置信息里面我配了两个表，测试过都存在这种问题
pg版本： postgresql 9.6.12
表结构： CREATE TABLE db40t.t_mon_kafka ( update_cnt numeric(20,0), update_time timestamp without time zone )
更新脚本： update db40t.t_mon_kafka set update_time = now() ,update_cnt = update_cnt + 1;
配置信息 { "pg_dump_path": "", "subscribes": [{ "dump": false, "slotName": "amazonriver", "pgConnConf": { "host": "192.168.216.87", "port": 6432, "database": "grabdb", "user": "replica", "password": "******" }, "rules": [ { "table": "dbm_monitor_timeliness", "pks": ["table_name"], "topic": "db40.dbm_monitor_timeliness" }, { "table": "t_mon_kafka", "topic": "db40t.t_mon_kafka" } ], "kafkaConf": { "addrs": ["192.168.216.158:9092","192.168.216.157:9092","192.168.200.94:9092"] }, "retry": 0 }], "prometheus_address": ":8084" }
t_mon_kafka这个表没有主键，但是另一个表dbm_monitor_timeliness是有主键的，先出现了丢数据的问题；然后我用t_mon_kafka这个表做了一个监控，每半小时更新一次，数据丢失概率在50%以上；
现在，我已经给表加上主键了， 再测试一天时间，结果反馈给你！
t_mon_kafka这个表我加了主键，下午测试3个半小时，每半个小时更新一次数据，一共更新了7次，kafka只接收到4条消息，丢了3条，同时运行你们的tunnel这个工具，没有丢数据，但是tunnel也有问题，运行一段时间报错。 更新时间间隔不长的情况不会丢数据 { "table": "t_mon_kafka", "pks": ["seq"], "topic": "db40t.t_mon_kafka" }
INFO[0000] Initializing new client
INFO[0000] ClientID is the default of 'sarama', you should consider setting it to something application-specific.
INFO[0000] ClientID is the default of 'sarama', you should consider setting it to something application-specific.
INFO[0000] client/metadata fetching metadata for all topics from broker 172.16.0.79:9092
INFO[0000] Connected to broker at 172.16.0.79:9092 (unregistered)
INFO[0000] client/brokers registered new broker #0 at ubuntu:9092
INFO[0000] client/metadata found some partitions to be leaderless
INFO[0000] client/metadata retrying after 250ms... (3 attempts remaining)
INFO[0000] client/metadata fetching metadata for all topics from broker 172.16.0.79:9092
INFO[0000] client/metadata found some partitions to be leaderless
INFO[0000] client/metadata retrying after 250ms... (2 attempts remaining)
INFO[0000] client/metadata fetching metadata for all topics from broker 172.16.0.79:9092
INFO[0000] client/metadata found some partitions to be leaderless
INFO[0000] client/metadata retrying after 250ms... (1 attempts remaining)
INFO[0000] client/metadata fetching metadata for all topics from broker 172.16.0.79:9092
INFO[0000] client/metadata found some partitions to be leaderless
INFO[0000] Successfully initialized new client
INFO[0000] start prometheus handler
INFO[0000] start amazon...
INFO[0000] start stream for slot_for_kafka
DEBU[0000] send heartbeat
INFO[0000] handle wal data: &{BEGIN map[] 1568083179376 200770952
{ "pg_dump_path": "", "subscribes": [{
"dump": true,
"slotName": "slot_for_kafka",
"pgConnConf": {
"host": "172.16.0.45",
"port": 5432,
"database": "test",
"user": "test_rep",
"password": "123456"
},
"rules": [
{
"table": "user",
"pks": ["id"],
"topic": "user_topic02"
}
],
"kafkaConf": {
"addrs": ["172.16.0.79:9092"]
},
"retry": -1
}],
"prometheus_address": ":8080"
}
