chunjun icon indicating copy to clipboard operation
chunjun copied to clipboard

flinkx1.12性能问题:mysql-hive 如何提高离线同步效率

Open biandou1313 opened this issue 2 years ago • 1 comments

Search before asking

  • [ ] I had searched in the issues and found no similar question.

  • [ ] I had googled my question but i didn't get any help.

  • [ ] I had read the documentation: ChunJun doc but it didn't help me.

Description

1、数据量9000万+时,mysql同步至hive时,一是不能设置多通道,因为hive sink组件不支持 二是、需要设置fetchSize 实现可读,我设置2048000,但是同步依然很慢,大概需要4个小时 2、j机器性能很好,mysql数据库服务器也很好,我想问如何提交同步效率 1660180537(1)

1660180613(1)

Code of Conduct

biandou1313 avatar Aug 11 '22 01:08 biandou1313

@biandou1313 hdfs相关插件支持并发写入的。你这边的配置发下

zoudaokoulife avatar Aug 11 '22 07:08 zoudaokoulife