chunjun
chunjun copied to clipboard
flinkx1.12性能问题:mysql-hive 如何提高离线同步效率
Search before asking
-
[ ] I had searched in the issues and found no similar question.
-
[ ] I had googled my question but i didn't get any help.
-
[ ] I had read the documentation: ChunJun doc but it didn't help me.
Description
1、数据量9000万+时,mysql同步至hive时,一是不能设置多通道,因为hive sink组件不支持 二是、需要设置fetchSize 实现可读,我设置2048000,但是同步依然很慢,大概需要4个小时
2、j机器性能很好,mysql数据库服务器也很好,我想问如何提交同步效率
Code of Conduct
- [X] I agree to follow this project's Code of Conduct
@biandou1313 hdfs相关插件支持并发写入的。你这边的配置发下