chunjun

chunjun copied to clipboard

[Feature][chunjun-connector-hdfs] hdfs消费百亿数据时的速度不如spark

2

### Search before asking - [X] I had searched in the [issues](https://github.com/DTStack/chunjun/issues) and found no similar feature requirement. ### Description hdfs到hdfs的同步任务 ：数据源有300亿，300个文件，设定并行度为300，任务完成时间为3个小时左右，而spark运行相同的任务仅50多分钟。通过分析chunjun-core 与chunjun-connector-hdfs 代码 猜测是因为metric指标（包括脏数据）处理影响了消费速度，通过摘除chunjun-core中metric指标监控（不包含numWrite），hdfs的同步速度快了三倍。具体的，将去除指标chunjun-core与chunjun-connector-hdfs合并到一起创建了新的hdfs连接器。  ### Use case _No...

david-gao1

执行binglog2hive报错

2

### Search before asking - [ ] I had searched in the [issues](https://github.com/DTStack/chunjun/issues) and found no similar question. - [ ] I had googled my question but i didn't get...

fuliang0904

### Search before asking - [ ] I had searched in the [issues](https://github.com/DTStack/chunjun/issues) and found no similar question. - [ ] I had googled my question but i didn't get...

cs3163077

从ORACLE日志归档模式到HIVE2时，字段类型有问题

1

### Search before asking - [ ] I had searched in the [issues](https://github.com/DTStack/chunjun/issues) and found no similar question. - [ ] I had googled my question but i didn't get...

sunrfun

[Question][binlog-x] binlog-x为什么不能同步删除数据

1

### Search before asking - [X] I had searched in the [issues](https://github.com/DTStack/chunjun/issues) and found no similar question. - [X] I had googled my question but i didn't get any help....

617450941

如何使用chunjun-kubernetes-session.sh在kubernets上部署

1

### Search before asking - [X] I had searched in the [issues](https://github.com/DTStack/chunjun/issues) and found no similar question. - [X] I had googled my question but i didn't get any help....

YishuiLi

[Question]ftp采集数据输出到kafka报空指针

1

### Search before asking - [X] I had searched in the [issues](https://github.com/DTStack/chunjun/issues) and found no similar question. - [X] I had googled my question but i didn't get any help....

noob1214

[Bug] [sync] 两个带kerberos认证的CDH-6.3.2集群之间同步数据报错

1

### Search before asking - [X] I had searched in the [issues](https://github.com/DTStack/chunjun/issues) and found no similar issues. ### What happened  以上是我的报错。 ------------------------我是分割线--------------------- ```json { "job": { "content": [ {...

wangxiaoming-wxm

Can not use GreenplumWriter

1

source:binlogreader sink:greenplumreader can not init greenplum jdbc driver java.lang.ClassCastExeception: com.pivotal.jdbc.greenplumbase.ddu cannot be cast to org.postgresql.core.BaseConnection  json: 

zhangchao930216

"Hive connector doesn't support update/delete!"

2

### Search before asking - [X] I had searched in the [issues](https://github.com/DTStack/chunjun/issues) and found no similar feature requirement. ### Description hive目前不支持update和delete？后续会考虑吗？hudi等有机会支持吗？ ### Use case _No response_ ### Related issues _No...

KerYooz