DataX
DataX copied to clipboard
alibaba
DataX是阿里云DataWorks数据集成的开源版本。
https://xie.infoq.cn/article/12ffb9991783e866394ea502f 目前处于测试阶段，有了解官方版的设计的，可以讨论一下
datax同步数据到hdfs时报错{com.mongodb.MongoCommandException: Command failed with error 18: 'Authentication failed.' on server 172.16.11.29:27017. The full response is { "operationTime" : { "$timestamp" : { "t" : 1673335462, "i" : 1 } },...
hdfs中的数据字段中包含双引号，DataX读取时会将双引号之间的数据当成一个字段进行处理，配置的列分隔符不起作用，导致数据错乱报错，无法将数据从原有大数据集群迁移至MaxCompute中
使用datax elasticsearchwriter往 es 7.9.2 版本写入数据，column配置没有生效 elasticsearch版本7.9.2 datax配置: "writer": { "name": "elasticsearchwriter", "parameter": { "endpoint": "http://XXX:9200", "accessId": "xxxx", "accessKey": "xxxx", "index": "XXXX", "type": "_doc", "cleanup": true, "settings": {"index": {"number_of_shards": 1, "number_of_replicas":...
经DataX智能分析,该任务最可能的错误原因是: com.alibaba.datax.common.exception.DataXException: Code:[ESWriter-03], Description:[mappings错误.]. - javax.net.ssl.SSLPeerUnverifiedException: Host name '10.10.20.64' does not match the certificate subject provided by the peer (CN=instance) at com.alibaba.datax.common.exception.DataXException.asDataXException(DataXException.java:26) at com.alibaba.datax.plugin.writer.elasticsearchwriter.ESWriter$Job.prepare(ESWriter.java:76) at com.alibaba.datax.core.job.JobContainer.prepareJobWriter(JobContainer.java:724) at com.alibaba.datax.core.job.JobContainer.prepare(JobContainer.java:309) at com.alibaba.datax.core.job.JobContainer.start(JobContainer.java:115)...
在项目实践中同时触发两个job任务 datax 打印出日志任务 2021-07-14 19:56:22.531 INFO 14772 --- [job-26] com.alibaba.datax.core.job.JobContainer : 任务启动时刻 : 2021-07-14 19:56:12 任务结束时刻 : 2021-07-14 19:56:22 任务总计耗时 : 10s 任务平均流量 : 2.49KB/s 记录写入速度 : 17rec/s 读出记录总数 :...
Line 532: 532 .listObjects(bucket).getObjectSummaries();
