Addax
Addax copied to clipboard
wgzhao
[Bug]:kudu同步kudu失败
What happened?
2024-08-08 13:57:45.942 [ job-0] INFO StandAloneJobContainerCommunicator - Total 0 records, 0 bytes | Speed 0B/s, 0 records/s | Error 0 records, 0 bytes | All Task WaitWriterTime 0.000s | All Task WaitReaderTime 0.000s | Percentage 0.00% 2024-08-08 13:57:45.944 [ job-0] ERROR Engine - com.wgzhao.addax.common.exception.AddaxException: java.lang.ClassCastException: com.alibaba.fastjson2.JSONObject cannot be cast to java.lang.String at com.wgzhao.addax.plugin.writer.kuduwriter.KuduWriter$Job.validateParameter(KuduWriter.java:86) at com.wgzhao.addax.plugin.writer.kuduwriter.KuduWriter$Job.init(KuduWriter.java:51) at com.wgzhao.addax.core.job.JobContainer.initJobWriter(JobContainer.java:610) at com.wgzhao.addax.core.job.JobContainer.init(JobContainer.java:266) at com.wgzhao.addax.core.job.JobContainer.start(JobContainer.java:122) at com.wgzhao.addax.core.Engine.start(Engine.java:62) at com.wgzhao.addax.core.Engine.entry(Engine.java:113) at com.wgzhao.addax.core.Engine.main(Engine.java:139)
at com.wgzhao.addax.common.exception.AddaxException.asAddaxException(AddaxException.java:66)
at com.wgzhao.addax.core.job.JobContainer.start(JobContainer.java:165)
at com.wgzhao.addax.core.Engine.start(Engine.java:62)
at com.wgzhao.addax.core.Engine.entry(Engine.java:113)
at com.wgzhao.addax.core.Engine.main(Engine.java:139)
Caused by: java.lang.ClassCastException: com.alibaba.fastjson2.JSONObject cannot be cast to java.lang.String at com.wgzhao.addax.plugin.writer.kuduwriter.KuduWriter$Job.validateParameter(KuduWriter.java:86) at com.wgzhao.addax.plugin.writer.kuduwriter.KuduWriter$Job.init(KuduWriter.java:51) at com.wgzhao.addax.core.job.JobContainer.initJobWriter(JobContainer.java:610) at com.wgzhao.addax.core.job.JobContainer.init(JobContainer.java:266) at com.wgzhao.addax.core.job.JobContainer.start(JobContainer.java:122) ... 3 more
Version
4.1.3 (Default)
OS Type
No response
Java JDK Version
Oracle JDK 1.8.0
Relevant log output
{
"job": {
"setting": {
"speed": {
"channel": 3
}
},
"content": [
{
"reader": {
"name": "kudureader",
"parameter": {
"masterAddress": "idc-cdh-119-73:7051,idc-cdh-119-74:7051,idc-cdh-119-75:7051,idc-cdh-119-80:7051,idc-cdh-119-81:7051",
"table": "impala::kududb.cl_vr_dispatch_order",
"column": ["id", "create_time", "modifier"],
"splitPk": "id",
"where": [
"create_time >= 2024-06-01 00:00:00",
"create_time <= 2024-06-02 00:00:00"
],
"lowerBound": 1,
"upperBound": 2236273,
"readTimeout": 30,
"scanTimeout": 60
}
},
"writer": {
"name": "kuduwriter",
"parameter": {
"masterAddress": "cdhtest01:7051,cdhtest02:7051,cdhtest04:7051",
"timeout": 60,
"sessionTimeout": 60,
"table": "impala::kududb.users",
"replicaCount": 3,
"truncate": false,
"writeMode": "upsert",
"column": [
{"name":"id","type":"int"},
{"name":"create_time", "type":"string"},
{"name":"modifier", "type":"string"}
],
"batchSize": 1024,
"bufferSize": 2048,
"skipFail": false,
"encoding": "UTF-8"
}
}
}
]
}
}
{ "job": { "setting": { "speed": { "channel": 1, "bytes": -1 } }, "content": [ { "reader": { "name": "streamreader", "parameter": { "column": [ { "random": "1,1000", "type": "long" }, { "random": "1,10", "type": "string" }, { "random": "1000,50000", "type": "double" } ], "sliceRecordCount": 1000 } }, "writer": { "name": "kuduwriter", "parameter": { "masterAddress": "10.60.172.153:7051,10.60.172.153:7151,10.60.172.153:7251", "timeout": 60, "sessionTimeout": 60, "table": "users", "replicaCount": 3, "truncate": false, "writeMode": "upsert", "column": [ {"name":"user_id","type":"int"}, {"name":"user_name", "type":"long"}, {"name":"salary", "type":"double"} ], "batchSize": 1024, "bufferSize": 2048, "skipFail": false, "encoding": "UTF-8" } } } ] } } 官网给的案例也不行
___ _ _
/ _ \ | | | |
/ /_\ \ __| | __| | __ ___ __
| _ |/ _` |/ _` |/ _` \ \/ /
| | | | (_| | (_| | (_| |> <
\_| |_/\__,_|\__,_|\__,_/_/\_\
:: Addax version :: (v4.1.6-SNAPSHOT)
2024-08-08 15:46:37.646 [ main] INFO VMInfo - VMInfo# operatingSystem class => sun.management.OperatingSystemImpl
2024-08-08 15:46:37.663 [ main] INFO Engine -
{
"setting":{
"speed":{
"channel":3
}
},
"content":{
"reader":{
"name":"kudureader",
"parameter":{
"masterAddress":"idc-cdh-119-73:7051,idc-cdh-119-74:7051,idc-cdh-119-75:7051,idc-cdh-119-80:7051,idc-cdh-119-81:7051",
"table":"impala::kududb.cl_vr_dispatch_order",
"column":[
"id",
"create_time",
"modifier"
],
"splitPk":"id",
"where":[
"create_time >= 2024-06-01 00:00:00",
"create_time <= 2024-06-02 00:00:00"
],
"lowerBound":1,
"upperBound":2236273,
"readTimeout":30,
"scanTimeout":60
}
},
"writer":{
"name":"kuduwriter",
"parameter":{
"masterAddress":"cdhtest01:7051,cdhtest02:7051,cdhtest04:7051",
"timeout":60,
"sessionTimeout":60,
"table":"impala::kududb.users",
"replicaCount":3,
"truncate":false,
"writeMode":"upsert",
"column":[
{
"name":"id",
"type":"int8"
},
{
"name":"create_time",
"type":"string"
},
{
"name":"modifier",
"type":"string"
}
],
"batchSize":1024,
"bufferSize":2048,
"skipFail":false,
"encoding":"UTF-8"
}
}
}
}
2024-08-08 15:46:37.686 [ main] INFO JobContainer - The jobContainer begins to process the job.
2024-08-08 15:46:37.713 [ job-0] INFO KuduWriter$Job - Try to connect kudu with cdhtest01:7051,cdhtest02:7051,cdhtest04:7051
2024-08-08 15:46:38.736 [ job-0] INFO StandAloneJobContainerCommunicator - Total 0 records, 0 bytes | Speed 0B/s, 0 records/s | Error 0 records, 0 bytes | All Task WaitWriterTime 0.000s | All Task WaitReaderTime 0.000s | Percentage 0.00%
2024-08-08 15:46:38.738 [ job-0] ERROR Engine - com.wgzhao.addax.common.exception.AddaxException: java.lang.ClassCastException: com.alibaba.fastjson2.JSONObject cannot be cast to java.lang.String
at com.wgzhao.addax.plugin.writer.kuduwriter.KuduWriter$Job.validateParameter(KuduWriter.java:86)
at com.wgzhao.addax.plugin.writer.kuduwriter.KuduWriter$Job.init(KuduWriter.java:51)
at com.wgzhao.addax.core.job.JobContainer.initJobWriter(JobContainer.java:610)
at com.wgzhao.addax.core.job.JobContainer.init(JobContainer.java:266)
at com.wgzhao.addax.core.job.JobContainer.start(JobContainer.java:122)
at com.wgzhao.addax.core.Engine.start(Engine.java:62)
at com.wgzhao.addax.core.Engine.entry(Engine.java:113)
at com.wgzhao.addax.core.Engine.main(Engine.java:139)
at com.wgzhao.addax.common.exception.AddaxException.asAddaxException(AddaxException.java:66)
at com.wgzhao.addax.core.job.JobContainer.start(JobContainer.java:165)
at com.wgzhao.addax.core.Engine.start(Engine.java:62)
at com.wgzhao.addax.core.Engine.entry(Engine.java:113)
at com.wgzhao.addax.core.Engine.main(Engine.java:139)
Caused by: java.lang.ClassCastException: com.alibaba.fastjson2.JSONObject cannot be cast to java.lang.String
at com.wgzhao.addax.plugin.writer.kuduwriter.KuduWriter$Job.validateParameter(KuduWriter.java:86)
at com.wgzhao.addax.plugin.writer.kuduwriter.KuduWriter$Job.init(KuduWriter.java:51)
at com.wgzhao.addax.core.job.JobContainer.initJobWriter(JobContainer.java:610)
at com.wgzhao.addax.core.job.JobContainer.init(JobContainer.java:266)
at com.wgzhao.addax.core.job.JobContainer.start(JobContainer.java:122)
... 3 more
/ _ \ | | | |
/ /_\ \ __| | __| | __ ___ __
| _ |/ _` |/ _` |/ _` \ \/ /
| | | | (_| | (_| | (_| |> <
\_| |_/\__,_|\__,_|\__,_/_/\_\
:: Addax version :: (v4.0.2)
2024-08-08 15:50:44.928 [ main] INFO VMInfo - VMInfo# operatingSystem class => sun.management.OperatingSystemImpl
2024-08-08 15:50:44.952 [ main] INFO Engine -
{
"content":[
{
"reader":{
"parameter":{
"masterAddress":"idc-cdh-119-73:7051,idc-cdh-119-74:7051,idc-cdh-119-75:7051,idc-cdh-119-80:7051,idc-cdh-119-81:7051",
"upperBound":2236273,
"readTimeout":30,
"column":[
"id",
"mark",
"mark_nm",
"yyyymmdd",
"tab_no",
"data_time",
"start_time",
"end_time",
"memo",
"create_date",
"str1",
"str2",
"str3",
"str4"
],
"lowerBound":1,
"splitPk":"id",
"table":"impala::kududb.bi001t0",
"scanTimeout":60
},
"name":"kudureader"
},
"writer":{
"parameter":{
"masterAddress":"cdhtest01:7051,cdhtest02:7051,cdhtest04:7051",
"replicaCount":3,
"truncate":false,
"skipFail":false,
"column":[
{
"name":"id",
"type":"int64"
},
{
"name":"mark",
"type":"string"
},
{
"name":"mark_nm",
"type":"string"
},
{
"name":"yyyymmdd",
"type":"string"
},
{
"name":"tab_no",
"type":"string"
},
{
"name":"data_time",
"type":"string"
},
{
"name":"start_time",
"type":"string"
},
{
"name":"end_time",
"type":"string"
},
{
"name":"memo",
"type":"string"
},
{
"name":"create_date",
"type":"int64"
},
{
"name":"str1",
"type":"string"
},
{
"name":"str2",
"type":"string"
},
{
"name":"str3",
"type":"string"
},
{
"name":"str4",
"type":"string"
}
],
"sessionTimeout":60,
"writeMode":"upsert",
"batchSize":1024,
"encoding":"UTF-8",
"timeout":60,
"table":"impala::kududb.bi001t0",
"bufferSize":2048
},
"name":"kuduwriter"
}
}
],
"setting":{
"speed":{
"channel":3
}
}
}
2024-08-08 15:50:44.972 [ main] INFO PerfTrace - PerfTrace traceId=job_-1, isEnable=false, priority=0
2024-08-08 15:50:44.973 [ main] INFO JobContainer - Addax jobContainer starts job.
2024-08-08 15:50:44.974 [ main] INFO JobContainer - Set jobId = 0
2024-08-08 15:50:44.986 [ job-0] INFO JobContainer - Addax Reader.Job [kudureader] do prepare work .
2024-08-08 15:50:44.986 [ job-0] INFO JobContainer - Addax Writer.Job [kuduwriter] do prepare work .
2024-08-08 15:50:45.998 [ job-0] INFO JobContainer - Job set Channel-Number to 3 channels.
2024-08-08 15:50:45.999 [ job-0] INFO JobContainer - Addax Reader.Job [kudureader] splits to [3] tasks.
2024-08-08 15:50:46.000 [ job-0] INFO JobContainer - Addax Writer.Job [kuduwriter] splits to [3] tasks.
2024-08-08 15:50:46.025 [ job-0] INFO JobContainer - Scheduler starts [1] taskGroups.
2024-08-08 15:50:46.032 [ taskGroup-0] INFO TaskGroupContainer - taskGroupId=[0] start [3] channels for [3] tasks.
2024-08-08 15:50:46.036 [ taskGroup-0] INFO Channel - Channel set byte_speed_limit to -1, No bps activated.
2024-08-08 15:50:46.036 [ taskGroup-0] INFO Channel - Channel set record_speed_limit to -1, No tps activated.
2024-08-08 15:50:46.136 [0-0-0-writer] INFO KuduWriterTask - kuduwriter begin to write!
2024-08-08 15:50:46.145 [0-0-1-writer] INFO KuduWriterTask - kuduwriter begin to write!
2024-08-08 15:50:46.227 [0-0-2-writer] INFO KuduWriterTask - kuduwriter begin to write!
2024-08-08 15:50:46.236 [0-0-1-reader] INFO KuduClient - defaultSocketReadTimeoutMs is deprecated
2024-08-08 15:50:46.236 [0-0-2-reader] INFO KuduClient - defaultSocketReadTimeoutMs is deprecated
2024-08-08 15:50:46.244 [0-0-0-reader] INFO KuduClient - defaultSocketReadTimeoutMs is deprecated
2024-08-08 15:50:47.492 [0-0-0-writer] ERROR WriterRunner - Writer Runner Received Exceptions:
java.lang.IllegalArgumentException: id isn't [Type: int32], it's int64
at org.apache.kudu.client.PartialRow.checkColumn(PartialRow.java:1228)
at org.apache.kudu.client.PartialRow.addInt(PartialRow.java:280)
at org.apache.kudu.client.PartialRow.addInt(PartialRow.java:293)
at com.wgzhao.addax.plugin.writer.kuduwriter.KuduWriterTask.startWriter(KuduWriterTask.java:125)
at com.wgzhao.addax.plugin.writer.kuduwriter.KuduWriter$Task.startWrite(KuduWriter.java:125)
at com.wgzhao.addax.core.taskgroup.runner.WriterRunner.run(WriterRunner.java:81)
at java.lang.Thread.run(Thread.java:745)
Exception in thread "taskGroup-0" com.wgzhao.addax.common.exception.AddaxException: Code:[Framework-13], Description:[Addax 插件运行时出错, 具体原因请参看Addax 运行结束时的错误诊断信息 .]. - java.lang.IllegalArgumentException: id isn't [Type: int32], it's int64
at org.apache.kudu.client.PartialRow.checkColumn(PartialRow.java:1228)
at org.apache.kudu.client.PartialRow.addInt(PartialRow.java:280)
at org.apache.kudu.client.PartialRow.addInt(PartialRow.java:293)
at com.wgzhao.addax.plugin.writer.kuduwriter.KuduWriterTask.startWriter(KuduWriterTask.java:125)
at com.wgzhao.addax.plugin.writer.kuduwriter.KuduWriter$Task.startWrite(KuduWriter.java:125)
at com.wgzhao.addax.core.taskgroup.runner.WriterRunner.run(WriterRunner.java:81)
at java.lang.Thread.run(Thread.java:745)
- java.lang.IllegalArgumentException: id isn't [Type: int32], it's int64
at org.apache.kudu.client.PartialRow.checkColumn(PartialRow.java:1228)
at org.apache.kudu.client.PartialRow.addInt(PartialRow.java:280)
at org.apache.kudu.client.PartialRow.addInt(PartialRow.java:293)
at com.wgzhao.addax.plugin.writer.kuduwriter.KuduWriterTask.startWriter(KuduWriterTask.java:125)
at com.wgzhao.addax.plugin.writer.kuduwriter.KuduWriter$Task.startWrite(KuduWriter.java:125)
at com.wgzhao.addax.core.taskgroup.runner.WriterRunner.run(WriterRunner.java:81)
at java.lang.Thread.run(Thread.java:745)
at com.wgzhao.addax.common.exception.AddaxException.asAddaxException(AddaxException.java:66)
at com.wgzhao.addax.core.taskgroup.TaskGroupContainer.start(TaskGroupContainer.java:218)
at com.wgzhao.addax.core.taskgroup.runner.TaskGroupContainerRunner.run(TaskGroupContainerRunner.java:47)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.IllegalArgumentException: id isn't [Type: int32], it's int64
at org.apache.kudu.client.PartialRow.checkColumn(PartialRow.java:1228)
at org.apache.kudu.client.PartialRow.addInt(PartialRow.java:280)
at org.apache.kudu.client.PartialRow.addInt(PartialRow.java:293)
at com.wgzhao.addax.plugin.writer.kuduwriter.KuduWriterTask.startWriter(KuduWriterTask.java:125)
at com.wgzhao.addax.plugin.writer.kuduwriter.KuduWriter$Task.startWrite(KuduWriter.java:125)
at com.wgzhao.addax.core.taskgroup.runner.WriterRunner.run(WriterRunner.java:81)
... 1 more
2024-08-08 15:50:49.042 [ job-0] ERROR JobContainer - 运行scheduler出错.
2024-08-08 15:50:49.044 [ job-0] INFO StandAloneJobContainerCommunicator - Total 396 records, 39983 bytes | Speed 39.05KB/s, 396 records/s | Error 0 records, 0 bytes | All Task WaitWriterTime 0.000s | All Task WaitReaderTime 1.350s | Percentage 33.33%
2024-08-08 15:50:49.044 [ job-0] ERROR Engine - Code:[Framework-13], Description:[Addax 插件运行时出错, 具体原因请参看Addax 运行结束时的错误诊断信息 .]. - Code:[Framework-13], Description:[Addax 插件运行时出错, 具体原因请参看Addax 运行结束时的错误诊断信息 .]. - java.lang.IllegalArgumentException: id isn't [Type: int32], it's int64
at org.apache.kudu.client.PartialRow.checkColumn(PartialRow.java:1228)
at org.apache.kudu.client.PartialRow.addInt(PartialRow.java:280)
at org.apache.kudu.client.PartialRow.addInt(PartialRow.java:293)
at com.wgzhao.addax.plugin.writer.kuduwriter.KuduWriterTask.startWriter(KuduWriterTask.java:125)
at com.wgzhao.addax.plugin.writer.kuduwriter.KuduWriter$Task.startWrite(KuduWriter.java:125)
at com.wgzhao.addax.core.taskgroup.runner.WriterRunner.run(WriterRunner.java:81)
at java.lang.Thread.run(Thread.java:745)
- java.lang.IllegalArgumentException: id isn't [Type: int32], it's int64
at org.apache.kudu.client.PartialRow.checkColumn(PartialRow.java:1228)
at org.apache.kudu.client.PartialRow.addInt(PartialRow.java:280)
at org.apache.kudu.client.PartialRow.addInt(PartialRow.java:293)
at com.wgzhao.addax.plugin.writer.kuduwriter.KuduWriterTask.startWriter(KuduWriterTask.java:125)
at com.wgzhao.addax.plugin.writer.kuduwriter.KuduWriter$Task.startWrite(KuduWriter.java:125)
at com.wgzhao.addax.core.taskgroup.runner.WriterRunner.run(WriterRunner.java:81)
at java.lang.Thread.run(Thread.java:745)
这个是用低版本
- 当前kudu 版本是多少？
4.1.6-SNAPSHOT中的报错 在KuduWriter.java:86这行代码不涉及到 JSON 转换，是否自己修改过插件代码？
你的配置有误，kuduwriter 的 column 应该配置成字段列表，类似如下：
{
"column": ["id", "create_time", "modifier"]
}
不需要指定类型。
参考官方文档时要注意文档版本与软件版本保持一致，因为软件在更新过程中，文件配置项有可能发生改变。我看你参考的是 4.0.4 及更老的版本文档，而运行的软件却是最新的。