seatunnel icon indicating copy to clipboard operation
seatunnel copied to clipboard
verified publisher icon apache

2.3.4同步数据到doris异常

Open imsprd opened this issue 1 year ago • 4 comments

Search before asking

  • [X] I had searched in the issues and found no similar issues.

What happened

读取数据同步至doris报错，CheckpointCoordinator inside have error. 从2.3.3升级到2.3.4，同样的脚本运行不了

SeaTunnel Version

2.3.4

SeaTunnel Config

env {
    job.name = "douyin_xingtu_star_works_record"
    job.mode = "BATCH"
}

source {
    Jdbc {
    url = "jdbc:mysql://ccccc:3392/data_center?zeroDateTimeBehavior=convertToNull"
    driver = "com.mysql.cj.jdbc.Driver"
    user = ""
    password = ""
    query = "select * from  douyin_xingtu_star_works_record_v3"
    result_table_name ="source_kdouyin_xingtu_star_works_record_v3"
    }
}
transform {
}
sink {
      Doris {
        source_table_name="source_kdouyin_xingtu_star_works_record_v3"
        fenodes = "xxx:8030,xxx:8030,xxxx:8030"
        username = ""
        password = ""
        database = "ims_wemedia"
        table = "douyin_video_xingtu"
        sink.enable-2pc = "true"
        sink.label-prefix = "label_douyin_video_xingtu_20240204"
        doris.config = {
          format="json"
          read_json_by_line="true"
        }
    }
}

Running Command

/bin/seatunnel.sh --config douyin_xingtu_video_to_doris.conf -e local

Error Exception

2024-02-04 17:38:39,953 ERROR [o.a.s.c.s.SeaTunnel           ] [main] - Exception StackTrace:org.apache.seatunnel.core.starter.exception.CommandExecuteException: SeaTunnel job executed failed
        at org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand.execute(ClientExecuteCommand.java:199)
        at org.apache.seatunnel.core.starter.SeaTunnel.run(SeaTunnel.java:40)
        at org.apache.seatunnel.core.starter.seatunnel.SeaTunnelClient.main(SeaTunnelClient.java:34)
Caused by: org.apache.seatunnel.engine.common.exception.SeaTunnelEngineException: org.apache.seatunnel.engine.server.checkpoint.CheckpointException: CheckpointCoordinator inside have error.
        at org.apache.seatunnel.engine.server.checkpoint.CheckpointCoordinator.handleCoordinatorError(CheckpointCoordinator.java:270)
        at org.apache.seatunnel.engine.server.checkpoint.CheckpointCoordinator.handleCoordinatorError(CheckpointCoordinator.java:266)
        at org.apache.seatunnel.engine.server.checkpoint.CheckpointCoordinator.reportCheckpointErrorFromTask(CheckpointCoordinator.java:372)
        at org.apache.seatunnel.engine.server.checkpoint.CheckpointManager.reportCheckpointErrorFromTask(CheckpointManager.java:183)
        at org.apache.seatunnel.engine.server.checkpoint.operation.CheckpointErrorReportOperation.run(CheckpointErrorReportOperation.java:48)
        at com.hazelcast.spi.impl.operationservice.Operation.call(Operation.java:189)
        at com.hazelcast.spi.impl.operationservice.impl.OperationRunnerImpl.call(OperationRunnerImpl.java:273)
        at com.hazelcast.spi.impl.operationservice.impl.OperationRunnerImpl.run(OperationRunnerImpl.java:248)
        at com.hazelcast.spi.impl.operationservice.impl.OperationRunnerImpl.run(OperationRunnerImpl.java:213)
        at com.hazelcast.spi.impl.operationexecutor.impl.OperationThread.process(OperationThread.java:175)
        at com.hazelcast.spi.impl.operationexecutor.impl.OperationThread.process(OperationThread.java:139)
        at com.hazelcast.spi.impl.operationexecutor.impl.OperationThread.executeRun(OperationThread.java:123)
        at com.hazelcast.internal.util.executor.HazelcastManagedThread.run(HazelcastManagedThread.java:102)
Caused by: org.apache.seatunnel.common.utils.SeaTunnelException: org.apache.seatunnel.connectors.doris.exception.DorisConnectorException: ErrorCode:[Doris-01], ErrorDescription:[stream load error]
        at org.apache.seatunnel.connectors.doris.sink.writer.DorisSinkWriter.initializeLoad(DorisSinkWriter.java:115)
        at org.apache.seatunnel.connectors.doris.sink.writer.DorisSinkWriter.<init>(DorisSinkWriter.java:98)
        at org.apache.seatunnel.connectors.doris.sink.DorisSink.createWriter(DorisSink.java:83)
        at org.apache.seatunnel.connectors.seatunnel.common.multitablesink.MultiTableSink.createWriter(MultiTableSink.java:70)
        at org.apache.seatunnel.engine.server.task.flow.SinkFlowLifeCycle.restoreState(SinkFlowLifeCycle.java:297)
        at org.apache.seatunnel.engine.server.task.SeaTunnelTask.lambda$restoreState$16(SeaTunnelTask.java:425)
        at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:184)
        at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
        at java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:175)
        at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1382)
        at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481)
        at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471)
        at java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:151)
        at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:174)
        at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
        at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:418)
        at org.apache.seatunnel.engine.server.task.SeaTunnelTask.restoreState(SeaTunnelTask.java:422)
        at org.apache.seatunnel.engine.server.checkpoint.operation.NotifyTaskRestoreOperation.lambda$null$0(NotifyTaskRestoreOperation.java:106)
        at java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1626)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.http.client.ClientProtocolException: URI does not specify a valid host name: http://172.18.222.76:8030,172.18.222.74:8030/api/ims_wemedia/douyin_video_xingtu/_stream_load
        at org.apache.http.impl.client.CloseableHttpClient.determineTarget(CloseableHttpClient.java:95)
        at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
        at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:108)
        at org.apache.seatunnel.connectors.doris.sink.writer.DorisStreamLoad.abortPreCommit(DorisStreamLoad.java:144)
        at org.apache.seatunnel.connectors.doris.sink.writer.DorisSinkWriter.initializeLoad(DorisSinkWriter.java:112)
        ... 21 more

        ... 11 more

        at org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand.execute(ClientExecuteCommand.java:191)
        ... 2 more

2024-02-04 17:38:39,954 ERROR [o.a.s.c.s.SeaTunnel           ] [main] -
===============================================================================



Exception in thread "main" org.apache.seatunnel.core.starter.exception.CommandExecuteException: SeaTunnel job executed failed
        at org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand.execute(ClientExecuteCommand.java:199)
        at org.apache.seatunnel.core.starter.SeaTunnel.run(SeaTunnel.java:40)
        at org.apache.seatunnel.core.starter.seatunnel.SeaTunnelClient.main(SeaTunnelClient.java:34)
Caused by: org.apache.seatunnel.engine.common.exception.SeaTunnelEngineException: org.apache.seatunnel.engine.server.checkpoint.CheckpointException: CheckpointCoordinator inside have error.
        at org.apache.seatunnel.engine.server.checkpoint.CheckpointCoordinator.handleCoordinatorError(CheckpointCoordinator.java:270)
        at org.apache.seatunnel.engine.server.checkpoint.CheckpointCoordinator.handleCoordinatorError(CheckpointCoordinator.java:266)
        at org.apache.seatunnel.engine.server.checkpoint.CheckpointCoordinator.reportCheckpointErrorFromTask(CheckpointCoordinator.java:372)
        at org.apache.seatunnel.engine.server.checkpoint.CheckpointManager.reportCheckpointErrorFromTask(CheckpointManager.java:183)
        at org.apache.seatunnel.engine.server.checkpoint.operation.CheckpointErrorReportOperation.run(CheckpointErrorReportOperation.java:48)
        at com.hazelcast.spi.impl.operationservice.Operation.call(Operation.java:189)
        at com.hazelcast.spi.impl.operationservice.impl.OperationRunnerImpl.call(OperationRunnerImpl.java:273)
        at com.hazelcast.spi.impl.operationservice.impl.OperationRunnerImpl.run(OperationRunnerImpl.java:248)
        at com.hazelcast.spi.impl.operationservice.impl.OperationRunnerImpl.run(OperationRunnerImpl.java:213)
        at com.hazelcast.spi.impl.operationexecutor.impl.OperationThread.process(OperationThread.java:175)
        at com.hazelcast.spi.impl.operationexecutor.impl.OperationThread.process(OperationThread.java:139)
        at com.hazelcast.spi.impl.operationexecutor.impl.OperationThread.executeRun(OperationThread.java:123)
        at com.hazelcast.internal.util.executor.HazelcastManagedThread.run(HazelcastManagedThread.java:102)
Caused by: org.apache.seatunnel.common.utils.SeaTunnelException: org.apache.seatunnel.connectors.doris.exception.DorisConnectorException: ErrorCode:[Doris-01], ErrorDescription:[stream load error]
        at org.apache.seatunnel.connectors.doris.sink.writer.DorisSinkWriter.initializeLoad(DorisSinkWriter.java:115)
        at org.apache.seatunnel.connectors.doris.sink.writer.DorisSinkWriter.<init>(DorisSinkWriter.java:98)
        at org.apache.seatunnel.connectors.doris.sink.DorisSink.createWriter(DorisSink.java:83)
        at org.apache.seatunnel.connectors.seatunnel.common.multitablesink.MultiTableSink.createWriter(MultiTableSink.java:70)
        at org.apache.seatunnel.engine.server.task.flow.SinkFlowLifeCycle.restoreState(SinkFlowLifeCycle.java:297)
        at org.apache.seatunnel.engine.server.task.SeaTunnelTask.lambda$restoreState$16(SeaTunnelTask.java:425)
        at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:184)
        at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
        at java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:175)
        at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1382)
        at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481)
        at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471)
        at java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:151)
        at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:174)
        at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
        at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:418)
        at org.apache.seatunnel.engine.server.task.SeaTunnelTask.restoreState(SeaTunnelTask.java:422)
        at org.apache.seatunnel.engine.server.checkpoint.operation.NotifyTaskRestoreOperation.lambda$null$0(NotifyTaskRestoreOperation.java:106)
        at java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1626)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.http.client.ClientProtocolException: URI does not specify a valid host name: http://172.18.222.76:8030,172.18.222.74:8030/api/ims_wemedia/douyin_video_xingtu/_stream_load
        at org.apache.http.impl.client.CloseableHttpClient.determineTarget(CloseableHttpClient.java:95)
        at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
        at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:108)
        at org.apache.seatunnel.connectors.doris.sink.writer.DorisStreamLoad.abortPreCommit(DorisStreamLoad.java:144)
        at org.apache.seatunnel.connectors.doris.sink.writer.DorisSinkWriter.initializeLoad(DorisSinkWriter.java:112)
        ... 21 more

        ... 11 more

        at org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand.execute(ClientExecuteCommand.java:191)
        ... 2 more
2024-02-04 17:38:39,958 INFO  [s.c.s.s.c.ClientExecuteCommand] [ForkJoinPool.commonPool-worker-57] - run shutdown hook because get close signal

Zeta or Flink or Spark Version

Zeta

Java or Scala Version

1.8.0_181

Screenshots

No response

Are you willing to submit PR?

  • [ ] Yes I am willing to submit a PR!

Code of Conduct

imsprd avatar Feb 04 '24 09:02 imsprd

问题以解决，fenodes不支持多ip加端口的写法了，只能写一个 fenodes = "xxx.xxx.xxx.xx:8030,xxx.xxx.xxx.xxx:8030" 更换成 fenodes = "xxx.xxx.xxx.xx:8030"

imsprd avatar Feb 04 '24 09:02 imsprd

image 我看2.3.4下面的代码没有了，只保留了上面的代码

imsprd avatar Feb 04 '24 09:02 imsprd

image 我看2.3.4下面的代码没有了，只保留了上面的代码

image @imsprd hi, This is indeed a problem, but the place where it happens should be the code above

Carl-Zhou-CN avatar Feb 06 '24 02:02 Carl-Zhou-CN

请问下2.3.4哪里下载？感谢

bulolo avatar Feb 06 '24 13:02 bulolo

This issue has been automatically marked as stale because it has not had recent activity for 30 days. It will be closed in next 7 days if no further activity occurs.

github-actions[bot] avatar Mar 08 '24 00:03 github-actions[bot]

This issue has been closed because it has not received response for too long time. You could reopen it if you encountered similar problems in the future.

github-actions[bot] avatar Mar 16 '24 00:03 github-actions[bot]