seatunnel [Bug] [Doris Sink] Sink Doris Error

[Bug] [Doris Sink] Sink Doris Error

Open taohaozhi1129 opened this issue 10 months ago • 0 comments

Search before asking

[X] I had searched in the issues and found no similar issues.

What happened

When using Sink Doris to write data into Doris, in the current state evaluation, if either of the conditions is not met, the write failure is not logged. However, the write operation ultimately fails. Therefore, the judgment should be modified to consider any status other than "success" as a failure.

SeaTunnel Version

SeaTunnel Version 2.3.4

SeaTunnel Config

sink {
    Doris {
      fenodes = "localhost:8030"
      username = "root"
      password = "123#7*"
      database = "ods_itd_sit"
      table = "ods_oa_ecology_hrmdepartment_df"
      sink.enable-2pc = "true"
      sink.label-prefix = "st"
      data_save_mode="DROP_DATA"
      doris.config = {
      format = "json"
      read_json_by_line="true"
      }
  }
}

Running Command

seatunnel run job

Error Exception

2024-04-23 01:53:08,513 ERROR org.apache.seatunnel.core.starter.SeaTunnel - Reason:SeaTunnel job executed failed 
	
	2024-04-23 01:53:08,514 ERROR org.apache.seatunnel.core.starter.SeaTunnel - Exception StackTrace:org.apache.seatunnel.core.starter.exception.CommandExecuteException: SeaTunnel job executed failed
		at org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand.execute(ClientExecuteCommand.java:191)
		at org.apache.seatunnel.core.starter.SeaTunnel.run(SeaTunnel.java:40)
		at org.apache.seatunnel.core.starter.seatunnel.SeaTunnelClient.main(SeaTunnelClient.java:34)
	Caused by: org.apache.seatunnel.engine.common.exception.SeaTunnelEngineException: org.apache.seatunnel.engine.server.checkpoint.CheckpointException: Checkpoint notify complete failed
		at org.apache.seatunnel.engine.server.checkpoint.CheckpointCoordinator.handleCoordinatorError(CheckpointCoordinator.java:255)
		at org.apache.seatunnel.engine.server.checkpoint.CheckpointCoordinator.handleCoordinatorError(CheckpointCoordinator.java:251)
		at org.apache.seatunnel.engine.server.checkpoint.CheckpointCoordinator.notifyCompleted(CheckpointCoordinator.java:328)
		at org.apache.seatunnel.engine.server.checkpoint.CheckpointCoordinator.completePendingCheckpoint(CheckpointCoordinator.java:736)
		at org.apache.seatunnel.engine.server.checkpoint.CheckpointCoordinator.lambda$startTriggerPendingCheckpoint$7(CheckpointCoordinator.java:480)
		at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774)
		at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750)
		at java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:456)
		at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
		at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
		at java.lang.Thread.run(Thread.java:750)
	Caused by: java.util.concurrent.CompletionException: org.apache.seatunnel.engine.common.exception.SeaTunnelEngineException: org.apache.seatunnel.connectors.doris.exception.DorisConnectorException: ErrorCode:[Doris-02], ErrorDescription:[commit error] - {
	    "status": "ANALYSIS_ERROR",
	    "msg": "errCode = 2, detailMessage = transaction [1501371] is already visible, not pre-committed."
	}
		at org.apache.seatunnel.connectors.doris.sink.committer.DorisCommitter.commitTransaction(DorisCommitter.java:132)
		at org.apache.seatunnel.connectors.doris.sink.committer.DorisCommitter.commit(DorisCommitter.java:71)
		at org.apache.seatunnel.engine.server.task.flow.SinkFlowLifeCycle.notifyCheckpointComplete(SinkFlowLifeCycle.java:239)
		at org.apache.seatunnel.engine.server.task.SeaTunnelTask.lambda$notifyCheckpointComplete$7(SeaTunnelTask.java:379)
		at org.apache.seatunnel.engine.common.utils.ExceptionUtil.sneaky(ExceptionUtil.java:133)
		at org.apache.seatunnel.engine.server.task.SeaTunnelTask.lambda$notifyAllAction$12(SeaTunnelTask.java:399)
		at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:184)
		at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
		at java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:175)
		at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384)
		at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482)
		at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472)
		at java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:151)
		at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:174)
		at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
		at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:418)
		at org.apache.seatunnel.engine.server.task.SeaTunnelTask.notifyAllAction(SeaTunnelTask.java:399)
		at org.apache.seatunnel.engine.server.task.SeaTunnelTask.notifyCheckpointComplete(SeaTunnelTask.java:379)
		at org.apache.seatunnel.engine.server.checkpoint.operation.CheckpointFinishedOperation.lambda$run$0(CheckpointFinishedOperation.java:91)
		at org.apache.seatunnel.common.utils.RetryUtils.retryWithException(RetryUtils.java:48)
		at org.apache.seatunnel.engine.server.checkpoint.operation.CheckpointFinishedOperation.run(CheckpointFinishedOperation.java:81)
		at com.hazelcast.spi.impl.operationservice.Operation.call(Operation.java:189)
		at com.hazelcast.spi.impl.operationservice.impl.OperationRunnerImpl.call(OperationRunnerImpl.java:273)
		at com.hazelcast.spi.impl.operationservice.impl.OperationRunnerImpl.run(OperationRunnerImpl.java:248)
		at com.hazelcast.spi.impl.operationservice.impl.OperationRunnerImpl.run(OperationRunnerImpl.java:213)
		at com.hazelcast.spi.impl.operationexecutor.impl.OperationThread.process(OperationThread.java:175)
		at com.hazelcast.spi.impl.operationexecutor.impl.OperationThread.process(OperationThread.java:139)
		at com.hazelcast.spi.impl.operationexecutor.impl.OperationThread.executeRun(OperationThread.java:123)
		at com.hazelcast.internal.util.executor.HazelcastManagedThread.run(HazelcastManagedThread.java:102)
	
		at java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:292)
		at java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:308)
		at java.util.concurrent.CompletableFuture.biRelay(CompletableFuture.java:1300)
		at java.util.concurrent.CompletableFuture$BiRelay.tryFire(CompletableFuture.java:1284)
		at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
		at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1990)
		at com.hazelcast.spi.impl.AbstractInvocationFuture.onComplete(AbstractInvocationFuture.java:1243)
		at com.hazelcast.spi.impl.AbstractInvocationFuture.complete0(AbstractInvocationFuture.java:1234)
		at com.hazelcast.spi.impl.AbstractInvocationFuture.completeExceptionallyInternal(AbstractInvocationFuture.java:1223)
		at com.hazelcast.spi.impl.operationservice.impl.Invocation.completeExceptionally(Invocation.java:680)
		at com.hazelcast.spi.impl.operationservice.impl.Invocation.notifyThrowable(Invocation.java:386)
		at com.hazelcast.spi.impl.operationservice.impl.Invocation.notifyError(Invocation.java:330)
		at com.hazelcast.spi.impl.operationservice.impl.Invocation.sendResponse(Invocation.java:230)
		at com.hazelcast.spi.impl.operationservice.Operation.sendResponse(Operation.java:483)
		at com.hazelcast.spi.impl.operationservice.impl.OperationRunnerImpl.sendResponseAfterOperationError(OperationRunnerImpl.java:426)
		at com.hazelcast.spi.impl.operationservice.impl.OperationRunnerImpl.handleOperationError(OperationRunnerImpl.java:420)
		at com.hazelcast.spi.impl.operationservice.impl.OperationRunnerImpl.run(OperationRunnerImpl.java:253)
		at com.hazelcast.spi.impl.operationservice.impl.OperationRunnerImpl.run(OperationRunnerImpl.java:213)
		at com.hazelcast.spi.impl.operationexecutor.impl.OperationThread.process(OperationThread.java:175)
		at com.hazelcast.spi.impl.operationexecutor.impl.OperationThread.process(OperationThread.java:139)
		at com.hazelcast.spi.impl.operationexecutor.impl.OperationThread.executeRun(OperationThread.java:123)
		at com.hazelcast.internal.util.executor.HazelcastManagedThread.run(HazelcastManagedThread.java:102)
	Caused by: org.apache.seatunnel.engine.common.exception.SeaTunnelEngineException: org.apache.seatunnel.connectors.doris.exception.DorisConnectorException: ErrorCode:[Doris-02], ErrorDescription:[commit error] - {
	    "status": "ANALYSIS_ERROR",
	    "msg": "errCode = 2, detailMessage = transaction [1501371] is already visible, not pre-committed."
	}
		at org.apache.seatunnel.connectors.doris.sink.committer.DorisCommitter.commitTransaction(DorisCommitter.java:132)

Zeta or Flink or Spark Version

No response

Java or Scala Version

No response

Screenshots

No response

Are you willing to submit PR?

[X] Yes I am willing to submit a PR!

Code of Conduct

[X] I agree to follow this project's Code of Conduct

Apr 25 '24 06:04 taohaozhi1129

seatunnel seatunnel copied to clipboard

[Bug] [Doris Sink] Sink Doris Error

Search before asking

What happened

SeaTunnel Version

SeaTunnel Config

Running Command

Error Exception

Zeta or Flink or Spark Version

Java or Scala Version

Screenshots

Are you willing to submit PR?

Code of Conduct

seatunnel
seatunnel copied to clipboard