doris-spark-connector icon indicating copy to clipboard operation
doris-spark-connector copied to clipboard

[feature] add write blocking properties to control write blocking.

Open CodeCooker17 opened this issue 2 years ago • 1 comments

Proposed changes

Issue Number: close #xxx

Problem Summary:

Describe the overview of changes. add three config to control write blocking. Prevent write failure due to cpmpaction not being timely to improve write success rate. --- max blocking times "doris.sink.max.blocking.times" = "8",
--- max blocking interval times per blocking "doris.sink.max.blocking.interval.ms" = "300000", --- need to blocking write keys "doris.sink.block.trigger.keys" = "-235,-238,-233"

Checklist(Required)

  1. Does it affect the original behavior: (Yes/No/I Don't know)
  2. Has unit tests been added: (Yes/No/No Need)
  3. Has document been added or modified: (Yes/No/No Need)
  4. Does it need to update dependencies: (Yes/No)
  5. Are there any changes that cannot be rolled back: (Yes/No)

Further comments

If this is a relatively large or complex change, kick off the discussion at [email protected] by explaining why you chose the solution you did and what alternatives you considered, etc...

CodeCooker17 avatar Jul 17 '23 03:07 CodeCooker17

After the optimization PR of https://github.com/apache/doris-spark-connector/pull/140, doris will stream the data when writing in http chuck mode, which to a certain extent avoids the compaction problems caused by large batch data load. And these exception codes are abnormal phenomena caused by doris failures, so they should be exposed and repaired on the doris side, and should not be blocked during writing.

gnehil avatar Sep 14 '23 10:09 gnehil