[feature] add write blocking properties to control write blocking.
Proposed changes
Issue Number: close #xxx
Problem Summary:
Describe the overview of changes.
add three config to control write blocking. Prevent write failure due to cpmpaction not being timely to improve write success rate.
--- max blocking times
"doris.sink.max.blocking.times" = "8",
--- max blocking interval times per blocking
"doris.sink.max.blocking.interval.ms" = "300000",
--- need to blocking write keys
"doris.sink.block.trigger.keys" = "-235,-238,-233"
Checklist(Required)
- Does it affect the original behavior: (Yes/No/I Don't know)
- Has unit tests been added: (Yes/No/No Need)
- Has document been added or modified: (Yes/No/No Need)
- Does it need to update dependencies: (Yes/No)
- Are there any changes that cannot be rolled back: (Yes/No)
Further comments
If this is a relatively large or complex change, kick off the discussion at [email protected] by explaining why you chose the solution you did and what alternatives you considered, etc...
After the optimization PR of https://github.com/apache/doris-spark-connector/pull/140, doris will stream the data when writing in http chuck mode, which to a certain extent avoids the compaction problems caused by large batch data load. And these exception codes are abnormal phenomena caused by doris failures, so they should be exposed and repaired on the doris side, and should not be blocked during writing.