doris-spark-connector icon indicating copy to clipboard operation
doris-spark-connector copied to clipboard

Add a parameter that controls the number of StreamLoad tasks committed per partition #92

Open baishaoisde opened this issue 1 year ago • 3 comments

Add a parameter that controls the number of StreamLoad tasks committed per partition

Issue Number: close #92

  1. Does it affect the original behavior: (I Don't know)
  2. Has unit tests been added: (No)
  3. Has document been added or modified: (Yes)
  4. Does it need to update dependencies: (Yes)
  5. Are there any changes that cannot be rolled back: (No)

baishaoisde avatar May 17 '23 12:05 baishaoisde

Thank you for your contribution, can you resolve the conflict?

JNSimba avatar May 26 '23 07:05 JNSimba

If the data is processed according to the partition, when a single partition is particularly large, there may be problems in a streamload?

JNSimba avatar May 26 '23 07:05 JNSimba

If the data is processed according to the partition, when a single partition is particularly large, there may be problems in a streamload?

Yes, for this problem, we need to prompt the user in the parameter description that "reparation needs to be used to ensure that the data volume per partition is in a reasonable range after this parameter is enabled". Do you think it is appropriate?

baishaoisde avatar May 29 '23 09:05 baishaoisde