doris icon indicating copy to clipboard operation
doris copied to clipboard

[fix](routine load) fix ROUTINE LOAD bug,kafka commit a lack of one(#17282)

Open yuxuan-luo opened this issue 1 year ago • 7 comments

Proposed changes

Issue Number: close #17282

Problem summary

base on f2a96fd commit test https://github.com/apache/doris/commit/f2a96fda62117412141e13e8df4a1f7589163956

Checklist(Required)

  • [ X] Does it affect the original behavior
  • [ X] Has unit tests been added
  • [ X] Has document been added or modified
  • [ X] Does it need to update dependencies
  • [ √] Is this PR support rollback (If NO, please explain WHY)

Further comments

If this is a relatively large or complex change, kick off the discussion at [email protected] by explaining why you chose the solution you did and what alternatives you considered, etc...

yuxuan-luo avatar Mar 01 '23 08:03 yuxuan-luo

clang-tidy review says "All clean, LGTM! :+1:"

github-actions[bot] avatar Mar 01 '23 08:03 github-actions[bot]

./run buildall

yiguolei avatar Mar 02 '23 04:03 yiguolei

TeamCity pipeline, clickbench performance test result: the sum of best hot time: 36.03 seconds stream load tsv: 470 seconds loaded 74807831229 Bytes, about 151 MB/s stream load json: 40 seconds loaded 2358488459 Bytes, about 56 MB/s stream load orc: 75 seconds loaded 1101869774 Bytes, about 14 MB/s stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/tmp/20230311144654_clickbench_pr_112805.html

hello-stephen avatar Mar 02 '23 06:03 hello-stephen

clang-tidy review says "All clean, LGTM! :+1:"

github-actions[bot] avatar Mar 03 '23 02:03 github-actions[bot]

Is this only a display problem or it will lost data in the past?

yiguolei avatar Mar 03 '23 03:03 yiguolei

这只是一个显示问题还是会丢失过去的数据? only a display problem

yuxuan-luo avatar Mar 03 '23 03:03 yuxuan-luo

once change the commit offset, you should also change consume logic for commit offset, or you may cause data lost. what's more, it would be not compatible with old verison doris, which means it would cause one data lost when you first upgrade doris cluster.

caiconghui avatar Mar 03 '23 14:03 caiconghui

If this is only a display problem, I think we'd better not modify it.

yiguolei avatar Mar 07 '23 04:03 yiguolei

clang-tidy review says "All clean, LGTM! :+1:"

github-actions[bot] avatar Mar 10 '23 03:03 github-actions[bot]

If this is only a display problem, I think we'd better not modify it.

This is the right way to commit the kafka offset to kafka server, So it LGTM

morningman avatar Mar 10 '23 04:03 morningman

run buildall

morningman avatar Mar 10 '23 04:03 morningman

PR approved by at least one committer and no changes requested.

github-actions[bot] avatar Mar 10 '23 04:03 github-actions[bot]

PR approved by anyone and no changes requested.

github-actions[bot] avatar Mar 10 '23 04:03 github-actions[bot]

clang-tidy review says "All clean, LGTM! :+1:"

github-actions[bot] avatar Mar 11 '23 07:03 github-actions[bot]

run P0 Regression

yuxuan-luo avatar Mar 11 '23 08:03 yuxuan-luo

run buildall

morningman avatar Mar 11 '23 14:03 morningman