doris
doris copied to clipboard
[fix](routine load) fix ROUTINE LOAD bug,kafka commit a lack of one(#17282)
Proposed changes
Issue Number: close #17282
Problem summary
base on f2a96fd commit test https://github.com/apache/doris/commit/f2a96fda62117412141e13e8df4a1f7589163956
Checklist(Required)
- [ X] Does it affect the original behavior
- [ X] Has unit tests been added
- [ X] Has document been added or modified
- [ X] Does it need to update dependencies
- [ √] Is this PR support rollback (If NO, please explain WHY)
Further comments
If this is a relatively large or complex change, kick off the discussion at [email protected] by explaining why you chose the solution you did and what alternatives you considered, etc...
clang-tidy review says "All clean, LGTM! :+1:"
./run buildall
TeamCity pipeline, clickbench performance test result: the sum of best hot time: 36.03 seconds stream load tsv: 470 seconds loaded 74807831229 Bytes, about 151 MB/s stream load json: 40 seconds loaded 2358488459 Bytes, about 56 MB/s stream load orc: 75 seconds loaded 1101869774 Bytes, about 14 MB/s stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/tmp/20230311144654_clickbench_pr_112805.html
clang-tidy review says "All clean, LGTM! :+1:"
Is this only a display problem or it will lost data in the past?
这只是一个显示问题还是会丢失过去的数据? only a display problem
once change the commit offset, you should also change consume logic for commit offset, or you may cause data lost. what's more, it would be not compatible with old verison doris, which means it would cause one data lost when you first upgrade doris cluster.
If this is only a display problem, I think we'd better not modify it.
clang-tidy review says "All clean, LGTM! :+1:"
If this is only a display problem, I think we'd better not modify it.
This is the right way to commit the kafka offset to kafka server, So it LGTM
run buildall
PR approved by at least one committer and no changes requested.
PR approved by anyone and no changes requested.
clang-tidy review says "All clean, LGTM! :+1:"
run P0 Regression
run buildall