doris
doris copied to clipboard
[Enhancement](load) Refine the load channel flush policy on mem limit
Proposed changes
Issue Number: close #xxx
Problem summary
Testing, will complete the description later.
Checklist(Required)
- Does it affect the original behavior:
- [ ] Yes
- [ ] No
- [ ] I don't know
- Has unit tests been added:
- [ ] Yes
- [ ] No
- [ ] No Need
- Has document been added or modified:
- [ ] Yes
- [ ] No
- [ ] No Need
- Does it need to update dependencies:
- [ ] Yes
- [ ] No
- Are there any changes that cannot be rolled back:
- [ ] Yes (If Yes, please explain WHY)
- [ ] No
Further comments
If this is a relatively large or complex change, kick off the discussion at [email protected] by explaining why you chose the solution you did and what alternatives you considered, etc...
PR approved by at least one committer and no changes requested.
PR approved by anyone and no changes requested.
PR approved by at least one committer and no changes requested.
PR approved by at least one committer and no changes requested.
LGTM
@zhannngchen Looking forward to updating the Problem summary
commit comment:
- Remove single load channel mem limit, only use load channel mgr mem limit
- Default load channel mgr mem limit from 50% to 80%
- load channel mgr add soft mem limit. When the soft limit is exceeded, other threads will not hang, only current thread triggers flush
- When exceed load channel mgr mem limit, find a load channel with the largest mem usage, continue to find a tablet channel with the largest mem usage, and try to flush 1/3 of the mem usage of this tablet channel.