doris icon indicating copy to clipboard operation
doris copied to clipboard

[Enhancement](load) Refine the load channel flush policy on mem limit

Open zhannngchen opened this issue 2 years ago • 2 comments

Proposed changes

Issue Number: close #xxx

Problem summary

Testing, will complete the description later.

Checklist(Required)

  1. Does it affect the original behavior:
    • [ ] Yes
    • [ ] No
    • [ ] I don't know
  2. Has unit tests been added:
    • [ ] Yes
    • [ ] No
    • [ ] No Need
  3. Has document been added or modified:
    • [ ] Yes
    • [ ] No
    • [ ] No Need
  4. Does it need to update dependencies:
    • [ ] Yes
    • [ ] No
  5. Are there any changes that cannot be rolled back:
    • [ ] Yes (If Yes, please explain WHY)
    • [ ] No

Further comments

If this is a relatively large or complex change, kick off the discussion at [email protected] by explaining why you chose the solution you did and what alternatives you considered, etc...

zhannngchen avatar Sep 18 '22 15:09 zhannngchen

PR approved by at least one committer and no changes requested.

github-actions[bot] avatar Sep 22 '22 04:09 github-actions[bot]

PR approved by anyone and no changes requested.

github-actions[bot] avatar Sep 22 '22 04:09 github-actions[bot]

PR approved by at least one committer and no changes requested.

github-actions[bot] avatar Sep 23 '22 15:09 github-actions[bot]

PR approved by at least one committer and no changes requested.

github-actions[bot] avatar Sep 24 '22 01:09 github-actions[bot]

LGTM

dataalive avatar Sep 24 '22 01:09 dataalive

@zhannngchen Looking forward to updating the Problem summary

xinyiZzz avatar Sep 24 '22 01:09 xinyiZzz

commit comment:

  1. Remove single load channel mem limit, only use load channel mgr mem limit
  2. Default load channel mgr mem limit from 50% to 80%
  3. load channel mgr add soft mem limit. When the soft limit is exceeded, other threads will not hang, only current thread triggers flush
  4. When exceed load channel mgr mem limit, find a load channel with the largest mem usage, continue to find a tablet channel with the largest mem usage, and try to flush 1/3 of the mem usage of this tablet channel.

xinyiZzz avatar Sep 24 '22 02:09 xinyiZzz