pulsar icon indicating copy to clipboard operation
pulsar copied to clipboard

[improve] [broker] improve the restart broker can load bundle as one of the best brokers when selectBroker in LeastResourceUsageWithWeight

Open Nicklee007 opened this issue 2 years ago • 1 comments

Motivation

As the brokerAvgResourceUsageWithWeight stored the broker's history resource usage, if a broker is high load before restart, after the broker restart always show a high load history resource usage, and loadBalancerHistoryResourcePercentage = 0.9, which will cause the first few times the broker can not be select as best broker to load bundles. Only when selectBrokerForAssignment invoke few time's , the brokerAvgResourceUsageWithWeight could be neutralized by the current resourceUsage. But bundles unload is not a frequently operator if not disable doLoadShedding,it will wait a long time; So if a broker restarted we can clean the brokers history resource usage to guaranty the no bundles or no msgRate broker can be select as the one of best brokers.

https://github.com/apache/pulsar/blob/bc94643bc1a7f365dfb75e389ac3e1156770a119/pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/impl/LeastResourceUsageWithWeight.java#L99-L100

Modifications

  1. If the broker not own bundles or the msgRate is 0, use current resourceUsage to cover the historyUsage.
  2. add some unit test.

Documentation

  • [X] doc-not-needed (Please explain why)

Nicklee007 avatar Jul 25 '22 07:07 Nicklee007

/pulsarbot run-failure-checks

Nicklee007 avatar Aug 30 '22 15:08 Nicklee007

The pr had no activity for 30 days, mark with Stale label.

github-actions[bot] avatar Oct 22 '22 02:10 github-actions[bot]

/pulsarbot run-failure-checks

Nicklee007 avatar Dec 08 '22 02:12 Nicklee007

Codecov Report

Merging #16766 (3c0c603) into master (99e26f5) will increase coverage by 0.41%. The diff coverage is 0.00%.

Impacted file tree graph

@@             Coverage Diff              @@
##             master   #16766      +/-   ##
============================================
+ Coverage     44.79%   45.20%   +0.41%     
- Complexity    10026    10817     +791     
============================================
  Files           703      763      +60     
  Lines         68808    73575    +4767     
  Branches       7375     7912     +537     
============================================
+ Hits          30822    33260    +2438     
- Misses        34378    36592    +2214     
- Partials       3608     3723     +115     
Flag Coverage Δ
unittests 45.20% <0.00%> (+0.41%) :arrow_up:

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files Coverage Δ
...loadbalance/impl/LeastResourceUsageWithWeight.java 0.00% <0.00%> (ø)
...pache/bookkeeper/mledger/impl/RangeSetWrapper.java 43.39% <0.00%> (-2.61%) :arrow_down:
...org/apache/bookkeeper/mledger/util/RangeCache.java 67.04% <0.00%> (-2.28%) :arrow_down:
...he/pulsar/broker/admin/v2/NonPersistentTopics.java 60.64% <0.00%> (-1.39%) :arrow_down:
...rg/apache/pulsar/client/impl/PulsarClientImpl.java 43.85% <0.00%> (-0.95%) :arrow_down:
...apache/pulsar/client/impl/AutoClusterFailover.java 69.44% <0.00%> (-0.56%) :arrow_down:
...he/pulsar/client/impl/MultiTopicsConsumerImpl.java 22.86% <0.00%> (-0.12%) :arrow_down:
...pulsar/metadata/impl/stats/MetadataStoreStats.java 88.88% <0.00%> (ø)
...g/apache/pulsar/metadata/api/NotificationType.java 100.00% <0.00%> (ø)
...ache/pulsar/metadata/api/MetadataStoreFactory.java 0.00% <0.00%> (ø)
... and 154 more

codecov-commenter avatar Dec 08 '22 08:12 codecov-commenter