pulsar icon indicating copy to clipboard operation
pulsar copied to clipboard

[improve][broker] Add dedicated partition configuration for system topics to reduce resource usage

Open zjxxzjwang opened this issue 11 months ago • 10 comments

Motivation

The number of partitions for creating system topics depends on the value of the "defaultNumPartitions" configuration field, but this field is also the number of partitions for automatically creating topics. If the value of defaultNumPartitions is set to a large value, the system theme partition is also large, causing resource waste.

Modifications

New "systemTopicDefaultNumPartitions" configuration field, isolation system topic creation and automatically create topic for "defaultNumPartitions" fields of mutual dependence. When metadata is mounted, the system topic and common topic depend on the default partition configuration parameters

Documentation

  • [ ] doc
  • [ ] doc-required
  • [x] doc-not-needed
  • [ ] doc-complete

Matching PR in forked repository

PR in forked repository:

zjxxzjwang avatar Dec 26 '24 07:12 zjxxzjwang

LGTM

HQebupt avatar Jan 03 '25 06:01 HQebupt

/pulsarbot rerun-failure-checks

HQebupt avatar Jan 03 '25 06:01 HQebupt

Please rename the title of this PR since it's currently misleading.

lhotari avatar Jan 03 '25 08:01 lhotari

/pulsarbot rerun-failure-checks

lhotari avatar Jan 03 '25 08:01 lhotari

/pulsarbot rerun-failure-checks

HQebupt avatar Jan 06 '25 08:01 HQebupt

Codecov Report

All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 74.16%. Comparing base (bbc6224) to head (5b52438). Report is 834 commits behind head on master.

Additional details and impacted files

Impacted file tree graph

@@             Coverage Diff              @@
##             master   #23785      +/-   ##
============================================
+ Coverage     73.57%   74.16%   +0.59%     
+ Complexity    32624     2376   -30248     
============================================
  Files          1877     1853      -24     
  Lines        139502   143471    +3969     
  Branches      15299    16292     +993     
============================================
+ Hits         102638   106406    +3768     
+ Misses        28908    28675     -233     
- Partials       7956     8390     +434     
Flag Coverage Δ
inttests 26.67% <25.00%> (+2.08%) :arrow_up:
systests 23.14% <25.00%> (-1.19%) :arrow_down:
unittests 73.69% <100.00%> (+0.85%) :arrow_up:

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines Coverage Δ
...org/apache/pulsar/broker/ServiceConfiguration.java 98.15% <100.00%> (-1.25%) :arrow_down:
...rg/apache/pulsar/broker/service/BrokerService.java 84.94% <100.00%> (+4.16%) :arrow_up:

... and 1019 files with indirect coverage changes

codecov-commenter avatar Jan 06 '25 09:01 codecov-commenter

@lhotari Hello, please help me review

zjxxzjwang avatar Jan 09 '25 10:01 zjxxzjwang

/pulsarbot rerun-failure-checks

HQebupt avatar Jan 09 '25 11:01 HQebupt

@lhotari hello,please help me review it.

zjxxzjwang avatar Jan 14 '25 08:01 zjxxzjwang

@lhotari i,Hello, can you help me review it

zjxxzjwang avatar Feb 24 '25 07:02 zjxxzjwang