pulsar icon indicating copy to clipboard operation
pulsar copied to clipboard

support setting bundle number for default namespace when set up cluster

Open aloyszhang opened this issue 2 years ago • 3 comments

Motivation

In the actual production environment, users may only use one default namespace (public/default). The number of bundles in this namespace is a default value of 16.

  1. if the cluster has more than 16 brokers, some brokers will serve no traffic
  2. if the topics in these bundles are not equal, the load will be unbalanced In addition, the manually split bundle is complicated and auto-split may lead to unstable services.

We expect to set a relatively large number of bundles for the default namespace when the cluster is built.

Modifications

support setting bundle number for default namespace when setting up cluster metadata and the bundle number is the defaultNumberOfNamespaceBundles from broker.conf

Verifying this change

  • [ ] Make sure that the change passes the CI checks.

This change added tests and can be verified as follows: ClusterMetadataSetupTest.testSetupBundleForNamespaces

Documentation

  • [x] doc-not-needed (Please explain why)

PR in for repository

https://github.com/aloyszhang/pulsar/pull/2

aloyszhang avatar Sep 19 '22 17:09 aloyszhang

checks in fork repo : https://github.com/aloyszhang/pulsar/pull/2

aloyszhang avatar Sep 20 '22 03:09 aloyszhang

/pulsarbot run-failure-checks

aloyszhang avatar Sep 20 '22 06:09 aloyszhang

@merlimat PTAL, thanks

aloyszhang avatar Sep 20 '22 13:09 aloyszhang

cc @merlimat

aloyszhang avatar Sep 26 '22 07:09 aloyszhang

@codelipenghui PTAL, thanks

aloyszhang avatar Sep 28 '22 07:09 aloyszhang

After discussing with @codelipenghui, getting bundle numbers for default namespaces from broker.conf has a disadvantage in the k8s environment(In k8s, we build pulsar as follows: build zk container, then start a container to init cluster metadata, at this point, we can't access the broker.conf at all).

So, adding a new option like --default-namespace-bundle-numbers is better. @merlimat

aloyszhang avatar Sep 28 '22 13:09 aloyszhang

cc @codelipenghui

aloyszhang avatar Sep 29 '22 09:09 aloyszhang