[KYUUBI #6034] Kyuubi Server HA&ZK get server from serverHosts support more strategy

[davidyuan1223]

:mag: Description

Issue References 🔗

This pull request fixes #6034

Describe Your Solution 🔧

Currently, use beeline to connect kyuubiServer with HA mode, the strategy only support random, this will lead to a high load on the machine. So i make this pr to support choose strategy. [description] First, we need know, beeline connect kyuubiServer dependency on kyuubi-hive-jdbc, it is isolated from the kyuubi cluster, so the code only support random choose serverHost from zk node /${namespace}. Because kyuubi-hive-jdbc is a stateless module, only run once, cannot store var about get serverHost from zk node. [Solution] This pr, we could implement a interface named ChooseServerStrategy to choose serverHost. I implement two strategy

1. poll: it will create a zk node named ${namespace}-counter, when a beeline client want connect kyuubiServer, the node will increment 1, use this value to take the remainder from serverHosts, like counter % serverHost.size, so we could get a order serverHost
2. random: random get serverHost from serverHosts
3. User Definied Class: implemented the ChooseServerStrategy, then put the jar to beeline-jars, it can use your strategy to choose serverHost

Types of changes :bookmark:

• [ ] Bugfix (non-breaking change which fixes an issue)
• [x] New feature (non-breaking change which adds functionality)
• [ ] Breaking change (fix or feature that would cause existing functionality to change)

Test Plan 🧪

Test the Strategy in my test Cluster

Behavior Without This Pull Request :coffin:

Behavior With This Pull Request :tada:

[Use Case]

1. poll: bin/beeline -u 'jdbc:hive2://xxx:2181/;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=kyuubi;zooKeeperStrategy=poll?spark.yarn.queue=root.kylin;spark.app.name=testspark;spark.shuffle.useOldFetchProtocol=true' -n mfw_hadoop --verbose=true --showNestedErrs=true
2. random: bin/beeline -u 'jdbc:hive2://xxx:2181/;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=kyuubi;zooKeeperStrategy=random?spark.yarn.queue=root.kylin;spark.app.name=testspark;spark.shuffle.useOldFetchProtocol=true' -n mfw_hadoop --verbose=true --showNestedErrs=true or bin/beeline -u 'jdbc:hive2://xxx:2181/;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=kyuubi?spark.yarn.queue=root.kylin;spark.app.name=testspark;spark.shuffle.useOldFetchProtocol=true' -n mfw_hadoop --verbose=true --showNestedErrs=true
3. YourStrategy: bin/beeline -u 'jdbc:hive2://xxx:2181/;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=kyuubi;zooKeeperStrategy=xxx.xxx.xxx.XxxChooseServerStrategy?spark.yarn.queue=root.kylin;spark.app.name=testspark;spark.shuffle.useOldFetchProtocol=true' -n mfw_hadoop --verbose=true --showNestedErrs=true

[Result: The Cluster have two Server (221,233)]

1. poll: 1.1. zkNode: counterValue

1.2. result:

2. random:

3. YourStrategy(the test case only get the first serverHost):

Related Unit Tests

There is no Unit Tests.

Checklist 📝

• [ ] This patch was not authored or co-authored using Generative Tooling

Be nice. Be informative.

[davidyuan1223]

@yaooqinn @wForget @pan3793 what do you think?

[yaooqinn]

LGTM in general, can we add some tests?

[davidyuan1223]

LGTM in general, can we add some tests?

i'm not sure did this improvement could execute unit test? But i will try

[yaooqinn]

function level unit test is OK

[cxzl25]

It may be similar to the idea of ​​selecting Engine that we are planning to implement. https://github.com/apache/kyuubi/pull/5662 cc @lsm1

[davidyuan1223]

@cxzl25 @yaooqinn what do you think about the unit test case in ha module(because we need start a zk docker to test the poll strategy)

[codecov-commenter]

Codecov Report

Attention: Patch coverage is 55.17241% with 13 lines in your changes are missing coverage. Please review.

Project coverage is 58.43%. Comparing base (3b2e674) to head (d6c58ee). Report is 170 commits behind head on master.

Files	Patch %	Lines
...che/kyuubi/jdbc/hive/strategy/StrategyFactory.java	16.66%	9 Missing and 1 partial :warning:
...i/jdbc/hive/strategy/zk/PollingChooseStrategy.java	66.66%	2 Missing and 1 partial :warning:

Additional details and impacted files

@@             Coverage Diff              @@
##             master    #6213      +/-   ##
============================================
- Coverage     61.10%   58.43%   -2.68%     
- Complexity       23       24       +1     
============================================
  Files           622      655      +33     
  Lines         37036    39673    +2637     
  Branches       5024     5456     +432     
============================================
+ Hits          22631    23182     +551     
- Misses        11967    14003    +2036     
- Partials       2438     2488      +50     

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.