[KYUUBI #3503] Beeline should respect kyuubi-defaults.conf in default

[lightning-L]

Why are the changes needed?

How was this patch tested?

• [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

• [ ] Add screenshots for manual tests if appropriate

• [ ] Run test locally before make a pull request

[codecov-commenter]

Codecov Report

Merging #3724 (210cb40) into master (bd35784) will decrease coverage by 0.14%. The diff coverage is 39.48%.

@@             Coverage Diff              @@
##             master    #3724      +/-   ##
============================================
- Coverage     52.65%   52.51%   -0.15%     
  Complexity       13       13              
============================================
  Files           494      496       +2     
  Lines         27788    27983     +195     
  Branches       3835     3851      +16     
============================================
+ Hits          14631    14694      +63     
- Misses        11765    11893     +128     
- Partials       1392     1396       +4     

Impacted Files	Coverage Δ
...in/java/org/apache/hive/beeline/KyuubiBeeLine.java	0.00% <0.00%> (ø)
...line/hs2connection/DefaultConnectionUrlParser.java	0.00% <0.00%> (ø)
.../scala/org/apache/kyuubi/beeline/BeelineConf.scala	93.90% <93.90%> (ø)
.../kyuubi/server/mysql/constant/MySQLErrorCode.scala	13.84% <0.00%> (-6.16%)	:arrow_down:
...ache/kyuubi/server/mysql/MySQLCommandHandler.scala	76.34% <0.00%> (-4.31%)	:arrow_down:
...ache/kyuubi/server/mysql/MySQLGenericPackets.scala	76.59% <0.00%> (-2.13%)	:arrow_down:
...rg/apache/kyuubi/ctl/cmd/log/LogBatchCommand.scala	63.93% <0.00%> (-1.64%)	:arrow_down:
...yuubi/server/metadata/jdbc/JDBCMetadataStore.scala	89.27% <0.00%> (-0.70%)	:arrow_down:
...che/kyuubi/server/KyuubiTHttpFrontendService.scala	59.86% <0.00%> (-0.69%)	:arrow_down:
...in/scala/org/apache/kyuubi/config/KyuubiConf.scala	97.46% <0.00%> (-0.08%)	:arrow_down:
... and 2 more

:mega: We’re building smart automated test selection to slash your CI/CD build times. Learn more

[turboFei]

FYI: we shall not introduce scala modules into kyuubi-hive-jdbc.

[lightning-L]

Before modification:

After modification:

[lightning-L]

FYI: we shall not introduce scala modules into kyuubi-hive-jdbc.

ok....

[pan3793]

FYI: we shall not introduce scala modules into kyuubi-hive-jdbc.

Yea, but it's allowed for kyuubi-hive-beeline, because it is not designed for downstream projects use it as a deps as kyuubi-hive-jdbc does.

[turboFei]

Sorry, my mistake, it is not for jdbc, beeline is OK.

[lightning-L]

@pan3793 @turboFei Could you help review the current version of this implementation?

Hive beeline has this functionality originally: https://cwiki.apache.org/confluence/display/hive/hiveserver2+clients#HiveServer2Clients-Usingbeeline-site.xmltoautomaticallyconnecttoHiveServer2 It will generate connection URL based on hive-site.xml. (In our case, it is kyuubi conf) Not all the URL properties can be derived from hive-site.xml and hence in order to use this feature user must create a configuration file called “beeline-hs2-connection.xml”. -------- Do we also need this one? Hive beeline also allows user to define a beeline-site.xml to specify many complete JDBC URLs. And user can use beeline -c <named_url> to connect to a specific URL.

Reference: https://github.com/apache/hive/blob/master/beeline/src/java/org/apache/hive/beeline/BeeLine.java#L1172

[pan3793]

Personally, I lean toward adding a BeelineConf just like HighAvailabilityConf as a replacement for beeline-hs2-connection.xml, and we may remove the hive deps from kyuubi-beeline eventually.

[lightning-L]

@pan3793 @turboFei Has completed rewriting. Will read connection params from BeelineConf.

[turboFei]

Could you rebase on master？ There is some ci issue.

[lightning-L]

Could you rebase on master？ There is some ci issue.

already rebased. but still has failures....

[turboFei]

- Fail connections on invalid sub domains *** FAILED ***
  Expected exception java.sql.SQLException to be thrown, but no exception was thrown (JDBCTestHelper.scala:51)

KyuubiOperationHiveCatalogSuite:
- get catalogs
- get schemas
- get tables
- get type info
- audit Kyuubi Hive JDBC connection common MetaData *** FAILED ***
  "[Hive JDBC]" did not equal "[Kyuubi Project Hive JDBC Shaded Client]" (SparkMetadataTests.scala:407)
  Analysis:
  "[Hive JDBC]" -> "[Kyuubi Project Hive JDBC Shaded Client]"

Seems it is using hive jdbc instead of kyuubi hive jdbc.

[turboFei]

Per the hive doc https://cwiki.apache.org/confluence/display/hive/hiveserver2+clients#HiveServer2Clients-Usingbeeline-site.xmltoautomaticallyconnecttoHiveServer2

<property>
  <name>beeline.hs2.jdbc.url.tcpUrl</name>
  <value>jdbc:hive2://localhost:10000/default;user=hive;password=hive</value>
</property>
 
<property>
  <name>beeline.hs2.jdbc.url.httpUrl</name>
  <value>jdbc:hive2://localhost:10000/default;user=hive;password=hive;transportMode=http;httpPath=cliservice</value>
</property>
 
<property>
  <name>beeline.hs2.jdbc.url.default</name>
  <value>tcpUrl</value>
</property>

Seems it is more intuitive.

[turboFei]

cc @pan3793

[lightning-L]

Per the hive doc https://cwiki.apache.org/confluence/display/hive/hiveserver2+clients#HiveServer2Clients-Usingbeeline-site.xmltoautomaticallyconnecttoHiveServer2

<property>
  <name>beeline.hs2.jdbc.url.tcpUrl</name>
  <value>jdbc:hive2://localhost:10000/default;user=hive;password=hive</value>
</property>
 
<property>
  <name>beeline.hs2.jdbc.url.httpUrl</name>
  <value>jdbc:hive2://localhost:10000/default;user=hive;password=hive;transportMode=http;httpPath=cliservice</value>
</property>
 
<property>
  <name>beeline.hs2.jdbc.url.default</name>
  <value>tcpUrl</value>
</property>

Seems it is more intuitive.

This is another feature: allows user to define a beeline-site.xml to specify many complete JDBC URLs. And user can use beeline -c <named_url> to connect to a specific URL.

Current implementation is similar to:https://cwiki.apache.org/confluence/display/hive/hiveserver2+clients#HiveServer2Clients-Usinghive-site.xmltoautomaticallyconnecttoHiveServer2 It will determine the connection URL based on configuration.

[lightning-L]

I could support beeline -c <named_url> too

[turboFei]

I could support beeline -c <named_url> too

What is beeline -c? a new beeline option.

[turboFei]

how do you think about? @pan3793

discussed with @lightning-L offline.

This pr is aimed to implement the function similar with beeline-hs2-connection.xm and hive-site.xml

For the named url, it should be available by leveraging the beeline-site.xml.

[lightning-L]

Since hive beeline already supports default connection url, we could set beeline-hs2-connection.xml and beeline-site.xml to get desired url as stated in the wiki: https://cwiki.apache.org/confluence/display/hive/hiveserver2+clients#HiveServer2Clients-Usingbeeline-site.xmltoautomaticallyconnecttoHiveServer2 I think we may not need to implement a kyuubi conversion (to get all values from kyuubi-defaults.conf).