[TASK][EASY] Support using flink-table-planner explicitly

[ysmintor]

Code of Conduct

• [X] I agree to follow this project's Code of Conduct

Search before asking

• [X] I have searched in the issues and found no similar issues.

Describe the feature

#Table Planner and Table Planner Loader

Starting from Flink 1.15, the distribution contains two planners:

• flink-table-planner_2.12-1.20-SNAPSHOT.jar, in /opt, contains the query planner
• flink-table-planner-loader-1.20-SNAPSHOT.jar, loaded by default in /lib, contains the query planner hidden behind an isolated classpath (you won’t be able to address any io.apache.flink.table.planner directly)

The two planners cannot co-exist at the same time in the classpath. If you load both of them in /lib your Table Jobs will fail.

Because Dinky and hudi use flink-table-planner, so I have to put flink-table-planner into flink /lib, however Apache kyuubi will check flink-table-planner-loader in path. So could you adapt to use flink-table-planner_2.12?

If use flink-table-planner, error will be

Error: Error operating ExecuteStatement: org.apache.flink.table.api.TableException: Error obtaining Flink planner module ClassLoader. Make sure a flink-table-planner-loader.jar is on the classpath
        at org.apache.kyuubi.engine.flink.operation.ExecuteStatement.getPlannerModuleClassLoader(ExecuteStatement.scala:117)
        at org.apache.kyuubi.engine.flink.operation.ExecuteStatement.parseExtendedStatement(ExecuteStatement.scala:90)
        at org.apache.kyuubi.engine.flink.operation.ExecuteStatement.executeStatement(ExecuteStatement.scala:69)
        at org.apache.kyuubi.engine.flink.operation.ExecuteStatement.runInternal(ExecuteStatement.scala:62)
        at org.apache.kyuubi.operation.AbstractOperation.run(AbstractOperation.scala:173)
        at org.apache.kyuubi.session.AbstractSession.runOperation(AbstractSession.scala:100)
        at org.apache.kyuubi.session.AbstractSession.$anonfun$executeStatement$1(AbstractSession.scala:130)
        at org.apache.kyuubi.session.AbstractSession.withAcquireRelease(AbstractSession.scala:81)
        at org.apache.kyuubi.session.AbstractSession.executeStatement(AbstractSession.scala:127)
        at org.apache.kyuubi.service.AbstractBackendService.executeStatement(AbstractBackendService.scala:66)
        at org.apache.kyuubi.service.TFrontendService.ExecuteStatement(TFrontendService.scala:253)
        at org.apache.kyuubi.shaded.hive.service.rpc.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1557)
        at org.apache.kyuubi.shaded.hive.service.rpc.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1542)
        at org.apache.kyuubi.shaded.thrift.ProcessFunction.process(ProcessFunction.java:38)
        at org.apache.kyuubi.shaded.thrift.TBaseProcessor.process(TBaseProcessor.java:38)
        at org.apache.kyuubi.service.authentication.TSetIpAddressProcessor.process(TSetIpAddressProcessor.scala:35)
        at org.apache.kyuubi.shaded.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:250)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:750)
Caused by: java.lang.NoSuchMethodException: Cannot find method: getInstance
        at org.apache.kyuubi.util.reflect.DynMethods$Builder.buildChecked(DynMethods.java:473)
        at org.apache.kyuubi.util.reflect.DynMethods$Builder.buildStaticChecked(DynMethods.java:500)
        at org.apache.kyuubi.engine.flink.operation.ExecuteStatement.getPlannerModuleClassLoader(ExecuteStatement.scala:105)
        ... 19 more (state=,code=0)

Motivation

No response

Describe the solution

No response

Additional context

No response

Are you willing to submit PR?

• [ ] Yes. I would be willing to submit a PR with guidance from the Kyuubi community to improve.
• [X] No. I cannot submit a PR at this time.

[github-actions[bot]]

Hello @ysmintor, Thanks for finding the time to report the issue! We really appreciate the community's efforts to improve Apache Kyuubi.

[pan3793]

cc @link3280

[link3280]

Yes, that's a valid request.

[Liam3433]

I tested it in Kyuubi version 1.8.2, it's fine to put the flink-table-planner_2.12-1.8.1.jar under the flink lib. But in Kyuubi version 1.9.0, it will give the above same error.

[Pandas886]

Any progress?

[tyoushinya]

@link3280 Any progress?