[Bug] ClassNotFoundException occurred when specifying `flink.yarn.ship-files` for flink engine

[wForget]

Code of Conduct

• [X] I agree to follow this project's Code of Conduct

Search before asking

• [X] I have searched in the issues and found no similar issues.

Describe the bug

ClassNotFoundException occurred when specifying flink.yarn.ship-files for flink engine.

Error detail:

2024-04-28 17:00:08,573 ERROR org.apache.flink.runtime.entrypoint.ClusterEntrypoint        [] - Could not create application program.
java.lang.RuntimeException: Could not look up the main(String[]) method from the class org.apache.kyuubi.engine.flink.FlinkSQLEngine: org/apache/flink/table/gateway/service/context/DefaultContext
	at org.apache.flink.client.program.PackagedProgram.hasMainMethod(PackagedProgram.java:315) ~[flink-dist-1.18.0.jar:1.18.0]
	at org.apache.flink.client.program.PackagedProgram.<init>(PackagedProgram.java:161) ~[flink-dist-1.18.0.jar:1.18.0]
	at org.apache.flink.client.program.PackagedProgram.<init>(PackagedProgram.java:65) ~[flink-dist-1.18.0.jar:1.18.0]
	at org.apache.flink.client.program.PackagedProgram$Builder.build(PackagedProgram.java:691) ~[flink-dist-1.18.0.jar:1.18.0]
	at org.apache.flink.client.program.DefaultPackagedProgramRetriever.getPackagedProgram(DefaultPackagedProgramRetriever.java:213) ~[flink-dist-1.18.0.jar:1.18.0]
	at org.apache.flink.yarn.entrypoint.YarnApplicationClusterEntryPoint.getPackagedProgram(YarnApplicationClusterEntryPoint.java:126) ~[flink-dist-1.18.0.jar:1.18.0]
	at org.apache.flink.yarn.entrypoint.YarnApplicationClusterEntryPoint.main(YarnApplicationClusterEntryPoint.java:96) [flink-dist-1.18.0.jar:1.18.0]
Caused by: java.lang.NoClassDefFoundError: org/apache/flink/table/gateway/service/context/DefaultContext
	at java.lang.Class.getDeclaredMethods0(Native Method) ~[?:1.8.0_382]
	at java.lang.Class.privateGetDeclaredMethods(Class.java:2701) ~[?:1.8.0_382]
	at java.lang.Class.privateGetMethodRecursive(Class.java:3048) ~[?:1.8.0_382]
	at java.lang.Class.getMethod0(Class.java:3018) ~[?:1.8.0_382]
	at java.lang.Class.getMethod(Class.java:1784) ~[?:1.8.0_382]
	at org.apache.flink.client.program.PackagedProgram.hasMainMethod(PackagedProgram.java:307) ~[flink-dist-1.18.0.jar:1.18.0]
	... 6 more
Caused by: java.lang.ClassNotFoundException: org.apache.flink.table.gateway.service.context.DefaultContext
	at java.net.URLClassLoader.findClass(URLClassLoader.java:387) ~[?:1.8.0_382]
	at java.lang.ClassLoader.loadClass(ClassLoader.java:418) ~[?:1.8.0_382]
	at org.apache.flink.util.FlinkUserCodeClassLoader.loadClassWithoutExceptionHandling(FlinkUserCodeClassLoader.java:67) ~[flink-dist-1.18.0.jar:1.18.0]
	at org.apache.flink.util.ChildFirstClassLoader.loadClassWithoutExceptionHandling(ChildFirstClassLoader.java:65) ~[flink-dist-1.18.0.jar:1.18.0]
	at org.apache.flink.util.FlinkUserCodeClassLoader.loadClass(FlinkUserCodeClassLoader.java:51) ~[flink-dist-1.18.0.jar:1.18.0]
	at java.lang.ClassLoader.loadClass(ClassLoader.java:351) ~[?:1.8.0_382]
	at java.lang.Class.getDeclaredMethods0(Native Method) ~[?:1.8.0_382]
	at java.lang.Class.privateGetDeclaredMethods(Class.java:2701) ~[?:1.8.0_382]
	at java.lang.Class.privateGetMethodRecursive(Class.java:3048) ~[?:1.8.0_382]
	at java.lang.Class.getMethod0(Class.java:3018) ~[?:1.8.0_382]
	at java.lang.Class.getMethod(Class.java:1784) ~[?:1.8.0_382]
	at org.apache.flink.client.program.PackagedProgram.hasMainMethod(PackagedProgram.java:307) ~[flink-dist-1.18.0.jar:1.18.0]
	... 6 more

Affects Version(s)

master

Kyuubi Server Log Output

No response

Kyuubi Engine Log Output

No response

Kyuubi Server Configurations

No response

Kyuubi Engine Configurations

flink.yarn.ship-files=customJars

Additional context

No response

Are you willing to submit PR?

• [X] Yes. I would be willing to submit a PR with guidance from the Kyuubi community to fix.
• [ ] No. I cannot submit a PR at this time.