apache
[Bug] Kyuubi job status always shows as "Succeeded" even when the actual job has failed
Code of Conduct
- [x] I agree to follow this project's Code of Conduct
Search before asking
- [x] I have searched in the issues and found no similar issues.
Describe the bug
Kyuubi-1.10.2 and spark-3.5.6. Kyuubi job status always shows as "Succeeded" even when the actual job has failed. My execution script is as follows: beeline -u "jdbc:hive2://kyuubiserver.my.com:10009/myuser;principal=hive/[email protected];hive.server2.proxy.user=myuser" --hiveconf spark.yarn.queue=center --hiveconf spark.executor.memory=4g -e "insert overwrite directory 'hdfs:///user/myuser/tmp/2025-07-23' select * from ..."
then job faied with error such as: Caused by: org.apache.kyuubi.plugin.spark.authz.AccessControlException: Permission denied: user [myuser] does not have [write] privilege on ...
But the final job status shown on the Spark log page is "Succeeded", which makes it impossible to determine the actual job outcome based on the job status. Job status monitoring and retry mechanisms become completely unusable.
Affects Version(s)
1.10.2
Kyuubi Server Log Output
kyuubi server log: 2025-08-01 10:50:48.340 INFO KyuubiSessionManager-exec-pool: Thread-166 org.apache.kyuubi.operation.ExecuteStatement: Processing myuser's query[71c32a89-bf36-405b-a581-3ca2b2a6202e]: RUNNING_STAT E -> ERROR_STATE, time taken: 0.439 seconds 2025-08-01 10:50:48.340 INFO KyuubiSessionManager-exec-pool: Thread-166 org.apache.kyuubi.operation.OperationAuditLogger: operation=71c32a89-bf36-405b-a581-3ca2b2a6202e opType=ExecuteSta tement state=ERROR user=myuser session=5d12d959-c7a6-4dfc-8c18-6a08baf63fbd 2025-08-01 10:50:48.412 INFO KyuubiTBinaryFrontendHandler-Pool: Thread-155 org.apache.kyuubi.operation.ExecuteStatement: Processing myuser's query[71c32a89-bf36-405b-a581-3ca2b2a6202e]: ERROR_STA TE -> CLOSED_STATE, time taken: 0.511 seconds 2025-08-01 10:50:48.412 INFO KyuubiTBinaryFrontendHandler-Pool: Thread-155 org.apache.kyuubi.operation.OperationAuditLogger: operation=71c32a89-bf36-405b-a581-3ca2b2a6202e opType=ExecuteSta tement state=CLOSED user=myuser session=5d12d959-c7a6-4dfc-8c18-6a08baf63fbd 2025-08-01 10:50:48.441 INFO KyuubiTBinaryFrontendHandler-Pool: Thread-155 org.apache.kyuubi.client.KyuubiSyncThriftClient: TCloseOperationReq(operationHandle:TOperationHandle(operationId:THand leIdentifier(guid:71 C3 2A 89 BF 36 40 5B A5 81 3C A2 B2 A6 20 2E, secret:C2 EE 5B 97 3E A0 41 FC AC 16 9B D7 08 ED 8F 38), operationType:EXECUTE_STATEMENT, hasResultSet:true)) succeed on engin e side 2025-08-01 10:50:48.471 INFO KyuubiTBinaryFrontendHandler-Pool: Thread-155 org.apache.kyuubi.server.KyuubiTBinaryFrontendService: Received request of closing SessionHandle [5d12d959-c7a6-4dfc-8 c18-6a08baf63fbd] 2025-08-01 10:50:48.473 INFO KyuubiTBinaryFrontendHandler-Pool: Thread-155 org.apache.kyuubi.session.KyuubiSessionManager: myuser's KyuubiSessionImpl with SessionHandle [5d12d959-c7a6-4dfc-8c18-6 a08baf63fbd] is closed, current opening sessions 0 2025-08-01 10:50:48.474 INFO KyuubiTBinaryFrontendHandler-Pool: Thread-155 org.apache.kyuubi.operation.LaunchEngine: Processing myuser's query[2af47c76-a3a6-42fe-bd34-4c3aea3933f2]: FINISHED_STAT E -> CLOSED_STATE, time taken: 34.601 seconds 2025-08-01 10:50:48.474 INFO KyuubiTBinaryFrontendHandler-Pool: Thread-155 org.apache.kyuubi.operation.OperationAuditLogger: operation=2af47c76-a3a6-42fe-bd34-4c3aea3933f2 opType=LaunchEngi ne state=CLOSED user=myuser session=5d12d959-c7a6-4dfc-8c18-6a08baf63fbd 2025-08-01 10:50:48.537 INFO KyuubiTBinaryFrontendHandler-Pool: Thread-155 org.apache.kyuubi.server.KyuubiTBinaryFrontendService: Finished closing SessionHandle [5d12d959-c7a6-4dfc-8c18-6a08baf 63fbd] 2025-08-01 10:54:25.523 INFO KyuubiSessionManager-timeout-checker: Thread-63 org.apache.kyuubi.session.KyuubiSessionManager: Checking sessions timeout, current count: 0
Kyuubi Engine Log Output
Kyuubi Server Configurations
Kyuubi Engine Configurations
Additional context
No response
Are you willing to submit PR?
- [x] Yes. I would be willing to submit a PR with guidance from the Kyuubi community to fix.
- [ ] No. I cannot submit a PR at this time.
It's by design, in this case, you will get an error report from your BeeLine client.
BeeLine client can get an error, But the existing job monitoring is all based on the YARN API. Is there any plan to support correcting job status in the future? Or is there a parameter can correcting job status ? You're very much appreciated.
I encountered the same issue under the sharelevel of connection. The JDBC client can receive the error, but the YARN app status shows as successful, which is somewhat troublesome for auditing. It seems likely that this is caused by the Kyuubi engine layer manually calling SparkContext.stop().
