apache
[Bug]: Kerberos authentication failure
What happened?
When using the External Catalog type, Mixed-Iceberg, Mixed-Hive, Iceberg table formats, and Kerberos authentication method, after creating a hive_catalog with Hive service tickets and keytab files, if you create an Iceberg table in that Catalog and a specific database and insert data, subsequent queries will prompt a client authentication failure. It will look like this:
Caused by: java.io.IOException: org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS] at org.apache.hadoop.ipc.Client$Connection$1.run(Client.java:754) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1953) at org.apache.hadoop.ipc.Client$Connection.handleSaslConnectionFailure(Client.java:709) at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:812) at org.apache.hadoop.ipc.Client$Connection.access$3800(Client.java:364) at org.apache.hadoop.ipc.Client.getConnection(Client.java:1649) at org.apache.hadoop.ipc.Client.call(Client.java:1473) ... 44 more Caused by: org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS] at org.apache.hadoop.security.SaslRpcClient.selectSaslClient(SaslRpcClient.java:179) at org.apache.hadoop.security.SaslRpcClient.saslConnect(SaslRpcClient.java:399) at org.apache.hadoop.ipc.Client$Connection.setupSaslConnection(Client.java:578) at org.apache.hadoop.ipc.Client$Connection.access$2100(Client.java:364) at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:799) at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:795) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1953) at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:795) ... 47 more
After I restarted the Amoro service using ams.sh restart, querying the iceberg table data worked normally. However, if I continued to click the query button, it would throw the authentication exception mentioned above again.
Interestingly, every time I restarted the service, the first query would succeed, but the N query would fail; then after restarting the service again, the first query would succeed, and the N query would fail...
Affects Versions
0.7.1
What table formats are you seeing the problem on?
Iceberg
What engines are you seeing the problem on?
AMS
How to reproduce
No response
Relevant log output
No response
Anything else
No response
Are you willing to submit a PR?
- [ ] Yes I am willing to submit a PR!
Code of Conduct
- [X] I agree to follow this project's Code of Conduct
It seems that AMS threw the error when querying from the web front, could you please share the whole log, thanks.
[org.apache.hadoop.ipc.Client] [] - Exception encountered while connecting to the server xxxxxxxx03/ip:端口 org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS] at org.apache.hadoop.security.SaslRpcClient.selectSaslClient(SaslRpcClient.java:179) ~[hadoop-common-3.4.0.jar:?] at org.apache.hadoop.security.SaslRpcClient.saslConnect(SaslRpcClient.java:399) ~[hadoop-common-3.4.0.jar:?] at org.apache.hadoop.ipc.Client$Connection.setupSaslConnection(Client.java:578) ~[hadoop-common-3.4.0.jar:?] at org.apache.hadoop.ipc.Client$Connection.access$2100(Client.java:364) ~[hadoop-common-3.4.0.jar:?] at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:799) ~[hadoop-common-3.4.0.jar:?] at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:795) ~[hadoop-common-3.4.0.jar:?] at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_141] at javax.security.auth.Subject.doAs(Subject.java:422) ~[?:1.8.0_141] at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1953) ~[hadoop-common-3.4.0.jar:?] at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:795) ~[hadoop-common-3.4.0.jar:?] at org.apache.hadoop.ipc.Client$Connection.access$3800(Client.java:364) ~[hadoop-common-3.4.0.jar:?] at org.apache.hadoop.ipc.Client.getConnection(Client.java:1649) ~[hadoop-common-3.4.0.jar:?] at org.apache.hadoop.ipc.Client.call(Client.java:1473) ~[hadoop-common-3.4.0.jar:?] at org.apache.hadoop.ipc.Client.call(Client.java:1426) ~[hadoop-common-3.4.0.jar:?] at org.apache.hadoop.ipc.ProtobufRpcEngine2$Invoker.invoke(ProtobufRpcEngine2.java:258) ~[hadoop-common-3.4.0.jar:?] at org.apache.hadoop.ipc.ProtobufRpcEngine2$Invoker.invoke(ProtobufRpcEngine2.java:139) ~[hadoop-common-3.4.0.jar:?] at com.sun.proxy.$Proxy61.getBlockLocations(Unknown Source) ~[?:?]
The front-end logs are as follows:
2024/12/06 19:16:00 prepare execute statement, line:1 2024/12/06 19:16:00 select * from dbb.cdrs 2024/12/06 19:16:00 meet exception during execution. 2024/12/06 19:16:00 org.apache.iceberg.exceptions.RuntimeIOException: Failed to get block locations for path: hdfs:///houser/tablespace/managed/hive/dbb.db/cdrs/data/create_time_day=2024-12-06/00120-6-e6f2443a-9344-4083-983a-0fcb79b5d9d9-00001.parquet at org.apache.iceberg.hadoop.HadoopInputFile.getBlockLocations(HadoopInputFile.java:217) at org.apache.iceberg.hadoop.Util.blockLocations(Util.java:111) at org.apache.iceberg.hadoop.Util.blockLocations(Util.java:84) at org.apache.iceberg.spark.source.SparkInputPartition.(SparkInputPartition.java:62) at org.apache.iceberg.spark.source.SparkBatch.lambda$planInputPartitions$0(SparkBatch.java:90) at org.apache.iceberg.util.Tasks$Builder.runTaskWithRetry(Tasks.java:413) at org.apache.iceberg.util.Tasks$Builder.access$300(Tasks.java:69) at org.apache.iceberg.util.Tasks$Builder$1.run(Tasks.java:315) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: DestHost:destPort xxxxxxxx:8010 , LocalHost:localPort xxxxxxxx06/ip:0. Failed on local exception: java.io.IOException: org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS] at sun.reflect.GeneratedConstructorAccessor92.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:948) at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:923) at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1588) at org.apache.hadoop.ipc.Client.call(Client.java:1529) at org.apache.hadoop.ipc.Client.call(Client.java:1426) at org.apache.hadoop.ipc.ProtobufRpcEngine2$Invoker.invoke(ProtobufRpcEngine2.java:258) at org.apache.hadoop.ipc.ProtobufRpcEngine2$Invoker.invoke(ProtobufRpcEngine2.java:139) at com.sun.proxy.$Proxy60.getBlockLocations(Unknown Source) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.lambda$getBlockLocations$0(ClientNamenodeProtocolTranslatorPB.java:340) at org.apache.hadoop.ipc.internal.ShadedProtobufHelper.ipc(ShadedProtobufHelper.java:160) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getBlockLocations(ClientNamenodeProtocolTranslatorPB.java:340) at sun.reflect.GeneratedMethodAccessor89.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:437) at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:170) at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:162) at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:100) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:366) at com.sun.proxy.$Proxy61.getBlockLocations(Unknown Source) at org.apache.hadoop.hdfs.DFSClient.callGetBlockLocations(DFSClient.java:931) at org.apache.hadoop.hdfs.DFSClient.getLocatedBlocks(DFSClient.java:920) at org.apache.hadoop.hdfs.DFSClient.getBlockLocations(DFSClient.java:977) at org.apache.hadoop.hdfs.DistributedFileSystem$2.doCall(DistributedFileSystem.java:289) at org.apache.hadoop.hdfs.DistributedFileSystem$2.doCall(DistributedFileSystem.java:286) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileBlockLocations(DistributedFileSystem.java:296) at org.apache.iceberg.hadoop.HadoopInputFile.getBlockLocations(HadoopInputFile.java:210) ... 12 more Caused by: java.io.IOException: org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS] at org.apache.hadoop.ipc.Client$Connection$1.run(Client.java:754) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1953) at org.apache.hadoop.ipc.Client$Connection.handleSaslConnectionFailure(Client.java:709) at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:812) at org.apache.hadoop.ipc.Client$Connection.access$3800(Client.java:364) at org.apache.hadoop.ipc.Client.getConnection(Client.java:1649) at org.apache.hadoop.ipc.Client.call(Client.java:1473) ... 36 more Caused by: org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS] at org.apache.hadoop.security.SaslRpcClient.selectSaslClient(SaslRpcClient.java:179) at org.apache.hadoop.security.SaslRpcClient.saslConnect(SaslRpcClient.java:399) at org.apache.hadoop.ipc.Client$Connection.setupSaslConnection(Client.java:578) at org.apache.hadoop.ipc.Client$Connection.access$2100(Client.java:364) at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:799) at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:795) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1953) at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:795) ... 39 more
Has the Kerberos ticket expired (after more than 7 days)? If you execute kinit -kt xx xx and then perform the same operation again, will you still encounter the same issue
This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs. To permanently prevent this issue from being considered stale, add the label 'not-stale', but commenting on the issue is preferred when possible.
![github-actions[bot] avatar](https://avatars.githubusercontent.com/in/15368?v=4 "Published by a pub.dev verified publisher"){kind=small}