[Bug] Kyuubi Spark Authz show databases/tables no data returned

[lugela]

Code of Conduct

• [X] I agree to follow this project's Code of Conduct

Search before asking

• [X] I have searched in the issues and found no similar issues.

Describe the bug

Kyuubi Spark Authz is connected to ranger1.2.0 and configured with relevant policies

This policy can be displayed normally on the spark-sql command line

However, through access in kyuubi, the beeline/ldap method returns empty，the select specific strategy table can query data normally

Affects Version(s)

1.8.0

Kyuubi Server Log Output

no

Kyuubi Engine Log Output

no

Kyuubi Server Configurations

no

Kyuubi Engine Configurations

no

Additional context

no

Are you willing to submit PR?

• [ ] Yes. I would be willing to submit a PR with guidance from the Kyuubi community to fix.
• [X] No. I cannot submit a PR at this time.

[yaooqinn]

Did beeline and spark-sql have the same user identity?

[lugela]

Did beeline and spark-sql have the same user identity?

yes ,It has been verified that it should be a bug in kyuubi 1.8.0. It is normal for kyuubi to fall back to 1.7.3

[lugela]

Did beeline and spark-sql have the same user identity?

yes ,It has been verified that it should be a bug in kyuubi 1.8.0. It is normal for kyuubi to fall back to 1.7.3

It was found that it was caused by this parameter spark.submit.deployMode=cluster

[yaooqinn]

Do you know if there is any additional log for cluster mode? I guess it might be a problem for the driver to talk with the ranger admin or localize policy files

[eubnara]

I'm worried about the fact that getting ranger policy json files from ranger admin process to all the spark applications. It may be ddos attack to ranger admin process, isn't it?

[eubnara]

Also, I use both cluster and client for spark.submit.deployMode but show databases and show tables gives me empty result. However, select * from query works.

(HiveMetastore 3.1.3, Spark 3.2.3, Kyuubi 1.8.1)

[pan3793]

I'm worried about the fact that getting ranger policy json files from ranger admin process to all the spark applications. It may be ddos attack to ranger admin process, isn't it?

Definitely, especially when you have a lot of policies, TBH, it's a limitation of Ranger's architecture, AFAIK, there are many users extend the Ranger to support the server-side policy evaluation to address such issues.

[eubnara]

Thanks for your opinion, I think if authorization only occurs on Kyuubi server side with ranger and uses spark just as engine like HiveServer2, the problem will be alleviated.