[KYUUBI #3300] [FEATURE] [AUTHZ] WIP: Support overriding usergroup with UserStore in AccessRequest

[bowenliang123]

Why are the changes needed?

Fix https://github.com/apache/incubator-kyuubi/issues/3300

User group based policies from Ranger are not working as expectedly. User can be binded to multiple user groups in Ranger Admin. But in privileage checking, user group is identified by plugin it self. In Authz, User group param setting in AccessRequest currently relies on UserGroupInformation returned by getAuthzUgi method of AuthZUtils. RangerPlugin is plainly use the value of it to check policies conditions.

I suggest,

support overriding usergroup in AccessRequest by fetching user-group relation mapping from UserStore (which introduced since Ranger 2.3) in RangerBasePlugin. UserStore is loaded and refreshed by base plugin with user-group mapping and user/group attributes and used for policy evaluation. add a switch config for it, disabled by default.

How was this patch tested?

• [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible

• [ ] Add screenshots for manual tests if appropriate

• [x] Run test locally before make a pull request

[codecov-commenter]

Codecov Report

Merging #3308 (636137a) into master (730fd57) will decrease coverage by 0.08%. The diff coverage is 31.81%.

@@             Coverage Diff              @@
##             master    #3308      +/-   ##
============================================
- Coverage     51.91%   51.82%   -0.09%     
  Complexity       13       13              
============================================
  Files           508      521      +13     
  Lines         28996    28785     -211     
  Branches       3982     3850     -132     
============================================
- Hits          15053    14919     -134     
+ Misses        12518    12501      -17     
+ Partials       1425     1365      -60     

Impacted Files	Coverage Δ
...uubi/plugin/spark/authz/ranger/AccessRequest.scala	67.92% <31.81%> (-25.83%)	:arrow_down:
.../kyuubi/server/mysql/constant/MySQLErrorCode.scala	13.84% <0.00%> (-6.16%)	:arrow_down:
...ache/kyuubi/server/mysql/MySQLCommandHandler.scala	77.77% <0.00%> (-4.05%)	:arrow_down:
...uubi/plugin/spark/authz/util/SemanticVersion.scala	50.00% <0.00%> (-3.58%)	:arrow_down:
...ache/kyuubi/server/mysql/MySQLGenericPackets.scala	76.59% <0.00%> (-2.13%)	:arrow_down:
.../org/apache/kyuubi/ha/client/DiscoveryClient.scala	34.78% <0.00%> (-1.59%)	:arrow_down:
...he/kyuubi/plugin/spark/authz/IcebergCommands.scala	96.29% <0.00%> (-1.38%)	:arrow_down:
.../plugin/spark/authz/ranger/RuleAuthorization.scala	80.95% <0.00%> (-0.45%)	:arrow_down:
...in/scala/org/apache/kyuubi/config/KyuubiConf.scala	97.43% <0.00%> (-0.08%)	:arrow_down:
...a/org/apache/kyuubi/ha/client/DiscoveryPaths.scala	100.00% <0.00%> (ø)
... and 26 more

:mega: We’re building smart automated test selection to slash your CI/CD build times. Learn more

[bowenliang123]

cc @yaooqinn Please have a look at this feature. The WIP tag removed and it should be ready for review. Unit test is not added which will not increate much code coverage as it requires methods in Ranger 2.3. (realted to bump of ranger version : https://github.com/apache/incubator-kyuubi/pull/3282) Docs could be enhanced after code review, for showing usage of the switch config.

[bowenliang123]

The logic change is ready and works correctly in practices. But ut is required some more investigation to activate UserStoreRefresher with RangerLocalClient to demostrate sample user groups mapping info.

RangerBasePlugin relies on addUserStoreEnricherIfNeeded and checking anyPolicyDeltaHasUserGroupAttributeExpression to activate UserStoreEnricher. That blocks the demo in ut.

[bowenliang123]

cc @yaooqinn @pan3793 .

This PR passes the CI with ut added over Ranger 2.1+. And I need some help to solve the compatibility for Ranger 2.0 and below.

[yaooqinn]

thanks, merged to master

[bowenliang123]

Remarks with removed unit test, which requires Ranger 2.1 or above, for further reference.

Patching RangerClientHelper trait

  def getUserStoreIfUpdated(
      lastKnownUserStoreVersion: Long,
      lastActivationTimeInMillis: Long): RangerUserStore

Patching RangerLocalClient.scala

  override def getUserStoreIfUpdated(
      lastKnownUserStoreVersion: Long,
      lastActivationTimeInMillis: Long): RangerUserStore = {
    val userGroupsMapping = new util.HashMap[String, util.Set[String]]()
    userGroupsMapping.put("bob", Set("group_a", "group_b").asJava)
    val groupAttrMapping = new util.HashMap[String, util.Map[String, String]]()
    val userAttrMapping = new util.HashMap[String, util.Map[String, String]]()
    val userAttr1 = new util.HashMap[String, String]()
    userAttr1.put("city", "guangzhou")
    userAttrMapping.put("bob", userAttr1)

    val userStore = new RangerUserStore()
    userStore.setUserGroupMapping(userGroupsMapping)
    userStore.setGroupAttrMapping(groupAttrMapping)
    userStore.setUserAttrMapping(userAttrMapping)
    userStore
  }

Adding AccessRequestSuite.scala

package org.apache.kyuubi.plugin.spark.authz.ranger

// scalastyle:off
import org.apache.hadoop.security.UserGroupInformation
import org.scalatest.funsuite.AnyFunSuite

import org.apache.kyuubi.plugin.spark.authz.{ObjectType, OperationType}
import org.apache.kyuubi.plugin.spark.authz.ranger.SparkRangerAdminPlugin.getFilterExpr

class AccessRequestSuite extends AnyFunSuite {
// scalastyle:on
  test("[[KYUUBI #3300]] overriding userGroups with UserStore") {
    SparkRangerAdminPlugin.init()

    val resource1 =
      AccessResource(ObjectType.COLUMN, "my_db_name", "my_table_name", "my_col_1,my_col_2")

    val ugi1 = UserGroupInformation.createRemoteUser("anonymous")
    val art1 = AccessRequest(resource1, ugi1, OperationType.QUERY, AccessType.SELECT)
    assert(art1.getUserGroups.isEmpty)

    val ugi2 = UserGroupInformation.createRemoteUser("bob")
    val art2 = AccessRequest(resource1, ugi2, OperationType.QUERY, AccessType.SELECT)
    assert(!art2.getUserGroups.isEmpty)
    assert(art2.getUserGroups.contains("group_a"))
    assert(art2.getUserGroups.contains("group_b"))

    val are = AccessResource(ObjectType.TABLE, "default", "src_group_row_filter", null)
    val art3 = AccessRequest(are, ugi2, OperationType.QUERY, AccessType.SELECT)
    val maybeString = getFilterExpr(art3)
    assert(maybeString.get === "city == 'guangzhou'")
  }
}

Patching ranger-spark-security.xml

  <property>
    <name>ranger.plugin.spark.use.usergroups.from.userstore.enabled</name>
    <value>true</value>
  </property>

  <property>
    <name>ranger.plugin.spark.enable.implicit.userstore.enricher</name>
    <value>true</value>
  </property>

  <property>
    <name>ranger.plugin.hive.policy.cache.dir</name>
    <value>target/tmp</value>
  </property>

Patching sparkSql_hive_jenkins.json

{
      "service": "hive_jenkins",
      "name": "row_filter_with_user_attr",
      "policyType": 2,
      "policyPriority": 0,
      "description": "",
      "isAuditEnabled": true,
      "resources": {
        "database": {
          "values": [
            "default"
          ],
          "isExcludes": false,
          "isRecursive": false
        },
        "table": {
          "values": [
            "src_group_row_filter"
          ],
          "isExcludes": false,
          "isRecursive": false
        }
      },
      "policyItems": [],
      "denyPolicyItems": [],
      "allowExceptions": [],
      "denyExceptions": [],
      "dataMaskPolicyItems": [],
      "rowFilterPolicyItems": [
        {
          "rowFilterInfo": {
            "filterExpr": "city == '${{USER.city}}'"
          },
          "accesses": [
            {
              "type": "select",
              "isAllowed": true
            }
          ],
          "users": [],
          "groups": ["group_b"],
          "conditions": [],
          "delegateAdmin": false
        }
      ],"id": 33,
      "guid": "e1caa15f-3280-4435-8dbd-f5c664ab9f0f",
      "isEnabled": true,
      "version": 1
    },