incubator-pegasus Feature: Integrate with Apache Ranger

Apache Ranger™ [1] is a framework to enable, monitor and manage comprehensive data security across the Hadoop platform. There are many big data components support to integate with Ranger, like HDFS，HBase，Hive，Yarn，Kafka，Kudu.

Now Pegasus supports Kerberos and built-in ACL, but it's a bit of difficult to manage it, we can make Pegasus interact with Ranger to make it easier for management.

https://ranger.apache.org/

Jul 18 '22 05:07 acelyc111

It is also tracked in Ranger community https://issues.apache.org/jira/browse/RANGER-3831

Jul 18 '22 06:07 acelyc111

If I understand it correctly, the ACL model is so simple now. There are two access_controller class.

meta_access_controller:

Super User is allowed to do anything. All other users are allowed to do the things listed in FLAGS_meta_acl_rpc_allow_list. There is no per-user settings. Default meta_acl_rpc_allow_list are

RPC_CM_LIST_APPS 
RPC_CM_LIST_NODES 
RPC_CM_CLUSTER_INFO 
RPC_CM_QUERY_PARTITION_CONFIG_BY_INDEX

replica_access_controller：

Super User is allowed to do anything. Users in the users list are allowed to do anything. Users are set by 'replica_access_controller.allowed_users' vars in replica::update_ac_allowed_users There seems no per-user settings too.

It seems we first have to enhance the access controller mech, add per-user / per-table support. Such as : { Table1: { user1: read, user2: read+write }, Table2: { user3: read, user4: read+write } }

Jul 18 '22 08:07 kirbyzhou

Add a draft of Ranger Service definition here

Jul 28 '22 10:07 kirbyzhou

The current ACL is described in https://github.com/apache/incubator-pegasus/issues/170 and Pegasus 安全认证

It can be summarized as following:

operation \ user	super user	table owner	other users
query cluster basic info	√	√	√
table read and write	√	√	×
cluster control	√	×	×

The extended ACL is based on the former design and detailed as following:

operation \ details	ACL symbol	rpc code	resource example	--
Global level	--	--	--
query cluster,server	metadata	cluster： RPC_CM_LIST_NODES RPC_CM_CLUSTER_INFO RPC_CM_LIST_APPS（can query all tables） server: RPC_QUERY_DISK_INFO
control cluster,server,multi tables	control	cluster+server： RPC_HTTP_SERVICE(http request，has no principal currently) set LB level: RPC_CM_CONTROL_META recover meta server though replica servers: RPC_CM_START_RECOVERY on replica server: migrate replica between disks: RPC_REPLICA_DISK_MIGRATE on replica server: add new disks: RPC_ADD_NEW_DISK on replica server: detect hot key: RPC_DETECT_HOTKEY cluster+server: remote command(include many operations): RPC_CLI_CLI_CALL (multi-tables)backup policy's add,modify(maybe removed later): RPC_CM_ADD_BACKUP_POLICY RPC_CM_MODIFY_BACKUP_POLICY	--	--
Database level	--	--	--	--
query cluster,server	list	cluster: RPC_CM_LIST_APPS(can only query tables in the database)	--	--
create table	create	RPC_CM_CREATE_APP	db1	can create tables prefixed by "db1_"
drop/recall table	drop	RPC_CM_DROP_APP RPC_CM_RECALL_APP	--	--
manager table - query	metadata	(multi-tables)query backup policy(may be removed later): RPC_CM_QUERY_BACKUP_POLICY (single-table)query backup policy: RPC_CM_QUERY_BACKUP_STATUS (single-table)query backup from policy status: RPC_CM_QUERY_RESTORE_STATUS (single-table)query duplication: RPC_CM_QUERY_DUPLICATION (single-table)query partition split: RPC_CM_QUERY_PARTITION_SPLIT (single-table)query bulk load: RPC_CM_QUERY_BULK_LOAD_STATUS (single-table)query manual compact: RPC_CM_QUERY_MANUAL_COMPACT_STATUS (single-table)query RF of a table: RPC_CM_GET_MAX_REPLICA_COUNT	--	--
manager table - control	control	(single-table)start backup: RPC_CM_START_BACKUP_APP (single-table)control restore from backup: RPC_CM_START_RESTORE (single-table)migrate one replica: RPC_CM_PROPOSE_BALANCER (single-table)add or modify duplication: RPC_CM_ADD_DUPLICATION RPC_CM_MODIFY_DUPLICATION (single-table)update table's envs: RPC_CM_UPDATE_APP_ENV (single-table)DDD diagnose for tables: RPC_CM_DDD_DIAGNOSE (single-table)start and control partition split: RPC_CM_START_PARTITION_SPLIT RPC_CM_CONTROL_PARTITION_SPLIT (single-table)start, clear up and control bulk load: RPC_CM_START_BULK_LOAD RPC_CM_CONTROL_BULK_LOAD RPC_CM_CLEAR_BULK_LOAD (single-table)start manual compact: RPC_CM_START_MANUAL_COMPACT (single-table)update table's RF: RPC_CM_SET_MAX_REPLICA_COUNT	--	--
Database/Table level	--	--	--	--
Read data	read	meta server: route info: RPC_CM_QUERY_PARTITION_CONFIG_BY_INDEX list: RPC_CM_LIST_APPS(can only query privileged tables) replica server: server level: replica_stub::on_client_read replica level: replica::on_client_read	db1/table1	can read table db1.table1
Write data	write	replica server: server level: replica_stub::on_client_write replica level: replica::on_client_write	db1/*	can write tables prefixed by 'db1'
server internal (not in ACL)	N/A	RPC_CM_CONFIG_SYNC RPC_CM_UPDATE_PARTITION_CONFIGURATION RPC_CM_REPORT_RESTORE_STATUS RPC_CM_DUPLICATION_SYNC RPC_CM_REGISTER_CHILD_REPLICA RPC_CM_NOTIFY_STOP_SPLIT RPC_CM_QUERY_CHILD_STATE RPC_NEGOTIATION RPC_CALL_RAW_MESSAGE RPC_CALL_RAW_SESSION_DISCONNECT RPC_NFS_GET_FILE_SIZE RPC_NFS_COPY RPC_FD_FAILURE_DETECTOR_PING RPC_CALL_RAW_MESSAGE RPC_CALL_RAW_SESSION_DISCONNECT RPC_CONFIG_PROPOSAL RPC_GROUP_CHECK RPC_QUERY_PN_DECREE RPC_QUERY_REPLICA_INFO RPC_QUERY_LAST_CHECKPOINT_INFO RPC_PREPARE RPC_GROUP_CHECK RPC_QUERY_APP_INFO RPC_LEARN RPC_LEARN_COMPLETION_NOTIFY RPC_LEARN_ADD_LEARNER RPC_REMOVE_REPLICARPC_COLD_BACKUP RPC_CLEAR_COLD_BACKUP RPC_SPLIT_NOTIFY_CATCH_UP RPC_SPLIT_UPDATE_CHILD_PARTITION_COUNT RPC_BULK_LOADRPC_GROUP_BULK_LOAD	--	--

Aug 15 '22 09:08 acelyc111

More restrict for common query type requests. a. As table above, users should be granted 'metadata' before query cluster info.
MetaServer have to support table level ACL a. For example, query route info of a table (i.e. RPC_CM_QUERY_PARTITION_CONFIG_BY_INDEX) must be in ACL
To support 'database' level ACL: a. on MetaServer: i. Handle requests on MetaServer, they are listed in the table above, 'Database level.list/create/drop/metadata/control'. ii. Add unordered_map<'table_prefix', unordered_set<user_name>> structure. iii. Parse table name from request messages when handle requests. a. on ReplicaServer: i. Add unordered_map<'table_prefix', unordered_set<user_name>> structure too. ii. Because "table prefix" string doesn't belong to any tables/replica, so besides replica envs, we have to add an extra server level envs for ACL. iii. We have to maintainance the relationship between table id and table name carefully. iv. Parse table id and transfer it to table name from request messages when handle requests.
Implemention a. The leader meta_server request ACL details from Apache Ranger though HTTP periodically. b. Parse the JSON formatted response to internal required structure. c. Set the structure to MetaServer and remote Zookeeper. d. Set to each tables. e. Send envs info to ReplciaServers though CconfigSync RPC.
The relationship between table name and database name: a. Use che '.' to split table name, suppose the part before '.' is database name. b. When bootstrap, for the tables already created, if they are not match the new naming rule, consider they are in the "default" database. c. When ACL enabled, it's not allowed to create table with name which not matched the rule. d. For the rename operation, it's not allowed to modify the prefix.

Aug 15 '22 10:08 acelyc111

需更严格的普通查看类请求： a. 如上表格，“Global级别权限”的“metadata”权限也需要ACL
meta server上增加表级的ACL： a. 如获取表的路由信息（RPC_CM_QUERY_PARTITION_CONFIG_BY_INDEX），也需做ACL
为支持database（前缀）粒度的控制： a. meta server上的： i. 针对如上列表中的“Database级别权限”的在metaserver上处理的请求 ii. 需增加unordered_map<表前缀, unordered_set<用户名>>的结构 iii. 在各个请求响应函数中，解析到请求的”表名“之后加以判断 replica server上的： i. 需增加与meta server上相似的数据结构（unordered_map<表前缀, unordered_set<用户名>>） ii. 因为”表前缀”并不属于任何表，所以除了表级的envs外，还需增加server级的envs，用于鉴权 iii. 因为在replica server上并不维护“表名”，而只有“表id”，所以还需增加表id与表名的映射关系 iv. 在各个请求处理函数中，解析到请求的”表id“之后，映射为表名，再加以判断
实现： a. leader meta_server定期地通过http请求从ranger获取pegasus的ACL b. 解析Json格式的ACL成需要的数据结构 c. 设置到meta server自身的acl结构中，也存储到远端存储（即zookeeper） d. 设置到各个表的envs中 e. 后续通过meta server → replica server的同步，将envs下发到各个replica server上
表名与database名的映射关系： a. 通过符号“.”来划分database名，“.”之前的即为database name b. 启动时，对于已创建的表，如果不符合分割方式，则他处于“default” database中 c. 开启ACL后，再创建不符合分割规则的表则报错 d. 对于rename操作，不允许修改前缀

Aug 15 '22 11:08 acelyc111

We have finished the service definition in ranger according to this draft. See https://issues.apache.org/jira/browse/RANGER-3831

Aug 17 '22 09:08 kirbyzhou

Introduce the implementation and how to use:

The class diagram

First, you need to add ACL related configurations. The client configuration has not changed, the server has added new configurations:

enable_ranger_acl: indicates whether to use ranger for acl
ranger_service_url： ranger server url
ranger_service_name: use ranger policy name
mandatory_enable_acl: mandatory use range policy, currently used for testing

The details are as follows:

server
[security]
  update_ranger_policy_interval_sec
[ranger]
  ranger_service_url
  ranger_service_name
  ranger_legacy_table_database_mapping_rule
  mandatory_enable_acl
[security]
  enable_auth = true
  krb5_keytab = /root/apache/pegasus.keytab
  krb5_config = /etc/krb5.conf
  krb5_principal = XXXXX
  sasl_plugin_path = /root/apache/incubator-pegasus/thirdparty/output/lib/sasl2
  service_fqdn = XXXXX
  service_name = XXXXX
  mandatory_auth = true
  enable_acl = true
  super_users =
  meta_acl_rpc_allow_list =
  enable_ranger_acl = true

java client
java
meta_servers = 127.0.0.1:34601,127.0.0.1:34602,127.0.0.1:34603
operation_timeout = 5000
async_workers = 4
enable_perf_counter = false
perf_counter_tags = cluster=onebox,app=unit_test
push_counter_interval_secs = 10
meta_query_timeout = 5000
auth_protocol = kerberos
kerberos_service_name = XXXXX
kerberos_service_fqdn = XXXXX
kerberos_keytab = /root/apache/pegasus.keytab
kerberos_principal = XXXXX

shell
[security]
  enable_auth = true
  krb5_keytab = /root/apache/pegasus.keytab
  krb5_config = /etc/krb5.conf
  krb5_principal = XXXXX
  sasl_plugin_path = /root/apache/incubator-pegasus/thirdparty/output/lib/sasl2
  service_fqdn = XXXXX
  service_name = XXXXX

Second compatibility: Retained the old ACL mode

Use the old ACL

enable_acl = true
enable_ranger_acl = false

user ranger for ACL

enable_acl = true
enable_ranger_acl = true

Third, define the ranger policy

Pegasus resources can be divided into multiple types, and the operation types of each resource can also be divided. One operation type corresponds to one ACL symbol
ACLs on each type of resource correspond to specific rpc_code

pegasus+ranger

After completing the integration of ranger with pegasus, you can set permissions on the ranger web page according to your own needs

Feb 22 '23 03:02 WHBANG

See https://issues.apache.org/jira/browse/RANGER-3831 You have to import the service definition into RANGER at first.

Using the REST API of ranger:

curl -X 'POST'
'http://rangerhost:6080/service/public/v2/api/servicedef' \ -H 'Accept: application/json' \ -H 'Content-Type: application/json'
-u 'admin:adminpassword'
-d @.***'

WHBANG @.***> 于2023年2月22日周三 11:45写道：

Introduce the implementation and how to use:

The class diagram [image: image] https://user-images.githubusercontent.com/38547944/218993763-91a7072b-086b-44fa-9041-636c5ea1d089.png

First, you need to add ACL related configurations. The client configuration has not changed, the server has added new configurations:

enable_ranger_acl: indicates whether to use ranger for acl

ranger_service_url： ranger server url

ranger_service_name: use ranger policy name

mandatory_enable_acl: mandatory use range policy, currently used for testing

The details are as follows:

server

[security]

update_ranger_policy_interval_sec

[ranger]

ranger_service_url

ranger_service_name

ranger_legacy_table_database_mapping_rule

mandatory_enable_acl

[security]

enable_auth = true

krb5_keytab = /root/apache/pegasus.keytab

krb5_config = /etc/krb5.conf

krb5_principal = XXXXX

sasl_plugin_path = /root/apache/incubator-pegasus/thirdparty/output/lib/sasl2

service_fqdn = XXXXX

service_name = XXXXX

mandatory_auth = true

enable_acl = true

super_users =

meta_acl_rpc_allow_list =

enable_ranger_acl = true

java client

java

meta_servers = 127.0.0.1:34601,127.0.0.1:34602,127.0.0.1:34603

operation_timeout = 5000

async_workers = 4

enable_perf_counter = false

perf_counter_tags = cluster=onebox,app=unit_test

push_counter_interval_secs = 10

meta_query_timeout = 5000

auth_protocol = kerberos

kerberos_service_name = XXXXX

kerberos_service_fqdn = XXXXX

kerberos_keytab = /root/apache/pegasus.keytab

kerberos_principal = XXXXX

shell

[security]

enable_auth = true

krb5_keytab = /root/apache/pegasus.keytab

krb5_config = /etc/krb5.conf

krb5_principal = XXXXX

sasl_plugin_path = /root/apache/incubator-pegasus/thirdparty/output/lib/sasl2

service_fqdn = XXXXX

service_name = XXXXX

Second compatibility: Retained the old ACL mode

Use the old ACL

enable_acl = true

enable_ranger_acl = false

user ranger for ACL

enable_acl = true

enable_ranger_acl = true

Third, define the ranger policy

Pegasus resources can be divided into multiple types, and the operation types of each resource can also be divided. One operation type corresponds to one ACL symbol

[image: image] https://user-images.githubusercontent.com/38547944/204430376-17e8ae9a-bdac-466c-a2a7-d4f27cae09e0.png

ACLs on each type of resource correspond to specific rpc_code

[image: image] https://user-images.githubusercontent.com/38547944/204430797-5b45e079-84c5-4f3e-a470-db063d6d87d5.png

[image: image] https://user-images.githubusercontent.com/38547944/204430865-ffc16771-90ca-43c7-903a-5451cb58c86c.png

[image: image] https://user-images.githubusercontent.com/38547944/204430917-febf6ba1-eddc-4db0-b428-a00e7e651828.png

pegasus+ranger

After completing the integration of ranger with pegasus, you can set permissions on the ranger web page according to your own needs [image: image] https://user-images.githubusercontent.com/38547944/204432012-230db07c-47d0-4960-9c10-04340c19b2ff.png [image: image] https://user-images.githubusercontent.com/38547944/204432186-344267c5-d966-4b21-99f5-cc6d4ade55db.png

— Reply to this email directly, view it on GitHub https://github.com/apache/incubator-pegasus/issues/1054#issuecomment-1439398491, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAZ6PHW5QXYPU2GTDNQM55LWYWDVRANCNFSM533BJVLA . You are receiving this because you commented.Message ID: @.***>

Feb 22 '23 08:02 kirbyzhou