AMBARI-25929: Add ranger 2.4 support in ambari bigtop stack

[JiaLiangC]

What changes were proposed in this pull request?

Main logic and changes of the commit:

Desc:

Ambari Ranger adaptation Ambari 2.8 uses bigtopstack 3.2.0, which corresponds to apache bigtop version 3.2.0

1.All components lack corresponding ranger property files. Therefore, all ranger-related configuration files for all components are added.

2.There is a lack of ranger integration, and here we integrate the ranger service. This ranger service is ported from hdp3.1.5's ranger service. The difference is that some initialization content in the install method is referenced to the /usr/bigtop/current directory, but this directory will only be linked with the "bigtop-select distro-select set component version" command in the post-hook after the install method is completed. Therefore, the initialization logic in the install method is moved to the configure method, and a mark file is set. Once the initialization is completed, a mark file is generated, and subsequent initialization will not be performed if the mark file exists.

3. Bug fix for ranger advisor not working After integrating Ranger, most of the component advisors were not functioning properly due to the presence of ambari/ambari-server/src/main/resources/stacks/BIGTOP/3.2.0/services/stack_advisor.py in the 2.8.0 branch. The working logic of stack_advisor is such that ambari/ambari-server/src/main/resources/scripts/stack_advisor.py, as the starting point, loads all advisors, including stack_advisor.py and service_advisor.py under ambari/ambari-server/src/main/resources/stacks directory. service_advisor.py loads all service_advisor.py under each component directory and retrieves the recommended configuration, such as ambari/ambari-server/src/main/resources/stacks/BIGTOP/3.2.0/services/HDFS/service_advisor.py, which contains all recommended HDFS configurations. However, in the 2.8.0 branch, ambari/ambari-server/src/main/resources/stacks/BIGTOP/3.2.0/services/stack_advisor.py hardcodes to only return a few service advisors, causing all other component service advisors to not work properly. Furthermore, ambari/ambari-server/src/main/resources/stacks/BIGTOP/3.2.0/services/stack_advisor.py is redundant, and the configurations are scattered in each service advisor. Therefore, removing this file fixed the issue. The fix can be found in this PR: https://github.com/apache/ambari/pull/3677.

How was this patch tested?

manual test before enable kerberos all compoent works smoonthly

after enable kerberos all compoent works smoonthly

Please review Ambari Contributing Guide before opening a pull request.

[vishalsuvagia]

/recheck

[virajjasani]

@vishalsuvagia do you think this is good to go?

[JiaLiangC]

@virajjasani @vishalsuvagia https://issues.apache.org/jira/browse/AMBARI-25929

Adding support for Ranger in Ambari is a rather complicated change. It involves packaging Ranger in Apache Bigtop, integration of Ambari Ranger service itself, integration with Ambari Infra, and fixing bugs in Ambari Advisor, since installing Ambari Ranger requires Advisor to provide a lot of recommended configurations. Therefore, I have organized all related pull requests (PRs), and the progress of these PRs can be found in the description of this issue.

Currently, all Ranger-related PRs on the Bigtop side have been reviewed and merged. As for the three bugs in Apache Ranger, they have been patched and placed in the resource package of Bigtop because there is currently no one reviewing them in the Apache Ranger community. Therefore, only the Ambari-related PRs need to be reviewed for Ranger support.

Thus, to support Ranger, we need to review and merge the Advisor-related PRs and the PRs for Ambari Infra integration first, and then review the PR for Ranger support. During my testing, I have locally reviewed and merged all the PRs mentioned in the issue description and conducted rigorous testing, and so far, no issues have been found.

[virajjasani]

@JiaLiangC thank you for your efforts! This is fantastic!

[virajjasani]

As of now, we need this to be merged first correct? https://github.com/apache/ambari/pull/3696

[JiaLiangC]

As of now, we need this to be merged first correct? #3696

@virajjasani I asked the community's PMC kevinw66, and currently, the Ranger integration will not be included in the 2.8.0 release but will be released in subsequent versions after the 2.8.0 release.

First, we need to review and merge several Advisor-related PRs, such as https://github.com/apache/ambari/pull/3677, because Ranger relies heavily on the configuration provided by Advisor. Otherwise, users will need to manually configure a large number of settings to make the Ranger plugin of each component work properly.

Secondly, we need to review and merge the PR for Ambari Infra because Ranger depends on the ambari-infra-client to perform some automation settings in Solr, such as creating a Ranger collection in Solr.

[JiaLiangC]

@virajjasani since Advisor-related PRs already merged, Could you kindly help review these ranger related PR, please?

https://github.com/apache/ambari-infra/pull/48 (infra solr upgrade) https://github.com/apache/ambari/pull/3696 (ambari infra support)

[virajjasani]

recent commit caused merge conflict for this PR

[JiaLiangC]

@virajjasani conflicts resolved

[smallyao]

@JiaLiangC @timyuer @virajjasani Should we move Ranger 2.4 to Bigtop 3.3.0 stack instead of 3.2.0? Bigtop 3.2.0 doesn't include Ranger.

[JiaLiangC]

@smallyao Thank you very much for your suggestion. You are right, https://issues.apache.org/jira/browse/BIGTOP-3909 is the release plan for Bigtop, and Ranger 2.4 will be supported in Bigtop 3.3. I will add a new directory under Bigtop stacks and move the Ranger service there. The downside is that Ambari releases are too slow, so it will be a long wait for users who want to use Ranger in the future.

[JiaLiangC]

@smallyao @virajjasani already moved ranger2.4 service into bigtop 3.3.0 stacks, and updated ranger support version in stack_features

[smallyao]

Hi @JiaLiangC. Ranger 2.4 uses logback instead of log4j.

[JiaLiangC]

log4j It was my oversight. I checked the default configuration of Ranger 2.4 and found that it was using the logback configuration. I will fix this and test it.

[JiaLiangC]

@smallyao The logback issue has been fixed, and I have also moved the Ranger 2.4 service to the Bigtop 3.2.0 directory for installation testing. The logback configuration is now working correctly.

[JiaLiangC]

@virajjasani All the issues discovered in Ranger have been fixed. Can you please help review this PR?

[virajjasani]

Thanks @JiaLiangC, seems like we are ready to go.

@smallyao are you fine with the latest changes?

[smallyao]

Thanks @JiaLiangC, seems like we are ready to go.

@smallyao are you fine with the latest changes?

Would it be better if we could finish other services support in bigtop stack 3.3.0 before continuing this work?

[JiaLiangC]

@smallyao @virajjasani I agree, as Ranger now supports moving to the directory of Bigtop stack 3.3, it is inconvenient to test. This PR can indeed wait until Ambari supports testing with Bigtop stack 3.3.

[virajjasani]

sounds good, thanks

[haoranchuixue]

Hello, I want to use Ranger on Ambari 2.8.0.

Whether it can be used directly

Use this directly "https://github.com/JiaLiangC/ambari/tree/AMBARI-25929"

Build Ambari

[JiaLiangC]

@feixue Ambari 2.8.0 supports Ranger steps:

1. Use the Ambari master code and merge the PR https://github.com/apache/ambari/pull/3697.
2. Move the directory ambari/ambari-server/src/main/resources/stacks/BIGTOP/3.3.0/services/RANGER to ambari/ambari-server/src/main/resources/stacks/BIGTOP/3.2.0/services/.
3. Modify ambari/ambari-server/src/main/resources/stacks/BIGTOP/3.2.0/properties/stack_features.json and replace all occurrences of "3.3.0" with "3.2.0".

Packaging: Use the Bigtop master branch and directly package the Ranger component.

[haoranchuixue]

@feixue Ambari 2.8.0 支持 Ranger 步骤：

1. 使用 Ambari 主代码并合并 PR AMBARI-25929：在 ambari bigtop 堆栈中添加 ranger 2.4 支持 #3697。
2. 将目录移动ambari/ambari-server/src/main/resources/stacks/BIGTOP/3.3.0/services/RANGER到ambari/ambari-server/src/main/resources/stacks/BIGTOP/3.2.0/services/.
3. 修改ambari/ambari-server/src/main/resources/stacks/BIGTOP/3.2.0/properties/stack_features.json所有出现的“3.3.0”并将其替换为“3.2.0”。

打包： 使用Bigtop master分支，直接打包Ranger组件。

Hello, Ambari didn't quite understand that step. Do you build Ambari based on this Master branch after incorporating PR and so on using the Ambari Master code?

How about throwing the ranger part into the service of ambari 2.8.0?

[smallyao]

Hi @JiaLiangC Could you continue this work based on #3718 ？

[JiaLiangC]

Hi @JiaLiangC Could you continue this work based on #3718 ？

ok, i will work on it.

[arjun-guptha]

Hi team, I successfully built Ranger from the Ambari trunk branch and followed the steps outlined below:

1.Moved the directory ambari/ambari-server/src/main/resources/stacks/BIGTOP/3.3.0/services/RANGER to ambari/ambari-server/src/main/resources/stacks/BIGTOP/3.2.0/services/. 2.Modified ambari/ambari-server/src/main/resources/stacks/BIGTOP/3.3.0/properties/stack_features.json by replacing all instances of "3.3.0" with "3.2.0".

I successfully added the Ranger service to Bigtop. However, I encountered an issue when attempting to enable HDFS plugins within Ranger. The changes are not taking effect in Ranger. HDFS plugins are not enabling in Ranger. As a result, I'm wondering if there are any specific configurations that need to be added or modified to address this issue. Additionally, I'm curious if there are any Git merges that need to be performed in order to successfully build the Ranger service.

Can anyone please help on this

[JiaLiangC]

@arjun-guptha restart hdfs and try again

[aghazeh]

do you checked amabri 2.8 with ranger 2.4 supports on el8 ?

[arjun-guptha]

Hello team,

I successfully built and tested Ranger in Ambari, and it worked as expected. Now, I'm attempting to enable Kerberos in Ambari. Prior to enabling Kerberos, I executed the "ambari-server setup-security" command for encryption. I've included a screenshot of the procedure below.

However, after running that command, the Ranger service is not starting. I'm encountering an error message that states: "ValueError: ciphertext block must be 16 bytes." I've attached a screenshot with detailed logs below. Screenshot of ambari-server setup-security Screenshot of Error in Ranger Service

Thanks in Advance

[amitkr146]

@JiaLiangC / @arjun-guptha ,

I am also trying to integrate ranger with ambari 2.8 . can you please guide me with steps either to use Apache Ambari or https://github.com/JiaLiangC/ambari/tree/AMBARI-25929

meanwhile i tried copying the RANGER service from https://github.com/JiaLiangC/ambari/tree/AMBARI-25929/ambari-server/src/main/resources/stacks/BIGTOP/3.3.0/services/RANGER

and placed it directly on installed ambari 2.8.0 ,i was able to see ranger in add service but then it was not able to identify the rpm package build with bigtop_3.2.1 . as it was expecting package name to be ranger_3_2_1-admin but ranger rpm build with bigtop was showing ranger-admin . so i changed package name using rpmbuild ,after this i was able to install ranger using Ambari 2.8 ,but it was not configuring plugins properly ,so finall not working .

can you please guide me how to proceed : which Ambari to build Do i need to build ranger rpm again using Apache bigtop 3.2.1 ?

Thanks in Advance .