RANGER-4792: Fix issue with creating index and import data in ElasticSearch as Audit database

[ognjen-it]

What changes were proposed in this pull request?

I tried to create a user on Apache Jira but I've not received confirmation yet. Until that, I'm opening this pull request for all whos seeing problems with audit. I'm not sure if I need to do the same on the master branch?

1. Lucene version - fixed problem with writing data to ElasticSearch

Error: java.lang.NoSuchFieldError: LUCENE_8_5_1

I tried to change minor version one by one, but only latest version fit for me.

Changes:

• agents-audit/pom.xml: 311
• pom.xml: 241

2. Elastic search protocol - fixed problem with changing protocol

Even though I changed ranger.audit.elasticsearch.protocol from http to https, audit plugin still using http protocol.

Changes:

• security-admin/scripts/ranger-admin-site-template.xml: 167-170
• security-admin/scripts/setup.sh: 79, 794-797
• security-admin/scripts/upgrade_admin.py: 116
• security-admin/src/main/resources/conf.dist/ranger-admin-site.xml: 53-57
• security-admin/src/test/java/org/apache/ranger/elasticsearch/ElasticSearchAccessAuditsServiceTest.java: 56

3. Audit plugin - cannot write audit to ES

Error: bootstrap method initialization exception

After changing the version of httpcomponents I started seeing audit

Changes:

• pom.xml: 137, 138, 140

4. Ranger admin console - Audit show 1-1-1970

Erro: Error converting value to date. Value = 2024-05-13T13:08:47.905Z

Even though evtTime was ok in ElasticSearch, ranger couldn't show it on GUI.

Changes:

• security-admin/src/main/java/org/apache/ranger/elasticsearch/ElasticSearchAccessAuditsService.java: 260
• security-admin/src/main/java/org/apache/ranger/solr/SolrAccessAuditsService.java: 239

How was this patch tested?

I tested it manually.

ElasticSearch:

Ranger:

[FerArribas14]

Hi,

I have built the project from the ranger-2.4 branch of the ognjen-it/ranger repository but it doesn’t write in ElasticSearch.

What version of elasticsearch are you testing on localhost:9200? I can't get it to write audit events to elasticsearch. There are no errors in the ranger admin logs and the index has been created correctly. Also, there is connectivity between ranger-admin container and elasticseatch.

I have everything deployed in docker. I have created an elasticsearch service in docker compose.

Thanks,

[ognjen-it]

Hi @FerArribas14,

The version of ElasticSearch is 7.10.2 (latest with Apache 2.0 license). Everything looks good on your screenshots, ranger-admin successfully created the index and now some plugins need to start writing data.

EDIT: Just to add that if you try to run ranger-admin from the Apache repo, you won't see the ranger_audits index. In logs you would see Initializing ElasticSearch but on GUI would see: Error loading audit logs!! And if you take a look in ews/logs/catalina.out file, you will see something like: java.lang.NoSuchFieldError: LUCENE_8_5_1

[FerArribas14]

Thank you @ognjen-it. I had a problem with the configuration of a plugin. Now it works correctly :)

[ognjen-it]

Closing since #319 has been merged.