SOLR-15857: Add AWS Secret Manager support for ZK ACL credentials

[laminelam]

https://issues.apache.org/jira/browse/SOLR-17020

Description

Solr currently connects to Zookeeper and manages ACLs using a set of credentials (when using Digest scheme). These credentials are either passed via the command line (system properties) or fetched from an unencrypted file on every host in the cluster. Clearly, this approach lacks sufficient security.

The proposed change of this contribution intends to integrate AWS Secret Manager for credential storage, offering two primary advantages: • Improved security measures. • Streamlined operations by centralizing credential storage, eliminating the need to update multiple hosts/files whenever passwords are changed.

Solution

This is a new module to support storing Zookeeper credentials in an AWS Secret Manager.

Added a new implementation of ZkCredentialsInjector that pulls the creds from AWS SM.

Ref guide in progress...

Tests

Added test cases. Integration tests also made in AWS environment.

Checklist

Please review the following and check all that apply:

• [x] I have reviewed the guidelines for How to Contribute and my code conforms to the standards described there to the best of my ability.
• [x] I have created a Jira issue and added the issue ID to my pull request title.
• [x] I have given Solr maintainers access to contribute to my PR branch. (optional but recommended)
• [x] I have developed this patch against the main branch.
• [x] I have run ./gradlew check.
• [x] I have added tests for my changes.
• [ ] I have added documentation for the Reference Guide

[github-actions[bot]]

This PR had no visible activity in the past 60 days, labeling it as stale. Any new activity will remove the stale label. To attract more reviewers, please tag someone or notify the [email protected] mailing list. Thank you for your contribution!

[janhoy]

Are we sure this fits as a Solr module? Since this is client-side solrj code, it could be in e.g. solrj-aws-secret instead?

[laminelam]

Are we sure this fits as a Solr module? Since this is client-side solrj code, it could be in e.g. solrj-aws-secret instead?

Actually, this is a server side code. When Solr starts, it reads ZK creds from a local (clear) text file and uses them to connect to ZK. With this feature, Solr would get the ZK creds from an AWS Secret Manager, and then proceed to connect to ZK.

Now, from SolrJ side we have 3 options:

• Use the existing mechanism. Get the ZK creds and set them in System Props using the standard way.
• The client can connect to AWS SM to get the creds before passing them to System Props.
• This module can be used to connect directly to AWS SM and inject the creds into SolrJ. Though, the libs have to be added to class path.

    System.setProperty("zkACLProvider", "org.apache.solr.common.cloud.DigestZkACLProvider");
    System.setProperty("zkCredentialsProvider", "org.apache.solr.common.cloud.DigestZkCredentialsProvider");
    System.setProperty("zkCredentialsInjector", "org.apache.solr.secret.zk.AWSSecretManagerCredentialsInjector");
    System.setProperty("zkCredentialsAWSSecretName", "myZkSecret");
    System.setProperty("zkCredentialsAWSRegion", "us-east-1");

    CloudSolrClient client = new CloudHttp2SolrClient.Builder(zkHosts)...

Somewhere down the line SolrZkClient will instantiate an AWSSecretManagerCredentialsInjector.

[github-actions[bot]]

This PR had no visible activity in the past 60 days, labeling it as stale. Any new activity will remove the stale label. To attract more reviewers, please tag someone or notify the [email protected] mailing list. Thank you for your contribution!