SOLR-15857: Add Secret Manager support for ZK ACL credentials

[laminelam]

https://issues.apache.org/jira/browse/SOLR-15857

Note: Attached a PDF copy of the ref guide in the JIRA issue. Rewrote significant part of the Zookeeper access control page with several usage examples.

Problem

Solr uses a list of credentials to connect to Zookeeper and to handle ACLs.

• 1- In the current implementation, the credentials are passed through the command line (system props) or read from a clear text file stored in all the cluster's hosts. Needless to say this is not safe enough.

• 2- On the other hand, the same code to load the credentials is called twice, first by ZkCredentialsProvider to connect to Zookeeper and a second time by ZkACLProvider to create the ACLs. The code is also duplicated, although it's only reading from system props.

  Following the current logic, adding a custom pair of ZkCredentialsProvider/ZkACLProvider to load the credentials from another source (ex a Secret Manager) would as well require duplicating the code and making repetitive calls to extract the same credentials.

• 3- There is no way to customize how credentials are passed without recompiling.

Proposed solution

Let’s start with problem 2).

Problem 2

• Refactor the way how the credentials are injected by passing them as a dependency. One code, called once and injected into the client class. Here the client classes are ZkCredentialsProvider and ZkACLProvider.

• Favor composition over inheritance to inject custom credentials loaders without changing the composing (container) class.

• Add a third interface ZkCredentialsInjector whose implementations load ZK credentials from a credentials source to be injected into ZkCredentialsProvider and ZkACLProvider

• The workflow is: Credentials source —> ZkCredentialsInjector —> ZkCredentialsProvider/ZkACLProvider —> Zookeeper

• The ZkCredentialsInjector gets the creds from an external source which get injected into zkCredentialsProvider and zkACLProvider. The "external source" here can be system props, a file, a Secret Manager, or any other local or remote source.

public interface ZkCredentialsInjector { 
    List<ZkCredential> getZkCredentials();
    ...
}

• Any class implementing ZkCredentialsInjector can be injected via system props in solr.ini.sh/cmd.

In the below example VMParamsZkCredentialsInjector is injected. Note: VMParamsAllAndReadonlyDigestZkACLProvider and VMParamsSingleSetCredentialsDigestZkCredentialsProvider would be deprecated and replaced with a combination of DigestZkACLProvider/DigestZkCredentialsProvider and VMParamsZkCredentialsInjector.

  SOLR_ZK_CREDS_AND_ACLS=“
     -DzkACLProvider=org.apache.solr.common.cloud.acl.DigestZkACLProvider \
     -DzkCredentialsProvider=org.apache.solr.common.cloud.acl.DigestZkCredentialsProvider \
     -DzkCredentialsInjector=org.apache.solr.common.cloud.acl.VMParamsZkCredentialsInjector \
     -DzkDigestUsername=admin-user -DzkDigestPassword=CHANGEME-ADMIN-PASSWORD \
     -DzkDigestReadonlyUsername=readonly-user -DzkDigestReadonlyPassword=CHANGEME-READONLY-PASSWORD"
 SOLR_OPTS="$SOLR_OPTS $SOLR_ZK_CREDS_AND_ACLS"

• Add DigestZkACLProvider/DigestZkCredentialsProvider classes to support Digest based scheme ZK authentication/authorization

Class DigestZkACLProvider implements ZkACLProvider{
	CredentialsInjector credentialsInjector;
	...
}

Class DigestZkCredentialsProvider implements ZkCredentialsProvider{
	CredentialsInjector credentialsInjector;
	...
}

This concept can be generalized to non-digest schemes (a kind of Strategy pattern) but that would require more refactoring, it can be achieved in a future contribution if this one is accepted.

Now apply this new feature and add a custom injector to solve problem 1).

Problem 1

• Store the credentials in a Secret Manager to have Solr pull them out at startup.
• Add SecretCredentialInjector class that contains a dependency interface (SecretCredentialsProvider) whose implementation pulls zk credentials from a Secret Manager and delegate the getZkCredentials call.

public class SecretCredentialInjector implements ZkCredentialsInjector {
    ...
    private SecretCredentialsProvider secretCredentialProvider;
    
    public List<ZkCredential> getZkCredentials() {
        ...
        return secretCredentialProvider.getZkCredentials(secretName);
    }
    ...
}

• In this contribution the offered implementating class is AWSSecretCredentialsProvider that gets zk credentials from AWS Secret Manager. To support any other Secret Manager provider all you need to do is add a class implementing SecretCredentialsProvider and pass it through system props (-DzkSecretCredentialsProvider)

SOLR_ZK_CREDS_AND_ACLS="-DzkACLProvider=org.apache.solr.common.cloud.acl.DigestZkACLProvider \
  -DzkCredentialsProvider=org.apache.solr.common.cloud.acl.DigestZkCredentialsProvider \
  -DzkCredentialsInjector=org.apache.solr.common.cloud.acl.SecretCredentialInjector \
      -DzkSecretCredentialsProvider=org.apache.solr.secret.zk.AWSSecretCredentialsProvider \
      -DzkSecretCredentialSecretName=zkSecret \
      -DzkCredentialsAWSSecretRegion=us-west-2"
SOLR_OPTS="$SOLR_OPTS $SOLR_ZK_CREDS_AND_ACLS"

Problem 3 A new contrib module (aws-secret-provider) is added where SecretCredentialsProvider implementing classes can be added without the need to add a new dependency to Solr core. All one needs to do after adding a new class is to pass it through system props via solr.ini.sh/cmd file. This module can be used in the future for other secrets injections, not specifically related to zk.

Thank you in advance for your time and your comments.

Checklist

Please review the following and check all that apply:

• [x] I have reviewed the guidelines for How to Contribute and my code conforms to the standards described there to the best of my ability.
• [x] I have created a Jira issue and added the issue ID to my pull request title.
• [x] I have given Solr maintainers access to contribute to my PR branch. (optional but recommended)
• [x] I have developed this patch against the main branch.
• [x] I have run ./gradlew check.
• [x] I have added tests for my changes.
• [x] I have added documentation for the Reference Guide