apache
Support JWT authentication
The Operator already has support for configuring Solr with BasicAuth, and the Operator is given BasicAuth credentials to be able to access /solr/admin/info/system, /solr/admin/collections and /solr/admin/{backup,restore}.
The operator should also work when JWTAuthPlugin is used for the cluster. The Operator will then need to obtain and use a JWT token as an Authorization: Bearer xxxx header for all requests to Solr endpoints, analogous to how it uses Authorization: Basic xxxx today.
What I propose is
- Operator can be configured to setup
security.jsonfor JWT - Operator can be configured to obtain a JWT token from an OIDC server to talk to Solr
The security.json is not that different from the BasicAuth one, something like:
{
"authentication": {
"blockUnknown": false,
"class": "solr.JWTAuthPlugin",
"redirectUris": "https://my.solr.server:8983/solr/,https://my.other.solr.server:8983/solr/",
"rolesClaim": "roles",
"issuers": [
{
"wellKnownUrl": "https://idp.example.com/.well-known/openid-configuration",
"clientId": "<MY_CLIENT_ID>",
}
]
},
"authorization": {
"class": "solr.ExternalRoleRuleBasedAuthorizationPlugin",
"permissions": [ ... ]
}
}
The permissions will mostly be the same, and the mapping from users to their roles will happen in the OIDC server, so we'll not care about usernames in security.json, just roles. To generate this, we need some more config values to operator:
spec:
...
solrSecurity:
authenticationType: JWT
jwt:
wellKnownUrl: <url>
solrClientId: <solr client-id as registered with OIDC>
rolesClaim: <jwt claim key where role name is stored>
oper-role: k8s
admin-role: admin
operClientId: <operator client-id as registered with OIDC>
operClientSecretName: <name of k8s secret where operator's client secret for OIDC is stored>
The user will ahead of time register Solr and SolrOperator with OIDC server to obtain client-ID and secret. The Operator will generate and provision security.json and connect to OIDC's token endpoint to obtain a JWT token for Solr.
The user should probably also be able to provision security.json manually, and in that case, only wellKnownUrl, operClientId and operClientSecretName would need to be configured. If users have more advanced JWT config needs than the basics above, such as multiple issuers, then a manual approach is better than bloating the operator with every single option.
Rather than polluting the SolrCloud & Prometheus Exporter CRDs with OIDC config settings, the operator could parse out the wellKnownUrl and other config from a security.json provided by the user in a ConfigMap? So then the CRD structure could look like:
spec:
...
solrSecurity:
authenticationType: OIDC
configMap: <user-supplied config map here with a security.json key>
oidc:
clientId: <operator client-id as registered with OIDC>
clientSecretName: <name of k8s secret where operator's client secret for OIDC is stored>
note: calling it JWT is confusing, this is OIDC, JWT's are more general purpose and don't require OIDC
That way, users have full control over the security.json and the operator only needs to add it to ZK. This approach does require users to understand how to structure the security.json for OIDC, but personally, I'd rather not put that on the operator and having the operator support a user-supplied security json is a good feature to have anyway.
I think it would be great for a user to provide a custom configMap for their security.json, or a secret for that matter (since it might have sensitive information).
spec:
solrSecurity:
authenticationType: <type>
bootstrapSecurityJson: <requires key and either secret or configMap>
secret: <optional>
configMap: <optional>
key: <defaults to security.json>
Researching this comment, it turns out there are a lot of options when configuring the JWT plugin, and what we would be doing is basically just templating the json for them. So maybe just allowing a configMap or secret would be the right way to go.
Honestly there isn't really a need to differentiate things at some point if they are doing a custom security JSON. we just need an authorization header to send to Solr. So authenticationType could be Basic or Bearer.
spec:
...
solrSecurity:
authenticationType: Bearer
bootstrapSecurityJson:
configMap: <user-supplied config map>
key: <defaults to security.json>
oidc:
clientId: <operator client-id as registered with OIDC>
clientSecret: <name and key of k8s secret where operator's client secret for OIDC is stored>
bearerTokenSecret: <just a plain secret with a JWT token to use that is already setup by the user>
For the OIDC option, would the operator have to go fetch the JWT string to use every time by getting the oidc wellKnownUrl from the secret/configMap, then sending a request to that url with the provided ID and secret? I'm not familiar with oidc, so not exactly sure how that process works. I left it in there as an option, since it seemed more complicated than just using a predefined secret with a JWT token.
Not sold on this, just my thoughts currently.
One problem with the bearerTokenSecret approach is that it would require a non-expiring access token, which are generally frowned upon, but if people want to use those, I guess it's not a bad thing for the operator to support.
The operator will have to go back to the OIDC provider using the wellKnownUrl only when a previously acquired access token expires. So there would be a lookup to the ConfigMap and some parsing involved but wouldn't be on every request, only when we get back an expired token response (suppose we could cache the expiration time with the token too so we could preemptively refresh it but that's getting into the weeds)
the operator could parse out the wellKnownUrl and other config from a security.json provided by the user
Yes, a user provided security.json template is probably good to have. Note that the JWTAuthPlugin does not force users to use OIDC, and well-known endpoint, they may supply JWK directly (in which case the bearerTokenSecret may be nice to support). Also, there may be a list of several issuers in the same config, each having a different wellKnownUrl. The operator needs to care only about the wellKnownUrl of the IODC server where the operator itself is added as a client - so it can go fetch a token. We could require an issuer with a wellKnownUrl, and if there are multiple, they can be distinguished by the name property.
The Well-Known endpoint is an unprotected endpoint that lists all the other endpoints of the OIDC server. So to obtain the Token endpoint you use the token_endpoint of the well-known json. To obtain a token from the token endpoint, the operator needs to provide clientID and clientSecret. The token will have an exp claim to tell when it expires, which may be used to know when to fetch a new one.
So here's another iteration, adding issuer as optional key:
spec:
...
solrSecurity:
authenticationType: Bearer
bootstrapSecurityJson:
configMap: <user-supplied config map>
key: <defaults to security.json>
issuer: <optionally name of issuer to pull wellKnownUrl from, if multiple>
oidc:
clientId: <operator client-id as registered with OIDC>
clientSecret: <name and key of k8s secret where operator's client secret for OIDC is stored>
bearerTokenSecret: <just a plain secret with a JWT token to use that is already setup by the user>
I feel perhaps that the operator could help generate security.json like it does for BasicAuth, in order to get the permissions and exceptions for metrics and health right - those are complex beasts to hand-craft. Perhaps if bootstrapSecurityJson does not have an authorization section, it could be added?
Also, what do you think about storing a very simple default security.json template that will be used if bootstrapSecurityJson is not provided? In that case the oidc section needs to have an explicit wellKnownUrl.
@janhoy I have a PR up for this but I'm on the fence as to whether it's worth merging? Given SolrJ doesn't support JWT, users will have to open up the Prometheus exporter endpoints to anonymous requests, same for bin/solr and the probe endpoints. I actually think your idea of supporting multiple authentication schemes in SOLR-12666 is probably the better approach and then the operator + exporter + probe client can all just use basic auth instead. Anyway, give this a look over and see if you can get it running with your OIDC provider to make sure I have all the config options covered, I tested with Keycloak and auth0
Looks like tehre is some traction on https://github.com/apache/solr/pull/355 which could be a workaround for operator, exporter, bin/solr etc.
Only worry I have is for some orgs that deem BasicAuth not secure enough that they don't want to enable it at all. So I wonder if it makes sense to introduce an IP-address allowlist to BasicAuth, so you can explicitly allow those servers on the network that need access. I realize this may be hard in k8s where IPs can change any time. I don't know if it is a valid concern though.
But perhaps those few user/pass combinations in basicAuth config could be copuled to a role that has very limited permissions in authz, and that solves the issue? Is it possible to make a permission that only allows CLUSTERSTATUS command to the collections API, i.e. not allow any write operations?