apache
readiness probes of solr pod with `probesRequireAuth` set to `true` always return "Success" regardless of the actual result
The Readiness Probes of the chart always return success as the command of the authenticated wget is always piped into a grep command which always has the return code 0 regardless weather they were able to access the Endpoint or not. This leads to the solr-pod starting clean and kept alive by k8s even though the authentication to the endpoints is actually failing. In my opinion this shall lead to the pod not becoming ready.
The same applies to the Liveness Probes respectively.
the generated yaml of the probes from the solr pod by the solr-operator:
livenessProbe:
exec:
command:
- sh
- -c
- JAVA_TOOL_OPTIONS="-Dbasicauth=$(cat /etc/secrets/solr-k8s-oper-credentials/username):$(cat
/etc/secrets/solr-k8s-oper-credentials/password) -Dsolr.httpclient.builder.factory=org.apache.solr.client.solrj.impl.PreemptiveBasicAuthClientBuilderFactory"
solr api -get "http://${SOLR_HOST}:8983/solr/admin/info/system" 2>&1 | grep
-v JAVA_TOOL_OPTIONS
failureThreshold: 3
periodSeconds: 20
successThreshold: 1
timeoutSeconds: 5
name: solrcloud-node
ports:
- containerPort: 8983
name: solr-client
protocol: TCP
readinessProbe:
exec:
command:
- sh
- -c
- JAVA_TOOL_OPTIONS="-Dbasicauth=$(cat /etc/secrets/solr-k8s-oper-credentials/username):$(cat /etc/secrets/solr-k8s-oper-credentials/password) -Dsolr.httpclient.builder.factory=org.apache.solr.client.solrj.impl.PreemptiveBasicAuthClientBuilderFactory" solr api -get "http://${SOLR_HOST}:8983/solr/admin/info/health" 2>&1 | grep -v JAVA_TOOL_OPTIONS
solr@solr-solrcloud-0:/opt/solr-9.2.1$ JAVA_TOOL_OPTIONS="-Dbasicauth=$(cat /etc/secrets/solr-k8s-oper-credentials/username):$(cat /etc/secrets/solr-k8s-oper-credentials/password) -Dsolr.httpclient.builder.factory=org.apache.solr.client.solrj.impl.PreemptiveBasicAuthClientBuilderFactory" solr api -get "http://${SOLR_HOST}:8983/solr/admin/info/health" 2>&1 | grep -v JAVA_TOOL_OPTIONS
ERROR: Parse error : <html>
<head>
<meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1"/>
<title>Error 401 Bad credentials</title>
</head>
<body><h2>HTTP ERROR 401 Bad credentials</h2>
<table>
<tr><th>URI:</th><td>/solr/admin/info/health</td></tr>
<tr><th>STATUS:</th><td>401</td></tr>
<tr><th>MESSAGE:</th><td>Bad credentials</td></tr>
<tr><th>SERVLET:</th><td>default</td></tr>
</table>
</body>
</html>
solr@solr-solrcloud-0:/opt/solr-9.2.1$ echo $?
0
solr@solr-solrcloud-0:/opt/solr-9.2.1$ JAVA_TOOL_OPTIONS="-Dbasicauth=$(cat /etc/secrets/solr-k8s-oper-credentials/username):$(cat /etc/secrets/solr-k8s-oper-credentials/password) -Dsolr.httpclient.builder.factory=org.apache.solr.client.solrj.impl.PreemptiveBasicAuthClientBuilderFactory" solr api -get "http://${SOLR_HOST}:8983/solr/admin/info/health" 2>&1
Picked up JAVA_TOOL_OPTIONS: -Dbasicauth=k8s-oper:***REDACTED*** -Dsolr.httpclient.builder.factory=org.apache.solr.client.solrj.impl.PreemptiveBasicAuthClientBuilderFactory
ERROR: Parse error : <html>
<head>
<meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1"/>
<title>Error 401 Bad credentials</title>
</head>
<body><h2>HTTP ERROR 401 Bad credentials</h2>
<table>
<tr><th>URI:</th><td>/solr/admin/info/health</td></tr>
<tr><th>STATUS:</th><td>401</td></tr>
<tr><th>MESSAGE:</th><td>Bad credentials</td></tr>
<tr><th>SERVLET:</th><td>default</td></tr>
</table>
</body>
</html>
solr@solr-solrcloud-0:/opt/solr-9.2.1$ echo $?
1
Downside is that without the grep -v the basicAuth Password may be logged. I can't figure out where it may be logged but, I assume the grep -v may have been introduce to suppress the logging of the credentials.
supplying the correct basic auth also renders the correct return code (only without the return code - suppressing grep):
solr@solr-solrcloud-0:/opt/solr-9.2.1$ JAVA_TOOL_OPTIONS="-Dbasicauth=$(cat /etc/secrets/solr-k8s-oper-credentials/username):$(cat /etc/secrets/solr-k8s-oper-credentials/password) -Dsolr.httpclient.builder.factory=org.apache.solr.client.solrj.impl.PreemptiveBasicAuthClientBuilderFactory" solr api -get "http://${SOLR_HOST}:8983/solr/admin/info/health" 2>&1
Picked up JAVA_TOOL_OPTIONS: -Dbasicauth=k8s-oper:***REDACTED*** -Dsolr.httpclient.builder.factory=org.apache.solr.client.solrj.impl.PreemptiveBasicAuthClientBuilderFactory
{
"responseHeader":{
"status":0,
"QTime":0},
"status":"OK"}
solr@solr-solrcloud-0:/opt/solr-9.2.1$ echo $?
0
A direct approach would use the HTTP probes with httpHeaders like:
startupProbe:
httpGet:
httpHeaders:
- name: Authorization
value: Basic azhzLW9wZXI6ZXhhbXBsZQ
⚠️ Which provides the username "k8s-oper" and the password "example" plain base64 encoded in the pod spec. But I guess the Solr Operator could create such a spec.
Maybe we can just use curl as command? The password can still be provided as mounted secret like now:
curl --user "$(cat /etc/secrets/solr-k8s-oper-credentials/username):$(cat /etc/secrets/solr-k8s-oper-credentials/password)" \
"http://${SOLR_HOST}:8983/solr/admin/info/health" | grep '"status":"OK"'
