basic auth for Admin API not working

[JakeSCahill]

What happened?

By default when you set auth.sasl.enabled=true, the Schema Registry and HTTP Proxy listeners are configured with basic auth. However, the Admin API is not.

You can enable it with config.cluster.admin_api_require_auth=true but this currently results in being unable to authenticate to the Admin API, even as a super user.

~For 23.3, we will also need to set the cluster property http_authentication='["BASIC"]' but the chart doesn't currently seem to support setting that array. @BenPope says BASIC should be the default in RP.~ This is now fixed.

What did you expect to happen?

You should be able to enable basic auth for the Admin API and authenticate using one of the available superusers.

How can we reproduce it (as minimally and precisely as possible)?. Please include values file.

With admin_api_require_auth=true:

export DOMAIN=customredpandadomain.local && \                                                                    
helm repo add redpanda https://charts.redpanda.com/
helm repo update
helm install redpanda redpanda/redpanda \
  --namespace redpanda \
  --create-namespace \
  --set external.domain=${DOMAIN} \
  --set statefulset.initContainers.setDataDirOwnership.enabled=true --set auth.sasl.enabled=true \
  --set "auth.sasl.users[0].name=superuser" \
  --set "auth.sasl.users[0].password=secretpassword" --set config.cluster.admin_api_require_auth=true

Cannot authenticate:

kubectl exec redpanda-0 -n redpanda -- rpk acl user list -X user=superuser -X pass=secretpassword
Defaulted container "redpanda" out of: redpanda, config-watcher, tuning (init), set-datadir-ownership (init), redpanda-configurator (init)
Request error, trying another node: request GET https://redpanda-2.redpanda.redpanda.svc.cluster.local.:9644/v1/security/users failed: Unauthorized, body: "{\"message\": \"Unauthorized\", \"code\": 401}"

Request error, trying another node: request GET https://redpanda-1.redpanda.redpanda.svc.cluster.local.:9644/v1/security/users failed: Unauthorized, body: "{\"message\": \"Unauthorized\", \"code\": 401}"

unable to list users: request GET https://redpanda-0.redpanda.redpanda.svc.cluster.local.:9644/v1/security/users failed: Unauthorized, body: "{\"message\": \"Unauthorized\", \"code\": 401}"

command terminated with exit code 1

Values:

```yaml auth: sasl: enabled: true users: - name: superuser password: secretpassword external: domain: customredpandadomain.local statefulset: initContainers: setDataDirOwnership: enabled: true config: cluster: admin_api_require_auth: true ```

Anything else we need to know?

No response

Which are the affected charts?

No response

Chart Version(s)

$ helm -n <redpanda-release-namespace> list 
redpanda-5.6.38	v23.2.14

Cloud provider

I was using kind

JIRA Link: K8S-72