basic auth for Admin API not working

[JakeSCahill]

What happened?

By default when you set auth.sasl.enabled=true, the Schema Registry and HTTP Proxy listeners are configured with basic auth. However, the Admin API is not.

You can enable it with config.cluster.admin_api_require_auth=true but this currently results in being unable to authenticate to the Admin API, even as a super user.

~For 23.3, we will also need to set the cluster property http_authentication='["BASIC"]' but the chart doesn't currently seem to support setting that array. @BenPope says BASIC should be the default in RP.~ This is now fixed.

What did you expect to happen?

You should be able to enable basic auth for the Admin API and authenticate using one of the available superusers.

How can we reproduce it (as minimally and precisely as possible)?. Please include values file.

With admin_api_require_auth=true:

export DOMAIN=customredpandadomain.local && \                                                                    
helm repo add redpanda https://charts.redpanda.com/
helm repo update
helm install redpanda redpanda/redpanda \
  --namespace redpanda \
  --create-namespace \
  --set external.domain=${DOMAIN} \
  --set statefulset.initContainers.setDataDirOwnership.enabled=true --set auth.sasl.enabled=true \
  --set "auth.sasl.users[0].name=superuser" \
  --set "auth.sasl.users[0].password=secretpassword" --set config.cluster.admin_api_require_auth=true

Cannot authenticate:

kubectl exec redpanda-0 -n redpanda -- rpk acl user list -X user=superuser -X pass=secretpassword
Defaulted container "redpanda" out of: redpanda, config-watcher, tuning (init), set-datadir-ownership (init), redpanda-configurator (init)
Request error, trying another node: request GET https://redpanda-2.redpanda.redpanda.svc.cluster.local.:9644/v1/security/users failed: Unauthorized, body: "{\"message\": \"Unauthorized\", \"code\": 401}"

Request error, trying another node: request GET https://redpanda-1.redpanda.redpanda.svc.cluster.local.:9644/v1/security/users failed: Unauthorized, body: "{\"message\": \"Unauthorized\", \"code\": 401}"

unable to list users: request GET https://redpanda-0.redpanda.redpanda.svc.cluster.local.:9644/v1/security/users failed: Unauthorized, body: "{\"message\": \"Unauthorized\", \"code\": 401}"

command terminated with exit code 1

Values:

```yaml auth: sasl: enabled: true users: - name: superuser password: secretpassword external: domain: customredpandadomain.local statefulset: initContainers: setDataDirOwnership: enabled: true config: cluster: admin_api_require_auth: true ```

Anything else we need to know?

No response

Which are the affected charts?

No response

Chart Version(s)

$ helm -n <redpanda-release-namespace> list 
redpanda-5.6.38	v23.2.14

Cloud provider

I was using kind

JIRA Link: K8S-72

[BenPope]

kubectl exec redpanda-0 -n redpanda -c redpanda -- rpk cluster config get http_authentication -X user=superuser -X pass=secretpassword
Property 'http_authentication' not found

This is likely due to the helm chart having redpanda v23.2.17, and that config doesn't exist until v23.3.1-rc1

[BenPope]

The AuthN failure when installing with admin_api_require_auth=true is a bit strange, it looks like everything should be set up properly. I managed to use rpk topic create with credentials (which means the user exists), but not able to access the Admin API with the same credentials.

Installing without admin_api_require_auth=true and then enabling it, also works as expected.

[segevd-opti]

Encountering the same issue with Redpanda 24.1.1, helm chart 5.8.7.

On the values.yaml, we've set SASL auth enabled with a superuser and also admin_api_require_auth to be true. The pods start running, but then we see the following error on the logs:

redpanda-post-upgrade-bcsbz redpanda-post-upgrade No changes were made: Validation errors:
redpanda-post-upgrade-bcsbz redpanda-post-upgrade  * admin_api_require_auth: May only be set by a superuser

seems like the post-upgrade job doesn't have the superuser credentials and so it can't apply the config changes.

Per @JakeSCahill advice, we've currently set our own job to run after the cluster is running and perform the config changes.

Slack thread

[JakeSCahill]

Tested in chart 5.9.0 and it's working now

[ligfx]

Reopening, can't get this to work on 5.9.1 with the following values.yaml:

auth:
  sasl:
    enabled: true
    mechanism: SCRAM-SHA-256
    users:
    - name: admin
      password: admin
      mechanism: SCRAM-SHA-256

config:
  cluster:
    admin_api_require_auth: true

[andrewstucki]

This should be fixed by https://github.com/redpanda-data/helm-charts/pull/1513 - if you go from an already bootstrapped cluster to full admin_api_require_auth however, when the version of this chart gets deployed you'll need to:

1. Deploy a version without admin_api_require_auth (but SASL enabled) so that the user normally configured during the bootstrapping phase can get backfilled.
2. Upgrade to enable admin_api_require_auth

Any newly deployed clusters should work with admin_api_require_auth + SASL enabled out of the box with that fix.