When updating the admin password cluster requires manual intervention

[dbason]

Not sure if this is actually a usecase we will support but currently if the admin password is changed the cluster healthchecks will fail, leading to a situation where rolling restarts will not continue.

[prudhvigodithi]

Hey @dbason, the expected way to change the admin password is to updated the secret passed via cluster.yaml

adminCredentialsSecret:
        name:  admin-secret

and

dashboards:
    opensearchCredentialsSecret:
       name: admin-secret

so you say health checks fail ever after changing the admin password secrets and applying the cluster.yaml?

[wkrause13]

@prudhvigodithi I can confirm that I am seeing this issue as well. From the logs of master-0:

[2022-07-27T13:25:33,665][WARN ][o.o.s.a.BackendRegistry  ] [v3-cluster-masters-0] Authentication finally failed for admin from 127.0.0.1:55360

Here is the relevant portion of my config:

  general:
    serviceName: v3-cluster
    version: 2.1.0
  security:
    config:
      adminCredentialsSecret:
        name:  opensearch-admin-password
  dashboards:
    opensearchCredentialsSecret:
      name: opensearch-admin-password

[prudhvigodithi]

Hey @wkrause13 I'm able to change the admin password and access the cluster, did you use latest helm chart ?

helm install my-opensearch-operator opensearch-operator/opensearch-operator --version 2.0.0

Following are the steps to change the admin password:

• Create the admin user secret.

apiVersion: v1
kind: Secret
metadata:  
  name: admin-credentials-secret
type: Opaque
data:
  # admin
  username: YWRtaW4=
  # test
  password: dGVzdA==

• Create a securityConfig secret modifying the internal_users.yml

apiVersion: v1
kind: Secret
metadata:
  name: securityconfig-secret
type: Opaque
## admin opassword hash for test "$2y$12$B6GMBQIwOUEV2qtBQrpJL.37MUMp1XkLxCyWzeTH5Q94QxNjw8ng6"
stringData:
      internal_users.yml: |-
        _meta:
          type: "internalusers"
          config_version: 2
        admin:
          hash: "$2y$12$tS0wrbNssQpVjOXDPrzqdO5phJC/Fmb9fNKSdJ9P2voGK.LNIqLxG"
          reserved: true
          backend_roles:
          - "admin"
          description: "Demo admin user"
        dashboarduser:
          hash: "$2a$12$4AcgAt3xwOWadA5s5blL6ev39OXDNhmOesEoo33eZtrq2N0YrU3H."
          reserved: true
          description: "Demo OpenSearch Dashboards user"

• Finally in cluster yaml file pass the above created secrets and apply the yaml.

spec:
  security:
    config: 
     securityConfigSecret:
        name: securityconfig-secret
     adminCredentialsSecret:
        name: admin-credentials-secret

  dashboards:
    opensearchCredentialsSecret:
      name: admin-credentials-secret

The operator should now start the securityconfig-update with passed internal_users.yml, creating the right security index and uses admin-credentials-secret to authenticate the cluster.

• Now with new password, curl call should work with new -u admin:test

curl -XGET https://localhost:9400 -u 'admin:test' --insecure
{
  "name" : "my-first-cluster-bootstrap-0",
  "cluster_name" : "my-first-cluster",
  "cluster_uuid" : "GXP3muycQ9ubAAAhChupnQ",
  "version" : {
    "distribution" : "opensearch",
    "number" : "2.1.0",
    "build_type" : "tar",
    "build_hash" : "388c80ad94529b1d9aad0a735c4740dce2932a32",
    "build_date" : "2022-06-30T21:31:04.823801692Z",
    "build_snapshot" : false,
    "lucene_version" : "9.2.0",
    "minimum_wire_compatibility_version" : "7.10.0",
    "minimum_index_compatibility_version" : "7.0.0"
  },
  "tagline" : "The OpenSearch Project: https://opensearch.org/"
}

• Dashboard logs with 200 success code

{"type":"response","@timestamp":"2022-07-28T12:18:14Z","tags":[],"pid":1,"method":"get","statusCode":200,"req":{"url":"/api/reporting/stats","method":"get","headers":{"host":"192.168.40.199:5601","user-agent":"kube-probe/1.22+","accept":"*/*","connection":"close"},"remoteAddress":"192.168.59.91","userAgent":"kube-probe/1.22+"},"res":{"statusCode":200,"responseTime":2,"contentLength":9},"message":"GET /api/reporting/stats 200 2ms - 9.0B"}

Please try this way @wkrause13

@dbason @idanl21 @segalziv @segalziv should we consider documenting this? :)

[KannappanSomu]

I tested this as well, works good !

[prudhvigodithi]

Thanks for the update @KannappanSomu , closing this issue, please feel free to re-open if required. Thank you