[Enhancement][opensearch] provide example for snapshot

[Hokwang]

Hi,

I want to make helm chart values.yaml file for using snapshot.

My env is AWS EKS, install using helm chart.

First, make s3 repository using repository-s3 plugin. there's several ways in your document, but best thing is using IRSA I guess.

following https://opensearch.org/docs/2.4/opensearch/snapshots/snapshot-restore/,

sudo ./bin/opensearch-plugin install repository-s3 -> possible to automate using plugins.installList in helm chart

but I don't know how to handle in values.yaml below command ./bin/opensearch-keystore add s3.client.default.role_arn ./bin/opensearch-keystore add s3.client.default.role_session_name

and we need to run ln -s ${AWS_WEB_IDENTITY_TOKEN_FILE} /usr/share/opensearch/plugins/repository-s3/token

Second, register the repository, can I use in helm chart?

PUT _snapshot/my-s3-repository
{
  "type": "s3",
  "settings": {
    "bucket": "my-s3-bucket",
    "base_path": "my/snapshot/directory"
  }
}

[rishabh6788]

Currently snapshot functionality is not available in helm-charts, you need to start the cluster and then follow the manual steps to create and use snapshots. @prudhvigodithi Please correct me if I am wrong. See https://github.com/Opster/opensearch-k8s-operator/issues/278 for configuring and using snapshots.

[hobti01]

@Hokwang, you are correct with the use of plugins.installList to install the repository-s3 plugin. Unfortunately the registry is not registered; we wrap this chart in another chart that runs a configuration job to create the registry. It would be nice if this chart had that functionality for at least AWS, Azure, and GCP - and if the configuration I outline below would be part of the chart.

Despite this documentation, I would not recommend putting your AWS credentials into your Dockerfile and image. @rishabh6788, performing any manual steps after a Kubernetes deployment is generally considered an anti-pattern. Manual steps are not necessary with the Helm Chart. Unfortunately Helm Chart usage is not covered in the docs.

• Create a secret using your own method/automation in the same namespace as the opensearch deployment with the necessary entries. Below I've assumed you use a client named 'default' although you may use other names:

apiVersion: v1
kind: Secret
metadata:
  name: opensearch-keystore
type: Opaque
data:
  s3.client.default.<property>: <value>

• Reference the secret in your values.yaml

  keystore:
    - secretName: opensearch-keystore

• The script within the opensearch init container loads the values into the keystore: https://github.com/opensearch-project/helm-charts/blob/opensearch-2.12.0/charts/opensearch/templates/statefulset.yaml#L262-L299

Using AWS access_key and secret_key: The following properties are needed in the keystore secret:

  s3.client.default.access_key: YOUR_BASE64_ENCODED_VALUE
  s3.client.default.secret_key: YOUR_BASE64_ENCODED_VALUE

Using AWS IRSA: opensearch does use the variables set by the AWS admission controller including AWS_WEB_IDENTITY_TOKEN_FILE. But if you attempt to use the default configuration you will receive a 500 error during the snapshot verification check, the important detail is:

"access denied (\"java.io.FilePermission\" \"/var/run/secrets/eks.amazonaws.com/serviceaccount/token\" \"read\")"

This is because the location of the file is not permitted by the java security settings (no documentation located). The docs say the identity file should be copied or linked into OPENSEARCH_PATH_CONFIG without defining what that is.

In our configuration we override the AWS_WEB_IDENTITY_TOKEN_FILE location by providing a value in the opensearch configuration:

• We remount the aws-iam-token volume that is created by the AWS admission controller to an allowed path. Note that this volume is added to the Pod upon admission and is not part of the Helm chart.

  extraVolumeMounts:
    - mountPath: /usr/share/opensearch/config/irsa-token
      name: aws-iam-token # volume is created by AWS admission controller
      readOnly: true
      subPath: token

NOTE you cannot mount directly to /usr/share/opensearch/plugins/repository-s3/token (the location from the docs) because this will break the plugin installation within the startup script

• Configure the mounted path in opensearch.yml:

  config:
    opensearch.yml: |
      # ... your other configuration
      s3.client.default.identity_token_file: /usr/share/opensearch/config/irsa-token

NOTE if you attempt to set the optional s3.client.default.role_session_name it is considered a secure setting in opensearch despite being part of the non-sensitive ~/.aws/config file.

With this configuration of the Helm Chart you can use AWS static credentials or IRSA for snapshots.