[k8s][elasticsearch] backup & restore

[ewanwalk]

There are multiple errors in the following files:

scripts/create-backup-infra.sh scripts/patch-sts-for-backup.sh

Wherein environment variables are not being properly passed / wrong ones are being mapped.

Upon fixing the above it still does not working due to the PVC seemingly using the NFS servers claim instead of the backup claim (e.g. only the default 5Gi will be available instead of whatever you've provided as the size of your backup-claim).

[devlware]

Hi @Ewan-Walker Can you take a look again in the scripts? They were updated and those problems with missing env vars shall be resolved.

[ewanwalk]

Hey,

It looks like it was updated in elastic-gke-logging but not the other elasticsearch directory.

However the update is missing the following:

storage: $BACKUP_DISK_SIZE

Found @scripts/backup-nfs.yaml.template#L44

As without setting the volume size correctly you may only allocate up to 5Gi as far as I was able to tell.

[devlware]

The code inside the script file create-backup-infra.sh inside the elastic-gke-logging dir looks good to me.

echo "Expanding manifest templates for NFS server and shared disk..."
readonly nfs_manifest_file="/tmp/backup-nfs-expanded.yaml"
export APP_INSTANCE_NAME="${app}"
export BACKUP_DISK_SIZE="${disk_size}"
export BACKUP_CLAIM_NAME="${backup_claim}"
cat scripts/backup-nfs.yaml.template \
  | envsubst '$APP_INSTANCE_NAME $BACKUP_DISK_SIZE $BACKUP_CLAIM_NAME' \
  > "${nfs_manifest_file}"

You can check that it calls envsubst for BACKUP-DISK_SIZE.

[ewanwalk]

Yes I saw you updated the one directory, but you did not mirror those changes to the other directory which shares the same code.

As well as the script does not work as intended because while it creates a PV for your desired storage amount, at least on GKE you must also edit the line I provided above in order to have enough storage to provision whatever sized claim you're asking for.

For example:

scripts/create-backup-infra.sh \
  --app elasticsearch-1 \
  --namespace default \
  --disk-size 10Gi \
  --backup-claim elasticsearch-1-backup

Will only have 5Gi available on the elasticsearch-1-backup claim unless you substitute L44 as I did above.

You also need to edit the persistentVolume template to include a storage class or it will not work (on GKE at least)

[devlware]

I agree with you with those points.

Maybe we could provide in the documentation that the user could create a StorageClass like:

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: fast
provisioner: kubernetes.io/gce-pd
parameters:
  type: pd-ssd

This way the user could use a ssd or the standard storageclass (if I recall GKE provides this) in another parameter that could be substituted in the template file.