GCS backend generates huge amount of operations

[oleo65]

I started to use stash to backup my persistent volumes to google cloud storage.

Setup has been done via helm and I was using v. 0.9.0-rc and most recently 2020.09.29

For evaluation I setup one GCS Repository and scheduled a backup of one PV to run once a week. Everything was working as expected.

A couple days ago I saw on the Google Bill a huge amount of Class A and Class B operations which started after installing and using stash.

Within almost a month of time stash generated over 1.1 million class B operations and over 0.3 million class A operations which simply does not explain to me. This generates a huge amount of overhead cost (compared to the actual storage cost) and I simply don't understand how to stop stash from doing that.

As a first countermeasure I had to delete the whole stash chart and the operations stopped immediately.

I will be happy to assist you with further information if needed to investigate this behaviour.

[hossainemruz]

@oleo65 Thank you for reporting the issue. We have to investigate it. There is no reason for Stash to perform such number of operation.

Can you please provide the following informations?

1. What is your Kubernetes version?
2. How many workload/volume/database are you taking backup?

[oleo65]

Thank you for your heads up.

• I am using currently Kubernetes version 1.19.2 (microk8s deployed via snaps on Ubuntu 20.04.1 LTS) as a 3 node cluster.
• Backup is (was) setup for 1 workload's volume via CRDs

apiVersion: stash.appscode.com/v1beta1
kind: BackupConfiguration
metadata:
  name: instructor-backup
  namespace: default
spec:
  driver: Restic
  repository:
    name: gcs-repo-instructor
  schedule: "45 21 * * 1"
  target:
    ref:
      apiVersion: apps/v1
      kind: Deployment
      name: instructor
    volumeMounts:
      - name: database
        mountPath: /opt/app/database
    paths:
      - /opt/app/database
  retentionPolicy:
    name: "keep-last-5"
    keepLast: 5
    prune: true

The /opt/app/database folder contains sqlite databases, nothing too fancy.