[k8s][Cassandra] make_backup.sh: garbage collect /tmp/backup/$table_* directories

[swarren83]

Category:

Kubernetes apps: Cassandra

Type:

• [x] Bug
• [ ] Feature Request
• [ ] Process

---

If a backup fails to complete successfully, subsequent backups will fail due to the /tmp/backup/$table_* directories not being removed.

Steps to reproduce:

1. ./backup.sh --keyspace system --namespace default --app_instance_name cassandra-1
2. CTRL+C to interrupt the backup after it commences the 'Running backup script' step
3. ./backup.sh --keyspace system --namespace default --app_instance_name cassandra-1

Log excerpt:

Running backup script
Defaulting container name to cassandra.
Use 'kubectl describe pod/cassandra-1-cassandra-0 -n default' to see all of the containers in this pod.
mkdir: cannot create directory '/tmp/backup/schema_functions': File exists
command terminated with exit code 1
Defaulting container name to cassandra.
Use 'kubectl describe pod/cassandra-1-cassandra-1 -n default' to see all of the containers in this pod.
mkdir: cannot create directory '/tmp/backup/size_estimates': File exists
command terminated with exit code 1
Defaulting container name to cassandra.
Use 'kubectl describe pod/cassandra-1-cassandra-2 -n default' to see all of the containers in this pod.
mkdir: cannot create directory '/tmp/backup/schema_columnfamilies': File exists
command terminated with exit code 1

https://github.com/GoogleCloudPlatform/click-to-deploy/blob/master/k8s/cassandra/scripts/make_backup.sh#L37

[swarren83]

The issue is that the delete_files_from_containers_cmd function is failing but not getting a non-zero exit code.