Incomplete Backups

[sandeepmallik]

Project board link

Backups are showing incomplete even though they are completed on all nodes without error. Medusa version is 0.7.1

[cassandra@cass-03 ~]$ /usr/local/bin/medusa backup --backup-name=test13 --mode=full [2020-09-10 08:55:12,350] INFO: Monitoring provider is noop [2020-09-10 08:55:13,124] WARNING: is ccm : 0 [2020-09-10 08:55:13,161] INFO: Creating snapshot [2020-09-10 08:55:13,161] INFO: Saving tokenmap and schema [2020-09-10 08:55:14,194] INFO: Starting backup . . [2020-09-10 08:55:20,528] INFO: Updating backup index [2020-09-10 08:55:21,412] INFO: Backup done [2020-09-10 08:55:21,413] INFO: - Started: 2020-09-10 08:55:12 - Started extracting data: 2020-09-10 08:55:14 - Finished: 2020-09-10 08:55:21 [2020-09-10 08:55:21,413] INFO: - Real duration: 0:00:07.335504 (excludes time waiting for other nodes) [2020-09-10 08:55:21,413] INFO: - 161 files, 159.54 KB [2020-09-10 08:55:21,413] INFO: - 161 files copied from host [2020-09-10 08:55:21,414] INFO: - 0 copied from previous backup (test12)

[cassandra@cass-01 ~]$ medusa list-backups | grep test13 test13 (started: 2020-09-10 08:24:34, finished: Incomplete [3 of 3 nodes finished])

[cassandra@cass-01 ~]$ hostname -i 10.1.1.1 [cassandra@cass-01 ~]$ medusa status --backup-name test13 test13 [Incomplete!]

• Started: 2020-09-10 08:24:34, Finished: never
• 1 nodes completed, 0 nodes incomplete, 2 nodes missing
• Missing nodes: 10.1.1.2 10.1.1.3
• 209 files, 193.14 KB

[cassandra@cass-02 ~]$ hostname -i 10.1.1.2 [cassandra@cass-02 ~]$ medusa status --backup-name test13 test13 [Incomplete!]

• Started: 2020-09-10 08:24:34, Finished: never
• 2 nodes completed, 0 nodes incomplete, 2 nodes missing
• Missing nodes: 10.1.1.2 10.1.1.3
• 394 files, 369.64 KB

[cassandra@cass-03 ~]$ hostname -i 10.1.1.3 [cassandra@cass-03 ~]$ medusa status --backup-name test13 test13 [Incomplete!]

• Started: 2020-09-10 08:24:34, Finished: never
• 3 nodes completed, 0 nodes incomplete, 2 nodes missing
• Missing nodes: 10.1.1.2 10.1.1.3
• 555 files, 529.18 KB

Below is the medusa configuration.

[cassandra] stop_cmd = systemctl stop cassandra start_cmd = systemctl start cassandra config_file = /opt/cassandra/current/conf/cassandra.yaml check_running = nodetool version

[storage] storage_provider = s3_us_west_oregon bucket_name = medusa-test key_file = /etc/medusa/medusa-s3-credentials

┆Issue is synchronized with this Jira Story by Unito

[rzvoncek]

Grooming note: This is probably because of DNS. Whatever is written into the backup metadata is not the same as what the nodes usually run with.