pruneBackups: error looking up candidates from remote storage

[Rwanito]

• I'm submitting a ...
  • [x] bug report
  • [ ] feature request
  • [ ] support request

Hello,

I'm looking for rotating and pruning my backups, since they are becoming quite big. So here my .env config :

BACKUP_RETENTION_DAYS=7
BACKUP_PRUNING_LEEWAY=1h
BACKUP_PRUNING_PREFIX=backup-

But I get this :

time="2022-07-22T09:14:59+02:00" level=info msg="Stopping 1 container(s) labeled `docker-volume-backup.stop-during-backup=true` out of 18 running container(s)."
time="2022-07-22T09:15:16+02:00" level=info msg="Created backup of `/backup` at `/tmp/backup-2022-07-22T09-14-58.tar.gz`."
time="2022-07-22T09:15:17+02:00" level=info msg="Restarted 1 container(s) and the matching service(s)."
time="2022-07-22T09:15:20+02:00" level=info msg="Encrypted backup using given passphrase, saving as `/tmp/backup-2022-07-22T09-14-58.tar.gz.gpg`."
time="2022-07-22T09:15:22+02:00" level=info msg="Uploaded a copy of backup `/tmp/backup-2022-07-22T09-14-58.tar.gz.gpg` to bucket `........`."
time="2022-07-22T09:15:23+02:00" level=error msg="Fatal error running backup: pruneBackups: error looking up candidates from remote storage: The specified key does not exist."
time="2022-07-22T09:15:23+02:00" level=info msg="Removed tar file `/tmp/backup-2022-07-22T09-14-58.tar.gz`."
time="2022-07-22T09:15:23+02:00" level=info msg="Removed GPG file `/tmp/backup-2022-07-22T09-14-58.tar.gz.gpg`."

But, at the end, the tar.gz.gpg is uploaded. I would like to know if that error lets DVB uploading in the bucket ? Or It's just a warning because my backups are younger than 7 days?

And I have another question, So, my stacks are divided by docker-compose.yml : one for specific goal. Each has a DVB for dedicated volumes.

But when I'm executing this command (inside a specific one) :

docker compose exec backup backup

I can see logs related to another containers and stacks with the labels : archive-pre. It is like an unnecessary operation, I don't know if I explain myself well :p !

Thank you for your work ! I really appreciate it !

Erwan

[m90]

I would like to know if that error lets DVB uploading in the bucket ?

Pruning happens at the last phase of the backup lifecycle, so a failed attempt does not block anything else from happening (as opposed to for example if upload would fail, pruning would not happen at all as the script exits).

The error you are describing is a bit mysterious to me. Here's what happens in the code: https://github.com/offen/docker-volume-backup/blob/26c8ba971f9e2a19f18485ae75c2776b20c3ad1c/cmd/backup/script.go#L655-L674

If nothing matches your query, I would expect the S3 server to simply return no objects in the response instead of the The specified key does not exist. (which is a 404 basically). If you have further details on how to reproduce this it would be great. Maybe extending the test case here https://github.com/offen/docker-volume-backup/tree/main/test/s3 so that it fails with the same error would be a good starting point for investigation?

---

I can see logs related to another containers and stacks with the labels : archive-pre.

If you need to limit the scope of your backup run when it comes to running commands in containers, you can use the EXEC_LABEL feature which lets you filter the number of applicable containers. E.g. if you have a app stack and a database stack, you can use a label of app (applying it to all services in that stack) and a database label (applying it to all services in that stack). There's more info in this issue: https://github.com/offen/docker-volume-backup/issues/129

[ghost]

Hello,

It looks like my S3 server is the problem. I'll debug after the last PR I saw because maybe It looks easier.

After, I will give you a PR to adjust some options or debug messages!

Thank you!