docker-volume-backup
docker-volume-backup copied to clipboard
What exactly happen when BACKUP_FROM_SNAPSHOT=true?
Hello! Can anyone explaine what exactly happens when i use variable BACKUP_FROM_SNAPSHOT=true?
I test backup prometheus DB without stop, but with BACKUP_FROM_SNAPSHOT and i receive an error
Failure running docker-volume-backup at 2022-01-18T00:00:00+03:00
Running docker-volume-backup failed with error: takeBackup: error creating snapshot: open /backup/prometheus_data/01FSM8TPFEXQ0QC28H11PMQZ0R: no such file or directory
Log output of the failed run was:
time="2022-01-18T00:04:00+03:00" level=error msg="Fatal error running backup: takeBackup: error creating snapshot: open /backup/prometheus_data/01FSM8TPFEXQ0QC28H11PMQZ0R: no such file or directory"
time="2022-01-18T00:04:03+03:00" level=info msg="Removed snapshot /tmp/backup."
If i use label which is stops container before backup - all ok
Here part of compose file
backup:
container_name: prometheus_backup
image: offen/docker-volume-backup:v2.7.2
restart: always
environment:
BACKUP_FILENAME: backup-{{ prometheus_name }}-%Y-%m-%dT%H-%M-%S.tar.gz
BACKUP_FROM_SNAPSHOT: "true"
BACKUP_RETENTION_DAYS: "7"
NOTIFICATION_LEVEL: "info"
NOTIFICATION_URLS: "{{ slack_hook}}"
# BACKUP_STOP_CONTAINER_LABEL: {{ prometheus_name }}
volumes:
- "{{ prometheus_volume_config }}:/backup/{{ prometheus_volume_config }}:ro"
- "{{ prometheus_volume_data }}:/backup/{{ prometheus_volume_data }}:ro"
- /var/run/docker.sock:/var/run/docker.sock:ro
- "/hdd/backup/{{ prometheus_name }}:/archive"
- /etc/timezone:/etc/timezone:ro
- /etc/localtime:/etc/localtime:ro
networks:
- docker_office
networks:
docker_office:
external: true
volumes:
{{ prometheus_volume_config }}:
external: true
{{ prometheus_volume_data }}:
external: true
BACKUP_FROM_SNAPSHOT
is supposed to be used in the following situation:
- you cannot stop your containers while taking the backup (this should always be the preferred option)
- you are backing up a highly volatile file system that will change while writing the tar archive, see #32
In your particular case it looks as if a file in the backup target gets deleted while / right before you are trying to copy it. This is probably an upstream issue in package copy
(https://github.com/otiai10/copy), maybe we should open an issue there or see if it has been reported before.
In the meantime, is stopping the containers during backup a workable option for you? You could also try running the backup without stopping containers and without using the snapshot option, but I would assume this fails as well, exposing the behavior described in #32
Thank you for your answer.
I create backup of prometheus container. Backup time - 40 minutes. If i stop prometheus for this time it will be not good.
From other side i can send to prometheus container some command like that
curl -XPOST -u $prom_user:$prom_pass http://prometheus/api/v1/admin/tsdb/snapshot
to create snapshot
But how can i send this curl request from your docker_backup solution before start backuping?
Also i have many containers such as svn
, postgres
in which always write any dynamical data and this container can't stop on during backup process and i can't use BACKUP_FROM_SNAPSHOT
because is just not working as need :(
But how can i send this curl request from your docker_backup solution before start backuping?
Right now you can't, see #45
Also i have many containers such as svn, postgres in which always write any dynamical data and this container can't stop on during backup process and i can't use BACKUP_FROM_SNAPSHOT because is just not working as need :(
I'm not sure what is happening exactly, but maybe a good start would be creating an issue in the upstream repository about this https://github.com/otiai10/copy
I created an upstream issue here https://github.com/otiai10/copy/issues/72
Not fixing the underlying issue, but running pre/post commands for preparing backups is now supported as of v2.13.0
An upstream fix is now included in v2.26.1