What exactly happen when BACKUP_FROM_SNAPSHOT=true?

[SergeiCherevko]

Hello! Can anyone explaine what exactly happens when i use variable BACKUP_FROM_SNAPSHOT=true?

I test backup prometheus DB without stop, but with BACKUP_FROM_SNAPSHOT and i receive an error

Failure running docker-volume-backup at 2022-01-18T00:00:00+03:00
Running docker-volume-backup failed with error: takeBackup: error creating snapshot: open /backup/prometheus_data/01FSM8TPFEXQ0QC28H11PMQZ0R: no such file or directory
Log output of the failed run was:
time="2022-01-18T00:04:00+03:00" level=error msg="Fatal error running backup: takeBackup: error creating snapshot: open /backup/prometheus_data/01FSM8TPFEXQ0QC28H11PMQZ0R: no such file or directory"
time="2022-01-18T00:04:03+03:00" level=info msg="Removed snapshot /tmp/backup."

If i use label which is stops container before backup - all ok

Here part of compose file

  backup:
    container_name: prometheus_backup
    image: offen/docker-volume-backup:v2.7.2
    restart: always
    environment:
      BACKUP_FILENAME: backup-{{ prometheus_name }}-%Y-%m-%dT%H-%M-%S.tar.gz
      BACKUP_FROM_SNAPSHOT: "true"
      BACKUP_RETENTION_DAYS: "7"
      NOTIFICATION_LEVEL: "info"
      NOTIFICATION_URLS: "{{ slack_hook}}"
      # BACKUP_STOP_CONTAINER_LABEL: {{ prometheus_name }}
    volumes:
      - "{{ prometheus_volume_config }}:/backup/{{ prometheus_volume_config }}:ro"
      - "{{ prometheus_volume_data }}:/backup/{{ prometheus_volume_data }}:ro"
      - /var/run/docker.sock:/var/run/docker.sock:ro
      - "/hdd/backup/{{ prometheus_name }}:/archive"
      - /etc/timezone:/etc/timezone:ro
      - /etc/localtime:/etc/localtime:ro
    networks:
      - docker_office

networks: 
  docker_office:
    external: true
  
volumes:
  {{ prometheus_volume_config }}:
    external: true  
  {{ prometheus_volume_data }}:
    external: true

[m90]

BACKUP_FROM_SNAPSHOT is supposed to be used in the following situation:

1. you cannot stop your containers while taking the backup (this should always be the preferred option)
2. you are backing up a highly volatile file system that will change while writing the tar archive, see #32

In your particular case it looks as if a file in the backup target gets deleted while / right before you are trying to copy it. This is probably an upstream issue in package copy (https://github.com/otiai10/copy), maybe we should open an issue there or see if it has been reported before.

In the meantime, is stopping the containers during backup a workable option for you? You could also try running the backup without stopping containers and without using the snapshot option, but I would assume this fails as well, exposing the behavior described in #32

[SergeiCherevko]

Thank you for your answer.

I create backup of prometheus container. Backup time - 40 minutes. If i stop prometheus for this time it will be not good. From other side i can send to prometheus container some command like that curl -XPOST -u $prom_user:$prom_pass http://prometheus/api/v1/admin/tsdb/snapshot to create snapshot

But how can i send this curl request from your docker_backup solution before start backuping?

Also i have many containers such as svn, postgres in which always write any dynamical data and this container can't stop on during backup process and i can't use BACKUP_FROM_SNAPSHOT because is just not working as need :(

[m90]

But how can i send this curl request from your docker_backup solution before start backuping?

Right now you can't, see #45

Also i have many containers such as svn, postgres in which always write any dynamical data and this container can't stop on during backup process and i can't use BACKUP_FROM_SNAPSHOT because is just not working as need :(

I'm not sure what is happening exactly, but maybe a good start would be creating an issue in the upstream repository about this https://github.com/otiai10/copy

[m90]

I created an upstream issue here https://github.com/otiai10/copy/issues/72

[m90]

Not fixing the underlying issue, but running pre/post commands for preparing backups is now supported as of v2.13.0

[m90]

An upstream fix is now included in v2.26.1