Borg fails when file changes on disk during archive creation

[dmaroulidis]

Describe the bug

Borg_ynh fails to backup an application when files on disk change during archive creation. Probably related to https://github.com/borgbackup/borg/issues/6989. In my situation specifically, as you can see from the backup logs, matrix-synapse adds to homeserver.log during archive creation which causes borg to emit a warning and probably a signal that backup-with-borg receives and stops the process. I tried stopping synapse before starting the borg.service and the backup completes successfully.

Context

• Hardware: VM with VPN
• YunoHost version: 11.2.12
• I have access to my server: Through SSH | through the webadmin | direct access via keyboard / screen
• Are you in a special context or did you perform some particular tweaking on your YunoHost instance?: no
• Using, or trying to install package version/branch: borg 1.2.8~ynh1

Steps to reproduce

Upgrade borg_ynh to the latest version 1.2.8~ynh1 and get notified of the issue by email every morning.

Expected behavior

A solution would be to stop each service before backing it up and restart it after the backup is finished. A better setup would be to take a snapshot of the filesystem and backup the snapshot which would result in zero downtime, but that cannot be accomplished without a filesystem that supports this.

Logs

https://paste.yunohost.org/raw/duzoyalaho

[utzer]

I got the same problem since the late upgrade of borg backup.

2024-06-03 10:17:49,897: DEBUG - + /var/www/borg/venv/bin/borg init -e repokey ssh://user01@server01/./backup
2024-06-03 10:17:50,872: WARNING - A repository already exists at ssh://user01@server01/./backup.
2024-06-03 10:17:50,890: DEBUG - + set -e
2024-06-03 10:17:50,890: DEBUG - + /var/www/borg/venv/bin/borg create ssh://user01@server01/./backup::_auto_synapse-2024-06-03_10:17 ./
2024-06-03 10:17:50,890: DEBUG - + log_with_timestamp
2024-06-03 10:17:50,891: DEBUG - + tee -a /var/log/backup_borg.err
2024-06-03 10:17:50,891: DEBUG - ++ date +%Y-%m-%d_%H:%M:%S
2024-06-03 10:17:50,892: DEBUG - + sed -e 's/^/[2024-06-03_10:17:50] /'
2024-06-03 10:18:09,837: DEBUG - [2024-06-03_10:17:50] apps/synapse/backup/var/log/matrix-synapse/homeserver.log: file changed while we backed it up
2024-06-03 10:18:10,838: ERROR - Could not run script: /etc/yunohost/hooks.d/backup_method/05-borg_app
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/yunohost/hook.py", line 298, in hook_callback
    hook_return = hook_exec(
  File "/usr/lib/python3/dist-packages/yunohost/hook.py", line 416, in hook_exec
    raise YunohostError("hook_exec_failed", path=path)
yunohost.utils.error.YunohostError: Could not run script: /etc/yunohost/hooks.d/backup_method/05-borg_app

[CodeShakingSheep]

Same.

[utzer]

Anyone knows if there is some work done on this and if this here it the right place to report, maybe it should be reported via synapase package?

[utzer]

Sorry, it just bugs me that I get a "backup failed" email every two hours now. :-(

[utzer]

is there a possibility to exclude files from the backup, I mean the log is nice to have in some cases, but not the most important as well, so maybe we can simply exclude it for now?

[dmaroulidis]

@utzer I've looked through the backup script of synapse (the file that exports the file list to be included in the backup) but I couldn't figure out how to exclude the log from backups. That's something the maintainers of ynh_synapse could help with, but that would be a patch.

The bigger issue is that this thing can happen with any service that changes its files while the backup is running. That's why I suggested stopping each service before backing it up. I think this can be done in the yunohost backup framework, so that might be something to take to the yunohost repo or at least the forum.

I reverted to my pre-upgrade backup of borg and I've been using the previous version for the last 3 weeks. I'll update when this is fixed. Note that the same situation (a file is changed during backup) can happen with the previous version as well but the backup won't fail if this happens.

[Thatoo]

I face the same issue since I last updated synapse indeed.

I opened an issue on synapse repository.

[Thatoo]

@Josue-T answered on the issue opened here https://github.com/YunoHost-Apps/synapse_ynh/issues/474#issuecomment-2192632966 Hoping it will help.

[utzer]

@fflorent I wonder what the status of the PR is?

I also wonder if it would be a good idea to add a function to the borg app to executed a command before the borg create starts and one after borg create finished? This way it would also be possible to run "systemctl stop synapse.service" before borg create and "systemctl start synapse.service" after borg create finished.

[dmaroulidis]

I also wonder if it would be a good idea to add a function to the borg app to executed a command before the borg create starts and one after borg create finished? This way it would also be possible to run "systemctl stop synapse.service" before borg create and "systemctl start synapse.service" after borg create finished.

@utzer that could be a way to fix this issue in order to have working (non-failing) backups, and I'm sure one could find other uses as well for such mechanism. Other borg wrappers like borgmatic implement such functionality already.

Since opening the issue, I've moved to an out-of-band way of taking YunoHost backups, namely taking snapshots of the whole system and backing those up. That's why I haven't returned back to this issue since June.

[TabTwo]

I run into the same error after upgrading to 12.1 One thing thats a bit different, every app fails to be backupped with the same error.

Heres one of the logfiles if this helps https://paste.yunohost.org/raw/beyuriyaxo

[fflorent]

@TabTwo Actually this error is just a generic one, the details are stored in /var/log/borg/DATE.log (where DATE is the one when the failed backup occurred). Sorry for the confusion.

Also you seemed to have an error with Element (the Matrix client) whereas the error above happened when backuping Synapse (the Matrix server).

Could you create another issue with a detailed description of what happened using the logs whose path is mentioned above please?

Thanks in advance!

[TabTwo]

Ignore the element part, the same error happens for every installed app (like a dozen).

the date.err - log contains only these two lines for every app ERROR Could not run script: /etc/yunohost/hooks.d/backup_method/05-borg_app ERROR Custom backup method could not get past the 'backup' step

the date.log file repeats these line for every app INFO Collecting files to be backed up for mautrix_signal… INFO Declaring files to be backed up... INFO Backing up the PostgreSQL database... INFO Backup script completed for mautrix_signal. (YunoHost will then actually copy those files to the archive). INFO Creating a backup archive from the collected files… INFO The archive will contain about 49MB of data. INFO The operation 'Create a backup archive' could not be completed. Please share the full log of this operation using the command 'yunohost log share 20250828-104353-backup_create' to get help

I reinstall borg and try again, also with a different target

[TabTwo]

Turns out, the problem was the storage provider I am using, rsync.net. They want the --borg-path variable to be set. If you cant set this variable (like with yunohost) you can ask the support nicely, the do that on the serverside for you. This solved the backup problems for me.