matrix-docker-ansible-deploy
matrix-docker-ansible-deploy copied to clipboard
spantaleev
borgbackup No space left on device
When running borgbackup (installed via the playbook) I get this log:
Aug 27 21:03:45 HOSTNAME systemd[1]: Starting Matrix Borg Backup...
Aug 27 21:03:50 HOSTNAME matrix-backup-borg[1241]: Remote: Warning: Permanently added 'borg.DOMAIN' (ED25519) to the list of known hosts.
Aug 27 21:05:10 HOSTNAME matrix-backup-borg[4553]: Remote: Warning: Permanently added 'borg.DOMAIN' (ED25519) to the list of known hosts.
Aug 27 21:06:24 HOSTNAME matrix-backup-borg[4553]: ------------------------------------------------------------------------------
Aug 27 21:06:24 HOSTNAME matrix-backup-borg[4553]: Original size Compressed size Deduplicated size
Aug 27 21:06:24 HOSTNAME matrix-backup-borg[4553]: Deleted data: 0 B 0 B 0 B
Aug 27 21:06:24 HOSTNAME matrix-backup-borg[4553]: All archives: 189.43 GB 184.96 GB 61.44 GB
Aug 27 21:06:24 HOSTNAME matrix-backup-borg[4553]: Unique chunks Total chunks
Aug 27 21:06:24 HOSTNAME matrix-backup-borg[4553]: Chunk index: 228362 754278
Aug 27 21:06:24 HOSTNAME matrix-backup-borg[4553]: ------------------------------------------------------------------------------
Aug 27 21:36:40 HOSTNAME matrix-backup-borg[4553]: /tmp/.borgmatic/postgresql_databases/matrix-postgres/synapse: read: [Errno 28] No space left on device: '/tmp/.cache/borg/aa203a7717d3abe4fe60d424157f749fafb5b7c84a29fbdb13d182bebc2d7a99/files' -> '/tmp/.cache/borg/aa203a7717d3abe4fe60d424157f749fafb5b7c84a29fbdb13d182bebc2d7a99/txn.tmp/files'
Aug 27 21:36:40 HOSTNAME matrix-backup-borg[4553]: [email protected]:/home/borg/my_servers/REPO: Error running actions for repository
Aug 27 21:36:40 HOSTNAME matrix-backup-borg[4553]: Command 'pg_dump --no-password --clean --if-exists --host matrix-postgres --port 5432 --username matrix --format custom synapse > /tmp/.borgmatic/postgresql_databases/matrix-postgres/synapse' died with <Signals.SIGPIPE: 13>.
Aug 27 21:36:40 HOSTNAME matrix-backup-borg[4553]: Error while creating a backup.
Aug 27 21:36:40 HOSTNAME matrix-backup-borg[4553]: /etc/borgmatic.d/config.yaml: Error running configuration file
Aug 27 21:36:40 HOSTNAME matrix-backup-borg[4553]: summary:
Aug 27 21:36:40 HOSTNAME matrix-backup-borg[4553]: /etc/borgmatic.d/config.yaml: Error running configuration file
Aug 27 21:36:40 HOSTNAME matrix-backup-borg[4553]: [email protected]:/home/borg/my_servers/HS_DOMAIN.matrix: Error running actions for repository
Aug 27 21:36:40 HOSTNAME matrix-backup-borg[4553]: Command 'pg_dump --no-password --clean --if-exists --host matrix-postgres --port 5432 --username matrix --format custom synapse > /tmp/.borgmatic/postgresql_databases/matrix-postgres/synapse' died with <Signals.SIGPIPE: 13>.
Aug 27 21:36:40 HOSTNAME matrix-backup-borg[4553]: Need some help? https://torsion.org/borgmatic/#issues
Aug 27 21:36:40 HOSTNAME systemd[1]: matrix-backup-borg.service: Main process exited, code=exited, status=1/FAILURE
Aug 27 21:36:40 HOSTNAME systemd[1]: matrix-backup-borg.service: Failed with result 'exit-code'.
Aug 27 21:36:40 HOSTNAME systemd[1]: Failed to start Matrix Borg Backup.
The backup was running for two weeks or so, w/o issues. Lots of space in SSD:
USER@HOSTNAME:~$ df -h
Filesystem Size Used Avail Use% Mounted on
udev 5.9G 0 5.9G 0% /dev
tmpfs 1.2G 1.8M 1.2G 1% /run
/dev/vda3 314G 117G 185G 39% /
tmpfs 5.9G 0 5.9G 0% /dev/shm
tmpfs 5.0M 0 5.0M 0% /run/lock
/dev/vda2 974M 181M 726M 20% /boot
tmpfs 1.2G 0 1.2G 0% /run/user/0
what could cause this behavior?
(Was )working, since #2070 solved. Misconficuration on my side and (at least to me) misleading error message. Solved.
was working 2 or 3 times and since then again - "no space left on device". Could that be a problem resulting from Playbook configuration? Or do I need to dig in the container "shipper"?
Hey,
looks like I found some hints regarding your issue (I had the same before I got a new issue ...):
it looks like we need to replace the folder /tmp/.cache/borg/<ID>/chunks.archive.d/ with a file like these guys did it:
# Fix too large Borg cache
# https://borgbackup.readthedocs.io/en/stable/faq.html#the-borg-cache-eats-way-too-much-disk-space-what-can-i-do
BORG_ID="$(borg config "$BORG_BACKUP_DIRECTORY" id)"
rm -r "/root/.cache/borg/$BORG_ID/chunks.archive.d"
touch "/root/.cache/borg/$BORG_ID/chunks.archive.d"
Infos from https://borgbackup.readthedocs.io
I guess fixing in the source-container is not possible, or? https://gitlab.com/etke.cc/borgmatic/-/blob/main/Dockerfile
Regards
ah interesting. But looks more like a workaround (?)
In a different context (don't remember, where) somebody suggested, to write the cache on a mapped volume instead of writing to the container's RAM which then fills up quickly. Sounds reasonable to me.
absolutely, but even the docs tells it is a log-term-workaround-hack but you are right -->
this is only recommended if you have a fast, low latency connection to your repo (e.g. if repo is local disk)
write the cache to a mapped volume would be much nicer.
UPDARTE: isn't it a volume already, but 100mb are not enought?
--tmpfs=/tmp:rw,noexec,nosuid,size=100m \
regards
Hey,
looks like I found some hints regarding your issue (I had the same before I got a new issue ...):
it looks like we need to replace the folder
/tmp/.cache/borg/<ID>/chunks.archive.d/with a file like these guys did it:# Fix too large Borg cache # https://borgbackup.readthedocs.io/en/stable/faq.html#the-borg-cache-eats-way-too-much-disk-space-what-can-i-do BORG_ID="$(borg config "$BORG_BACKUP_DIRECTORY" id)" rm -r "/root/.cache/borg/$BORG_ID/chunks.archive.d" touch "/root/.cache/borg/$BORG_ID/chunks.archive.d"Infos from https://borgbackup.readthedocs.io
I guess fixing in the source-container is not possible, or? https://gitlab.com/etke.cc/borgmatic/-/blob/main/Dockerfile
Regards
You can always send Merge Request
absolutely, but even the docs tells it is a log-term-workaround-hack but you are right -->
this is only recommended if you have a fast, low latency connection to your repo (e.g. if repo is local disk)
write the cache to a mapped volume would be much nicer.
UPDARTE: isn't it a volume already, but 100mb are not enought?
--tmpfs=/tmp:rw,noexec,nosuid,size=100m \regards
Same here, feel free to send PR
