Attempting to Backup Results in MemoryError

[env-media]

Have you checked borgbackup docs, FAQ, and open GitHub issues?

Yes

Is this a BUG / ISSUE report or a QUESTION?

Bug/Issue

System information. For client/server mode post info for both machines.

Synology DS1819+ running DSM 7.1.1-42962 Update 4

Your borg version (borg -V).

borg 1.2.0

Operating system (distribution) and version.

DSM 7.1.1-42962 Update 4

Hardware / network configuration, and filesystems used.

How much data is handled by borg?

Over 15 TB.

Full borg commandline that lead to the problem (leave away excludes and passwords)

time borg create --stats --progress --compression none --exclude "re:@eaDir" /volumeUSB2/usbshare/backup/pvmc/::bkp6 /volume2/Centaur/pvmc/

Describe the problem you're observing.

MemoryError upon running the command.

Can you reproduce the problem? If so, describe how. If not, describe troubleshooting steps you took before opening the issue.

Could before rebooting the system.

Include any warning/errors/backtraces from the system logs

$ time borg create --stats --progress --compression none --exclude "re:@eaDir" /volumeUSB2/usbshare/backup/pvmc/::bkp6 /volume2/Centaur/pvmc/
Enter passphrase for key /volumeUSB2/usbshare/backup/pvmc: 
Local Exception
Traceback (most recent call last):
  File "/volume2/@appstore/borgbackup/env/lib/python3.10/site-packages/borg/archiver.py", line 5089, in main
    exit_code = archiver.run(args)
  File "/volume2/@appstore/borgbackup/env/lib/python3.10/site-packages/borg/archiver.py", line 5020, in run
    return set_ec(func(args))
  File "/volume2/@appstore/borgbackup/env/lib/python3.10/site-packages/borg/archiver.py", line 183, in wrapper
    return method(self, args, repository=repository, **kwargs)
  File "/volume2/@appstore/borgbackup/env/lib/python3.10/site-packages/borg/archiver.py", line 649, in do_create
    with Cache(repository, key, manifest, progress=args.progress,
  File "/volume2/@appstore/borgbackup/env/lib/python3.10/site-packages/borg/cache.py", line 383, in __new__
    return local()
  File "/volume2/@appstore/borgbackup/env/lib/python3.10/site-packages/borg/cache.py", line 374, in local
    return LocalCache(repository=repository, key=key, manifest=manifest, path=path, sync=sync,
  File "/volume2/@appstore/borgbackup/env/lib/python3.10/site-packages/borg/cache.py", line 483, in __init__
    self.open()
  File "/volume2/@appstore/borgbackup/env/lib/python3.10/site-packages/borg/cache.py", line 538, in open
    self.rollback()
  File "/volume2/@appstore/borgbackup/env/lib/python3.10/site-packages/borg/cache.py", line 658, in rollback
    self._do_open()
  File "/volume2/@appstore/borgbackup/env/lib/python3.10/site-packages/borg/cache.py", line 523, in _do_open
    self.chunks = ChunkIndex.read(fd)
  File "src/borg/hashindex.pyx", line 114, in borg.hashindex.IndexBase.read
  File "src/borg/hashindex.pyx", line 99, in borg.hashindex.IndexBase.__cinit__
  File "/volume2/@appstore/borgbackup/env/lib/python3.10/site-packages/borg/crypto/file_integrity.py", line 32, in read
    return self.fd.read(n)
  File "/volume2/@appstore/borgbackup/env/lib/python3.10/site-packages/borg/crypto/file_integrity.py", line 82, in read
    data = super().read(n)
  File "/volume2/@appstore/borgbackup/env/lib/python3.10/site-packages/borg/crypto/file_integrity.py", line 32, in read
    return self.fd.read(n)
MemoryError

Platform: Linux hawking 4.4.180+ #42962 SMP Fri Feb 10 16:34:15 CST 2023 x86_64
Linux: Unknown Linux
Borg: 1.2.0  Python: CPython 3.10.4 msgpack: 1.0.3 fuse: None [pyfuse3,llfuse]
PID: 21958  CWD: /volume2/homes/admin
sys.argv: ['/usr/local/bin/borg', 'create', '--stats', '--progress', '--compression', 'none', '--exclude', 're:@eaDir', '/volumeUSB2/usbshare/backup/pvmc/::bkp6', '/volume2/Centaur/pvmc/']
SSH_ORIGINAL_COMMAND: None


real	0m12.469s
user	0m1.063s
sys	0m0.568s

[ThomasWaldmann]

MemoryError means that borg (Python) ran out of memory (RAM).

You can either add RAM or swap (in the latter case it will be very slow).

Also, you could temporarily stop some other services (or even VMs / containers) to free RAM, so it is available for borg.

Memory consumption of borg depends on many factors, see the docs and the issue tracker for hints.

[ThomasWaldmann]

Closing this, borg needs enough RAM.