rsync error: error allocating core memory buffers (code 22) at util2.c(111) [Receiver=3.1.3]

[W11T]

I used to sync the usb drives connected to two Raspberry-PIs 3B+ (1GB mem) using rsync. It looks like since i replaced the sender by a model 4B having 4GB mem rsync fails with:

sending incremental file list ERROR: buffer overflow in recv_rules [Receiver] rsync error: error allocating core memory buffers (code 22) at util2.c(111) [Receiver=3.1.3]

I reduced the files to sync using excludes down to an empty list but still get the same error. Is this possibly an issue of having less memory on the receiver side? Is there an option to handle this?

The command is:

/usr/bin/rsync -e '...' -vvv -auzr <src> <dest>

I tried some other options without any change.

[WayneD]

Don't use -vvv. Use -ii instead.

[W11T]

Don't use -vvv. Use -ii instead.

gives the same error but any messages at all

[WayneD]

Ah, I just noticed that it is an error in recv_rules, which means that an exclude overflowed the "line" buffer, which is BIGPATHBUFLEN. That should be at least 4K of space in a modern rsync. I don't see where you're using excludes, though. You might try specifying --dry-run (-n) and then tweak some options, such as removing -z and/or -u and see if you can figure out what is weird. You may also need to compile a newer rsync.

[WayneD]

If the remote rsync is a daemon, check its rsyncd.conf file to see if it has exclude and/or filter rules. It might be reading in a file with an overly long line in it.

[W11T]

I did not use excludes until i noticed this issue. I started using excludes to trace down the error. But it still appears when i exclude "everthing". I already removed -z and/or -u. No change. I'm not using rsync daemons.

I just tried a simple -a :

sudo /usr/bin/rsync -n -e 'ssh -i ... -l ...' -vvv -ii -a <src> <dest>

resulting in

ERROR: buffer overflow in recv_rules [Receiver] rsync error: error allocating core memory buffers (code 22) at util2.c(111) [Receiver=3.1.3]

... [sender] make_file(...,,2) [sender] pushing local filters for /.../ [sender] make_file(...,,2) ... [sender] make_file(...,*,2) [sender] _exit_cleanup(code=22, file=io.c, line=1642): about to call exit(22) (DRY RUN)

[kot0dama]

I stumbled on the same error messages, and they were caused by a forced command on the other side, not matching what the local side expected.

Might be worth checking if that's your case, if that's still relevant !

[NickNackGus]

I'm hitting the same issue on rsync 3.2.7, though the line number in util2.c is 80 instead of 111. Though in my case, I'm syncing two folders on the same computer. Specifically, I'm trying to delete tens of thousands of folders with small files with rsync -r --delete empty/ empty_me/, as this is the fastest way to delete an excessive number of small files.

I'm also getting this message when syncing files from a remote SSH server, but that's usually a much faster operation, particularly if all the remote files are put in a compressed tar file first, so it takes fewer tries to get working.

[b3]

I am hit by the same issue (also on line 80) on rsync version 3.2.7 protocol version 31 (from Debian bookworm) while trying to mirror two directories on the same host (no ssh involved just plain copy/delete).

The exact message is:

$ rsync -vaH --delete --delete-during --exclude lost+found --stats --log-file=/srv/log/mirror.log --exclude=/storage/video /home/ /media/disque8/b3
sending incremental file list
[generator] exceeded --max-alloc=1073741824 setting (file=ifuncs.h, line=30)
rsync error: error allocating core memory buffers (code 22) at util2.c(80) [generator=3.2.7]
# then deletion of files started but no copy at all

The box on which I try to mirror these two directories is very small (512Mo of RAM on Marvell Kirkwood ARM SoC running at 1.2Ghz) and the number of files very large (backups of around 1,3Tb total size).

The strange thing is that I get no trouble before I upgraded it from Debian bullseye, which mean rsync 3.2.3-4+deb11u1 package.

I am thus not sure if the trouble comes from rsync, something else in my Debian updated install, or simply because the host is not powerful enough now.