Memory leak or similar when deleting lots of files

[rossburton]

I regularly need to delete pretty large directory trees (a few gigabytes) and the internet suggested rsync from an empty directory being faster than rm due to it's more optimised file access patterns.

$ mkdir empty
$ rsync -a --delete empty/ dir/to/delete
[generator] exceeded --max-alloc=1073741824 setting (file=ifuncs.h, line=30)
rsync error: error allocating core memory buffers (code 22) at util2.c(80) [generator=3.2.7]

[realsimix]

Can you try with

--max-alloc=0

Regards, Simon

[rossburton]

I tried with the local tree I wanted to delete and it didn't crash, but watching ps the peak memory usage was just 1117120 so that wasn't big enough.

Isn't using --max-alloc a workaround rather than a fix?

[realsimix]

From how I understand it rsync tries to not use too much memory. Certain operations with large filesystems may require more memory than rsync uses by default and you can therefore specify the limit with the --max-alloc option. If you want rsync to consume as much memory as you have you can set max-alloc to 0.

So, it's not a workaround but just disables rsyncs' habit of being too memory friendly.