oom-kill

[michabbs]

I have problem with borg backup while trying to back up ~550.000 of small files. My borg lives inside a lxc container and I get this in dmesg:

[1576137.653081] oom-kill:constraint=CONSTRAINT_MEMCG,nodemask=(null),cpuset=ns,mems_allowed=0,oom_memcg=/lxc/5011,task_memcg=/lxc/5011/ns/system.slice/cron.service,task=borg,pid=2284332,uid=0
[1576137.653138] Memory cgroup out of memory: Killed process 2284332 (borg) total-vm:2418672kB, anon-rss:2003748kB, file-rss:2944kB, shmem-rss:0kB, UID:0 pgtables:4692kB oom_score_adj:0
[1576141.111381] oom_reaper: reaped process 2284332 (borg), now anon-rss:312kB, file-rss:108kB, shmem-rss:0kB

Borg process simply ends with message: [pid] Killed

The container has 2GB of ram and 8GB of swap. Borg seems to ignore swap at all.

In the beginning the container had only 1GB ram and borg was being killed every time during backup process. So I increased ram to 1.5GB and it helped, but after some time, when borg repository grew up, 1.5G became not enough and I increased ram to 2GB. Now 2GB seems to be not enough. What is going on? I suspect borg deduplication cache might take plenty of ram, but... why is swap ignored? (I dont care if backup process is slow, I only want it to be done...)

I have just increased ram to 2.25G and borg works again, but... any ideas how to solve the problem in other way than to again and again increase ram?

Borg version: 1.2.7 (both client and server)
System: Ubuntu 22.04.3 LTS
Hypervisor: Proxmox 8.1.3, Debian 12
Kernel: 6.5.11-7-pve #1 SMP PREEMPT_DYNAMIC PMX 6.5.11-7 (2023-12-05T09:44Z) x86_64 GNU/Linux
Cmdline: borg create --one-file-system --compression=auto,zstd --progress --stats ::[name] .

[ThomasWaldmann]

borg needs enough memory for its hashtables / indexes:

• repo index (server side when running client/server)
• chunks index (client side)
• files cache (client side)

There are some formulas in the docs that might be useful to estimate memory needs.

[ThomasWaldmann]

borg has no means to "ignore swap" - it just uses normal Python and a little C stdlib malloc/free, nothing special.

But I guess you would not want it to significantly use swap (if you have a choice), because that would be rather slow.

So, just give it 8 or 16 Gigs?

[michabbs]

Of course I'd prefer not to use swap than... to use swap. :-) But on the other hand I would obviously like borg used swap instead of silently dying...

Giving 8 gigs is not an option because I do not have 8 gigs... How to reduce client memory usage or force python to swap? Slower backup is not a problem for me. I have time and disk space. I don't have ram. :-)

[ThomasWaldmann]

As I said, borg has no influence on whether it uses swap or not. That's rather a kernel thing.

To reduce memory usage, you can try to have repos with fewer chunks or you can try to reduce the size of the files cache or work without files cache (much slower).

[ThomasWaldmann]

Closing this, seems to be rather an issue of not having enough RAM or a kernel issue.