Copr backend OOMs because of python-daemon inside docker

[nikromen]

Docker doesn't set ulimit, consuming a large amount of RAM. I've seen somebody complain about this in the community - AFAIK this wasn't an issue with podman, so podman sets ulimit?

Still, this should also be an issue on our servers, but it isn't...? it will probably after dnf update

https://bugzilla.redhat.com/show_bug.cgi?id=2307635

[FrostyX]

I've seen somebody complain about this in the community - AFAIK

It was jtate on our Matrix channel, which seems to be the Bugzilla reporter.

Still, this should also be an issue on our servers, but it isn't...? it will probably after dnf update

FTR In my container and on copr-be-dev, there is python3-daemon-2.3.2-4.fc39.noarch which works.

[josephtate]

I did report https://bugzilla.redhat.com/show_bug.cgi?id=2307635. 2.3.2 has the issue still unless it's been patched out. I didn't see any patches when I glanced at the spec file. 3.0.0 is the first version that doesn't create the large set of all possible file descriptors on import.

[josephtate]

BTW, I worked around the issue by using the following resource limits in my docker compose file

    ulimits:
      nproc: 65535
      nofile:
        soft: 32767
        hard: 65535

[praiskup]

We are afraid that our production servers might get affected soon (we should at least check if this is a container-only thing, and perhaps freeze the version if python3-deamon is guilty)

[josephtate]

If you're not in a container, under docker, ulimits are generally reasonable. Fedora 39's hard limit is 524288. If you use this as the size of the set that 'daemon' builds on import, it uses up ~15MB RAM. It's definitely wasted RAM, but it's not an ever growing leak, it's one time only.

[nikromen]

and perhaps freeze the version if python3-deamon is guilty guilty is current version we have, but server seems to be fine

Fedora 39's hard limit is 524288

@josephtate is right here, thanks for looking to it. On our backend:

# ulimit -Hn
524288

so our servers should be fine even with the faulty version, thus this is issue only for our local development inside containers then. Still I think it's worth fixing (idc if the fix would be nasty - e.g. just set ulimit inside dockerfile) since it's waste of time fix it everytime I spawn copr inside container for development :)