balena-engine icon indicating copy to clipboard operation
balena-engine copied to clipboard

Default limit for core dumps of balena engine (and thus all its containers) is too high

Open jellyfish-bot opened this issue 4 years ago • 8 comments

[hades32] Several customers had full disks because of too many core dumps. There is no reason to allow more than a few MB. (the number is in 1K units)

root@86fa459:~# grep Limit /lib/systemd/system/balena.service 
LimitNOFILE=1048576
LimitNPROC=1048576
LimitCORE=infinity # this

related: https://github.com/moby/moby/issues/38814

jellyfish-bot avatar Jul 28 '21 08:07 jellyfish-bot

@robertgzr any reason not to change this limit?

Hades32 avatar Jul 28 '21 08:07 Hades32

@Hades32 there's an issue on meta-balena that we should review the systemd service: https://github.com/balena-os/meta-balena/issues/1910

although my plan was to bring the upstream service file (https://github.com/moby/moby/blob/master/contrib/init/systemd/docker.service) into balenaOS, which would make all these limits infinity

robertgzr avatar Aug 12 '21 13:08 robertgzr

@robertgzr but why? Shouldn't we explicitly set it to a low number and NOT to infinity?

Hades32 avatar Aug 12 '21 13:08 Hades32

I can't think of anything against lowering them... maybe that the SV would also be affected, since they apply to all containers. cc @alexgg wdyt regaring ulimits on containers?

robertgzr avatar Aug 12 '21 13:08 robertgzr

@robbyt I realize it was confusing that I added the two other limits in the code snippet above. I was specifically talking about LimitCORE. I don't think we have regular use for core dumps...

Hades32 avatar Aug 12 '21 14:08 Hades32

@robertgzr I think this is something we could add to a developmentMode image and remove in a production image.

alexgg avatar Aug 13 '21 10:08 alexgg

[kb2ma] This issue has attached support thread https://jel.ly.fish/f50e9cf8-d51b-4000-b5ed-84f8cae452cc

jellyfish-bot avatar Apr 04 '22 15:04 jellyfish-bot

I found that this is the issue causing our devices to brick themselves.

Petersdavis avatar Nov 06 '22 23:11 Petersdavis