openproject icon indicating copy to clipboard operation
openproject copied to clipboard

Use jemalloc to prevent memory bloating

Open top4ek opened this issue 1 year ago • 4 comments

Use dynamically linked jemalloc to prevent memory bloating in long living instances.

top4ek avatar Jun 25 '24 17:06 top4ek

Oh, thanks, will look into this. Quick googling gives some problems with arm and ppc platforms so can we use it just for x86? I can try to poke it with RPi4 later and build image accordingly for arm, but not ppc. I'm not sure we need to use unofficial jemalloc enabled ruby images.

top4ek avatar Jul 05 '24 17:07 top4ek

Added support for other architectures. Using USE_JEMALLOC=false enviromnent by default. I was unable to test it on my RPi4(low memory, it just hangs on image building). Also there's opened issues on arm and ppc architectures in jemalloc repo. x86 is battle tested already for couple of weeks on OP and it's reduced memory usage at least twice. (My previous experience at different project with jemalloc allowed us to use 4 times more sidekiq instances on the same hardware limitations). Other architectures MUST be tested, but I don't have access to them.

top4ek avatar Jul 08 '24 13:07 top4ek

I've tested this on ppc64 and double-checked via

MALLOC_CONF=stats_print:true ruby -e "exit"
# prints
#   --- End jemalloc statistics ---
# at the end

to see if ruby is indeed using JEMALLOC which it is. So that's good.

So this PR is fine for x86 and ppc64 at least. And I reckon arm64 will be fine too. This can go ahead if we add the export LD_PRELOAD=... to the web (docker/prod/web) and worker (docker/prod/worker) scripts. Could you please amend that @top4ek?

machisuji avatar Aug 27 '24 11:08 machisuji

LD_PRELOAD export in the web and worker scripts

The problem is that LD_PRELOAD is architecture dependent and it's can't be exported while building in RUN command. /etc/environment and /etc/profile(.d) don't cause any effect as per my tests. I beleive that web and worker scripts are fine, but I'm playing with docker ENTRYPOINT scripts ATM to set variable on start via additionally generated script on preinstall stage.

USE_JEMALLOC to true

Not sure if it's safe to do without tests as because of https://github.com/jemalloc/jemalloc/issues/2358

top4ek avatar Aug 27 '24 11:08 top4ek

Added entrypoint for slim container. Show jemalloc status at admin/info page.

top4ek avatar Sep 17 '24 18:09 top4ek

Thanks for your contribution @top4ek! Merging it (into dev) now. That means it should become available officially with OpenProject 14.6 which is currently scheduled to be released in about two weeks.

machisuji avatar Sep 23 '24 11:09 machisuji