Request to consider/include jemalloc to the container to help optimize throughput

[jrehm-mmm]

While working through an issue with a single source that's bursting to 100k events/second over 3 minutes, Bazsi chimed in with the suggestion that jemalloc should help reduce fragmentation, particularly in cases where there are lots of workers.

This is the Slack Channel thread where it was discussed (warning, entire thread is 250+ messages, however this was discussed between Bazsi - "I think I got it." and Bazsi - "I agree, I just never got around to do it."

https://splunk-usergroups.slack.com/archives/CNV918JCQ/p1669837198979099?thread_ts=1668790495.182089&cid=CNV918JCQ

Bazsi quotes

• "it's probably memory fragmentation jemalloc is for the rescue. If we have a high number of workers, the libc allocator does not really scale."
• "you just install jemalloc and set LD_PRELOAD"
• LD_PRELOAD=jemalloc-config --libdir/libjemalloc.so.jemalloc-config --revision app
• "No need to recompile, only to rebuild the image."
• https://github.com/jemalloc/jemalloc/wiki/Getting-Started
• just install the jemalloc library and set LD_PRELOAD so that it links it to the application **- "Sorry I haven't thought about it sooner"
• "I saw this resolve a number of similar issues."**

Ryan F quotes/thoughts

• "it could be configurable I suggest on be default with the ability to turn it off"

[rjha-splunk]

We will review it and prfioritize next quarter.

[MaxwellDPS]

It has been a year, please prioritize adding this as promised, lift is marginal

[bazsi]

This is already in axosyslog (GitHub.com/axoflow/axosyslog) which is the base image for sc4s. So it should be there already,.

Just check the environment variables for the syslog-ng process inside sc4s.

[rjha-splunk]

As @bazsi Mentioned it is already part of SC4S now.