Difference between LD_PRELOAD libjemalloc and building with jemalloc

[ksqrtr]

Hello!

Are there any differences between these two setups for ATS 9.2.x and which one is preferable nowadays?

Case 1:

1. CONFIG proxy.config.proxy_binary_opts STRING -M -f -F
2. Traffic server compiled without --with-jemalloc option.
3. LD_PRELOAD=/path/to/libjemalloc.so /path/to/traffic_manager

Case 2:

1. CONFIG proxy.config.proxy_binary_opts STRING -M -f -F
2. Traffic server compiled with --with-jemalloc=/path/to/libjemalloc
3. No LD_PRELOAD stuff is done.

[cmcfarlen]

Hi! Sorry for the delay responding to this issue.

If you configure using --with-jemalloc or --with-mimalloc then ATS will use a custom allocator for allocations when freelists are disabled (with -f). Additionally, with --with-jemalloc, when you send SIGUSR1 ATS will call malloc_stats_print to dump arena stats and will look for a custom format string in proxy.config.memory.malloc_stats_print_opts.

As an aside, the -f option disables freelists, which can definitely improve performance if you have a lot of cores and high transaction rates. On the other hand, -F disables the thread-local object caches which will re-use objects reallocated on the same thread (ProxyAllocators). Turning this off might actually decrease performance and put more pressure on the memory allocators. If you really want to tune ATS, you should run with both flags on/off to see how they impact performance and memory usage.

[ghost]

Hi @cmcfarlen We've also compiled ATS v8.1.7 for Debian using --with-jemalloc=/usr/local/lib. While your answer clarifies the function of -f versus -F, could you also shed some light on whether an ATS build that is compiled with --with-jemalloc=/usr/local/lib also needs an LD_PRELOAD=/usr/lib/x86_64-linux-gnu/libjemalloc.so.2 or not to function correctly with Jemalloc? Also, where can we find examples/samples for the string to put as value for proxy.config.memory.malloc_stats_print_opts?