incubator-gluten icon indicating copy to clipboard operation
incubator-gluten copied to clipboard

[VL] Link shared jemalloc lib to work with LD_PRELOAD

Open PHILO-HE opened this issue 1 year ago • 5 comments

What changes were proposed in this pull request?

<>

How was this patch tested?

<>

PHILO-HE avatar Sep 26 '24 17:09 PHILO-HE

Thanks for opening a pull request!

Could you open an issue for this pull request on Github Issues?

https://github.com/apache/incubator-gluten/issues

Then could you also rename commit message and pull request title in the following format?

[GLUTEN-${ISSUES_ID}][COMPONENT]feat/fix: ${detailed message}

See also:

github-actions[bot] avatar Sep 26 '24 17:09 github-actions[bot]

[100%] Linking CXX shared library ../releases/libvelox.so
Checking ld result of libvelox.so
ld: warning: cannot find entry symbol _start; not setting start address
Checking ldd result of libvelox.so
	linux-vdso.so.1 =>  (0x00007ffef44ea000)
	libgluten.so => /__w/incubator-gluten/incubator-gluten/cpp/build/releases/libgluten.so (0x00007f4fb1a8a000)
	libresolv.so.2 => /lib64/libresolv.so.2 (0x00007f4fb1870000)
	libdl.so.2 => /lib64/libdl.so.2 (0x00007f4fb166c000)
	librt.so.1 => /lib64/librt.so.1 (0x00007f4fb1464000)
	libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f4fb1248000)
	libm.so.6 => /lib64/libm.so.6 (0x00007f4fb0f46000)
	libc.so.6 => /lib64/libc.so.6 (0x00007f4fb0b78000)
	/lib64/ld-linux-x86-64.so.2 (0x00007f4fb9bbf000)

Log doesn't show libjemalloc.so as libvelox.so's dependency, which is good to me but is it expected?

zhztheplayer avatar Sep 27 '24 03:09 zhztheplayer

@zhztheplayer, there is an extra option --build_jemalloc (OFF by default) to control whether to link this lib. It's still OFF in CI build.

PHILO-HE avatar Sep 27 '24 07:09 PHILO-HE

@PHILO-HE I have a doubt about the PR title. What does this PR have to do with LD_PRELOAD? All I see is that we're building jemalloc as a shared library in /usr/local/lib

surnaik avatar Oct 26 '24 08:10 surnaik

Enabling this now, will add libjemalloc as a dependency of libvelox. Is this to force users to provide jemalloc using LD_PRELOAD?

surnaik avatar Oct 26 '24 08:10 surnaik

@PHILO-HE I have a doubt about the PR title. What does this PR have to do with LD_PRELOAD? All I see is that we're building jemalloc as a shared library in /usr/local/lib

@surnaik, this pr simply allows some profiling log printed when jemalloc is used with LD_PRELOAD (set by user in runtime env.), e.g., shows memory size allocated by jemalloc. It requires linked jemalloc lib is as same as the one set by LD_PRELOAD.

Enabling this now, will add libjemalloc as a dependency of libvelox. Is this to force users to provide jemalloc using LD_PRELOAD?

Not exactly. With the related build option turned on, it does not require LD_PRELOAD must be set by user. But without the use of LD_PRELOAD, turning on the option is useless since the profiling log has no helpful info (e.g., 0 byte allocated by jemalloc).

PHILO-HE avatar Nov 04 '24 03:11 PHILO-HE