Infinite loop when LD_PRELOADing calloc

[itamarst]

Hi,

I'm trying to use redhook to hook memory allocation functions, including malloc and calloc. The problem is that the hook implementation ends up allocating memory on the heap. This leads to infinite regress—here's part of the stack from gdb:

#1  0x00007f7641d8e86a in redhook::initialized ()
    at /home/itamarst/.cargo/registry/src/github.com-1ecc6299db9ec823/redhook-1.0.0/src/lib.r:
21                                                                                           
#2  0x00007f7641d89484 in calloc (nmemb=1, size=32)
    at <::redhook::ld_preload::hook macros>:24
#3  0x00007f76416ec673 in __cxa_thread_atexit_impl () from /lib64/libc.so.6
#4  0x00007f76414071a2 in std::sys::unix::fast_thread_local::register_dtor ()
    at src/libstd/sys/unix/fast_thread_local.rs:29
#5  std::thread::local::fast::Key<T>::try_register_dtor () at src/libstd/thread/local.rs:444
#6  std::thread::local::fast::Key<T>::try_initialize () at src/libstd/thread/local.rs:430
#7  0x00007f764141fd7e in std::thread::local::fast::Key<T>::get ()
    at src/libstd/thread/local.rs:416
#8  std::sys_common::thread_info::THREAD_INFO::__getit () at src/libstd/thread/local.rs:177
#9  std::thread::local::LocalKey<T>::try_with () at src/libstd/thread/local.rs:259
#10 std::sys_common::thread_info::ThreadInfo::with ()
    at src/libstd/sys_common/thread_info.rs:16
#11 std::sys_common::thread_info::current_thread ()
    at src/libstd/sys_common/thread_info.rs:29
#12 std::thread::current () at src/libstd/thread/mod.rs:634
#13 std::sync::once::Once::call_inner () at src/libstd/sync/once.rs:404
#14 0x00007f7641d8c219 in std::sync::once::Once::call_once (
    self=0x7f7641db2048 <pymemprofile_preload::calloc::get::ONCE>, f=...)
    at /rustc/73528e339aae0f17a15ffa49a8ac608f50c6cf14/src/libstd/sync/once.rs:224
#15 0x00007f7641d89449 in pymemprofile_preload::calloc::get (self=0x7f7641d99017)
    at <::redhook::ld_preload::hook macros>:14
#16 0x00007f7641d8c0aa in pymemprofile_preload::calloc::calloc::{{closure}} ()
    at <::redhook::ld_preload::hook macros>:29
#17 0x00007f7641d89dea in core::option::Option<T>::unwrap_or_else (self=..., f=...)
    at /rustc/73528e339aae0f17a15ffa49a8ac608f50c6cf14/src/libcore/option.rs:419
#18 0x00007f7641d8950e in calloc (nmemb=1, size=32)
    at <::redhook::ld_preload::hook macros>:24
#19 0x00007f76416ec673 in __cxa_thread_atexit_impl () from /lib64/libc.so.6
#20 0x00007f76414071a2 in std::sys::unix::fast_thread_local::register_dtor ()
    at src/libstd/sys/unix/fast_thread_local.rs:29
#21 std::thread::local::fast::Key<T>::try_register_dtor () at src/libstd/thread/local.rs:444
#22 std::thread::local::fast::Key<T>::try_initialize () at src/libstd/thread/local.rs:430
#23 0x00007f764141fd7e in std::thread::local::fast::Key<T>::get ()
    at src/libstd/thread/local.rs:416
#24 std::sys_common::thread_info::THREAD_INFO::__getit () at src/libstd/thread/local.rs:177
...

[itamarst]

The traditional generic solution for this sort of thing is a flag that tells you whether you're reentrantly calling the same function again, and if so not applying the hook. And that's problematic to implement already, but impossible given heap allocation inside the hook implementation.

[itamarst]

I've seen a couple of crates that implement static allocators, I wonder if they would solve my problem without this having to be fixed in redhook. Will go try that.

[itamarst]

1. Updated the traceback with the starting point; apparently it's redhook::initialized().
2. Static allocator doesn't seem to help, probably because it's using libc routines for the threading APIs.

[geofft]

The underlying problem here is described in https://github.com/geofft/redhook/blob/master/src/lib.rs - we're trying to work around another infinite loop by forcing Rust libstd to fully initialize itself in a static constructor. The approach doesn't seem to be working right, though, and anyway Rust switched away from jemalloc, so maybe the right approach is to remove this workaround and bump the MSRV (I think Rust 1.29?), or at least make the workaround conditional on whether jemalloc is in use, somehow.

[itamarst]

Thanks for the quick response. My current thought for my own project is to write the LD_PRELOAD part in C and call into Rust with dlopen/dlsym + RTLD_DEEPBIND, because overriding memory allocation is so fraught anyway.

Thanks for you work on redhook, in any case, this isn't an easy problem!

[geofft]

I've merged the change to remove redhook::initialized, but I think that doesn't actually fix the problem, so reopening.