memray icon indicating copy to clipboard operation
memray copied to clipboard

Potential deadlock when interposing symbols

Open sethtroisi opened this issue 6 months ago • 2 comments

Is there an existing issue for this?

  • [X] I have searched the existing issues

Current Behavior

I am seeing deadlock during dl_iterate_phdr similar to https://github.com/bloomberg/memray/commit/6fbadf72b66da53f1ac56ce2f505f79d6e2dbb2a

By adding a hack in the same place to avoid patching Smemray_Slib_Umemray_Ucc_Ulib.so everything works.

    if ( strlen(info->dlpi_name) < 1 || 
        strstr(info->dlpi_name, "memray_") || // <- new hack
        strstr(info->dlpi_name, "/ld-linux") ||
        strstr(info->dlpi_name, "/ld-musl") ||
        strstr(info->dlpi_name, "linux-vdso.so.1")) {
      // Avoid chaos by not overwriting the symbols in the linker.
      // TODO: Don't override the symbols in our shared library!
        LOG(WARNING) << "P1 " << info->dlpi_name;
      return 0;
    }

I'm not sure how to guess what the dlpi_name will be. But memray_ alone is dangerous because it's appearing in the path portion of the dlpi_name. Maybe checking that it's in the filename after any slashes?

The stack trace when the tests time out is with the dl being either libunwind.so or libcxxabi_Slibcxxabi_Ucc_Ulibrary.so

*** SIGTERM received by PID 3156388 (TID 3156388) on cpu 3 from PID 3156373; stack trace: ***
PC: @     0x7f38e0830a9a  (unknown)  memray::intercept::free()
    @     0x7f38ab409df1       1904  FailureSignalHandler()
    @     0x7f38fdd36e80  2138701600  (unknown)
    @     0x7f3865e9b3eb         32  std::__u::ios_base::~ios_base()
    @     0x7f38e082a818         80  memray::LOG::~LOG()
    @     0x7f38e082b1f7        368  memray::linker::patch_symbols()
    @     0x7f38e082abdd        400  memray::linker::phdrs_callback()
    @     0x7f38fd1c3a7d        160  dl_iterate_phdr
    @     0x7f38e082a666        352  memray::linker::SymbolPatcher::overwrite_symbols()
    @     0x7f38e085a61e        112  memray::tracking_api::Tracker::Tracker()
    @     0x7f38e085c3c2         64  memray::tracking_api::Tracker::createTracker()
    @     0x7f386442e946         80  __pyx_pw_6memray_7_memray_7Tracker_3__enter__()
    @     0x55d5139afe33         48  cfunction_vectorcall_NOARGS
    @     0x55d513a8ae39        384  _PyEval_EvalFrameDefault

Expected Behavior

No response

Steps To Reproduce

  1. Run memray with a Tracer

Memray Version

1.7.0

Python Version

3.11

Operating System

Linux

Anything else?

No response

sethtroisi avatar Aug 08 '24 23:08 sethtroisi