memray
memray copied to clipboard
bloomberg
Attaching to a running compiled & cythonized app
I'm not sure if this is supported, but I have an application that is cythonized (though there are not C extensions written) and then is compiled to a single binary using pyinstaller.
I've tried attaching memray with lldb and gdb, but both fail. Let me know if I'm trying to achieve something that won't work for sure or maybe I'm doing something wrong (maye missing something?).
LLDB:
~ $ doas python -m memray attach 13599 --verbose
sys.remote_exec is not available in this Python version
No gdb executable found
Debugger command line: lldb --batch -p 13599 --no-lldbinit -o 'expr char $libpath[]="/usr/lib/python3.11/site-packages/memray/_inject.abi3.so"' -o 'expr int $port=48907' -o 'expr void* $rtld_default=(void*)0' -o 'expr int $rtld_now=2' --source /usr/lib/python3.11/site-packages/memray/commands/_attach.lldb
debugger return code: 0
debugger output:
(lldb) process attach --pid 13599
Process 13599 stopped
* thread #1, name = 'my-compiled-app', stop reason = signal SIGSTOP
frame #0: 0x00007ff5c098b998 ld-musl-x86_64.so.1
-> 0x7ff5c098b998: retq
0x7ff5c098b999: jmp 0x7ff5c098859f
0x7ff5c098b99e: pushq %r12
0x7ff5c098b9a0: movl $0x2, %eax
thread #2, name = 'my-compiled-app', stop reason = signal SIGSTOP
frame #0: 0x00007ff5c098b998 ld-musl-x86_64.so.1
-> 0x7ff5c098b998: retq
0x7ff5c098b999: jmp 0x7ff5c098859f
0x7ff5c098b99e: pushq %r12
0x7ff5c098b9a0: movl $0x2, %eax
thread #3, name = 'my-compiled-app', stop reason = signal SIGSTOP
frame #0: 0x00007ff5c098b998 ld-musl-x86_64.so.1
-> 0x7ff5c098b998: retq
0x7ff5c098b999: jmp 0x7ff5c098859f
0x7ff5c098b99e: pushq %r12
0x7ff5c098b9a0: movl $0x2, %eax
thread #4, name = 'my-compiled-app', stop reason = signal SIGSTOP
frame #0: 0x00007ff5c098b998 ld-musl-x86_64.so.1
-> 0x7ff5c098b998: retq
0x7ff5c098b999: jmp 0x7ff5c098859f
0x7ff5c098b99e: pushq %r12
0x7ff5c098b9a0: movl $0x2, %eax
thread #5, name = 'my-compiled-app', stop reason = signal SIGSTOP
frame #0: 0x00007ff5c098b998 ld-musl-x86_64.so.1
-> 0x7ff5c098b998: retq
0x7ff5c098b999: jmp 0x7ff5c098859f
0x7ff5c098b99e: pushq %r12
0x7ff5c098b9a0: movl $0x2, %eax
thread #6, name = 'my-compiled-app', stop reason = signal SIGSTOP
frame #0: 0x00007ff5c098b998 ld-musl-x86_64.so.1
-> 0x7ff5c098b998: retq
0x7ff5c098b999: jmp 0x7ff5c098859f
0x7ff5c098b99e: pushq %r12
0x7ff5c098b9a0: movl $0x2, %eax
thread #7, name = 'my-compiled-app', stop reason = signal SIGSTOP
frame #0: 0x00007ff5c098b998 ld-musl-x86_64.so.1
-> 0x7ff5c098b998: retq
0x7ff5c098b999: jmp 0x7ff5c098859f
0x7ff5c098b99e: pushq %r12
0x7ff5c098b9a0: movl $0x2, %eax
thread #8, name = 'my-compiled-app', stop reason = signal SIGSTOP
frame #0: 0x00007ff5c098b998 ld-musl-x86_64.so.1
-> 0x7ff5c098b998: retq
0x7ff5c098b999: jmp 0x7ff5c098859f
0x7ff5c098b99e: pushq %r12
0x7ff5c098b9a0: movl $0x2, %eax
thread #9, name = 'my-compiled-app', stop reason = signal SIGSTOP
frame #0: 0x00007ff5c098b998 ld-musl-x86_64.so.1
-> 0x7ff5c098b998: retq
0x7ff5c098b999: jmp 0x7ff5c098859f
0x7ff5c098b99e: pushq %r12
0x7ff5c098b9a0: movl $0x2, %eax
thread #10, name = 'my-compiled-app', stop reason = signal SIGSTOP
frame #0: 0x00007ff5c098b998 ld-musl-x86_64.so.1
-> 0x7ff5c098b998: retq
0x7ff5c098b999: jmp 0x7ff5c098859f
0x7ff5c098b99e: pushq %r12
0x7ff5c098b9a0: movl $0x2, %eax
thread #11, name = 'my-compiled-app', stop reason = signal SIGSTOP
frame #0: 0x00007ff5c098b998 ld-musl-x86_64.so.1
-> 0x7ff5c098b998: retq
0x7ff5c098b999: jmp 0x7ff5c098859f
0x7ff5c098b99e: pushq %r12
0x7ff5c098b9a0: movl $0x2, %eax
thread #12, name = 'my-compiled-app', stop reason = signal SIGSTOP
frame #0: 0x00007ff5c098b998 ld-musl-x86_64.so.1
-> 0x7ff5c098b998: retq
0x7ff5c098b999: jmp 0x7ff5c098859f
0x7ff5c098b99e: pushq %r12
0x7ff5c098b9a0: movl $0x2, %eax
thread #13, name = 'my-compiled-app', stop reason = signal SIGSTOP
frame #0: 0x00007ff5c098b998 ld-musl-x86_64.so.1
-> 0x7ff5c098b998: retq
0x7ff5c098b999: jmp 0x7ff5c098859f
0x7ff5c098b99e: pushq %r12
0x7ff5c098b9a0: movl $0x2, %eax
Executable module set to "/usr/bin/my-compiled-app".
Architecture set to: x86_64-unknown-linux-musl.
(lldb) expr char $libpath[]="/usr/lib/python3.11/site-packages/memray/_inject.abi3.so"
(lldb) expr int $port=48907
(lldb) expr void* $rtld_default=(void*)0
(lldb) expr int $rtld_now=2
(lldb) command source -s 0 '/usr/lib/python3.11/site-packages/memray/commands/_attach.lldb'
Executing commands in '/usr/lib/python3.11/site-packages/memray/commands/_attach.lldb'.
(lldb) p "MEMRAY: Attached to process."
(const char[29]) $0 = "MEMRAY: Attached to process."
(lldb) p "MEMRAY: Checking if process is Python 3.7+."
(const char[44]) $1 = "MEMRAY: Checking if process is Python 3.7+."
(lldb) p ((void*(*)(size_t))PyMem_Malloc)
(void *(*)(size_t)) $2 = 0x00007ff5c0412f60 (libpython3.11.so.1.0`PyMem_Malloc)
(lldb) p ((void*(*)(size_t, size_t))PyMem_Calloc)
(void *(*)(size_t, size_t)) $3 = 0x00007ff5c0413060 (libpython3.11.so.1.0`PyMem_Calloc)
(lldb) p ((void*(*)(void *, size_t))PyMem_Realloc)
(void *(*)(void *, size_t)) $4 = 0x00007ff5c041b700 (libpython3.11.so.1.0`PyMem_Realloc)
(lldb) p ((void(*)(void*))PyMem_Free)
(void (*)(void *)) $5 = 0x00007ff5c0413220 (libpython3.11.so.1.0`PyMem_Free)
(lldb) p "MEMRAY: Process is Python 3.7+."
(const char[32]) $6 = "MEMRAY: Process is Python 3.7+."
(lldb) # When adding new breakpoints, also update _attach.gdb
(lldb) breakpoint set -b malloc -b calloc -b realloc -b free -b PyMem_Malloc -b PyMem_Calloc -b PyMem_Realloc -b PyMem_Free
Breakpoint 1: 8 locations.
(lldb) # Set commands to execute when breakpoint is reached
(lldb) breakpoint command add -e true
(lldb) continue
(lldb) breakpoint disable
All breakpoints disabled. (1 breakpoints)
(lldb) expr auto $dlsym = (void* (*)(void*, const char*))&::dlsym
(lldb) expr auto $dlopen = $dlsym($rtld_default, "dlopen")
(lldb) expr auto $dlerror = $dlsym($rtld_default, "dlerror")
(lldb) expr auto $dll = ((void*(*)(const char*, int))$dlopen)($libpath, $rtld_now)
(lldb) p ((char*(*)(void))$dlerror)()
(char *) $7 = 0x0000000000000000
(lldb) expr auto $spawn = $dlsym($dll, "memray_spawn_client")
(lldb) p ((int(*)(int))$spawn)($port)?"FAILURE":"SUCCESS"
(const char[8]) $8 = "SUCCESS"
Process 13599 resuming
Process 13599 stopped
* thread #1, name = 'my-compiled-app', stop reason = breakpoint 1.
frame #0: 0x00007ff5c0413220 libpython3.11.so.1.0`PyMem_Free
libpython3.11.so.1.0`PyMem_Free:
-> 0x7ff5c0413220 <+0>: movq 0x1f3239(%rip), %rax
0x7ff5c0413227 <+7>: leaq -0x3ece(%rip), %rdx ; ___lldb_unnamed_symbol6645
0x7ff5c041322e <+14>: movq 0x1f320b(%rip), %rcx
0x7ff5c0413235 <+21>: cmpq %rdx, %rax
Timed out waiting for connection from pid 13599
GDB:
~ $ doas python -m memray attach 13599 --verbose --method gdb
Debugger command line: gdb -batch -p 13599 -nx -nw '-iex=set auto-solib-add off' '-ex=set $rtld_now=2' '-ex=set $libpath="/usr/lib/python3.11/site-packages/memray/_inject.abi3.so"' '-ex=set $port=60917' -x=/usr/lib/python3.11/site-packages/memray/commands/_attach.gdb
debugger return code: 0
debugger output:
[New LWP 13619]
[New LWP 13621]
[New LWP 13711]
[New LWP 13712]
[New LWP 13713]
[New LWP 13714]
[New LWP 13715]
[New LWP 13716]
[New LWP 13717]
[New LWP 13719]
[New LWP 13751]
[New LWP 15190]
0x00007ff5c04293b6 in ?? () from /home/my_user/.tmp/_MEIhhcmlJ/libpython3.11.so.1.0
$1 = "MEMRAY: Attached to process."
From To Syms Read Shared Object Library
0x00007ff5c0916010 0x00007ff5c0923dcf No /lib/libz.so.1
0x00007ff5c0941070 0x00007ff5c09941f3 Yes (*) /lib/ld-musl-x86_64.so.1
0x00007ff5c02ee650 0x00007ff5c04e290e Yes (*) /home/my_user/.tmp/_MEIhhcmlJ/libpython3.11.so.1.0
0x00007ff5c00cd4e0 0x00007ff5c00d0105 Yes (*) /home/my_user/.tmp/_MEIhhcmlJ/lib-dynload/_struct.cpython-311-x86_64-linux-musl.so
0x00007ff5c00ae3a0 0x00007ff5c00b1265 Yes (*) /home/my_user/.tmp/_MEIhhcmlJ/lib-dynload/zlib.cpython-311-x86_64-linux-musl.so
0x00007ff5bff54a80 0x00007ff5bff5fc85 Yes (*) /home/my_user/.tmp/_MEIhhcmlJ/lib-dynload/_ctypes.cpython-311-x86_64-linux-musl.so
0x00007ff5c0094030 0x00007ff5c0098816 No /usr/lib/libffi.so.8
0x00007ff5bfd8d0d0 0x00007ff5bfd8d41c Yes (*) /home/my_user/.tmp/_MEIhhcmlJ/lib-dynload/_opcode.cpython-311-x86_64-linux-musl.so
0x00007ff5bfd37240 0x00007ff5bfd38ec5 Yes (*) /home/my_user/.tmp/_MEIhhcmlJ/lib-dynload/binascii.cpython-311-x86_64-linux-musl.so
0x00007ff5bfd192f0 0x00007ff5bfd1a2c5 Yes (*) /home/my_user/.tmp/_MEIhhcmlJ/lib-dynload/_bz2.cpython-311-x86_64-linux-musl.so
0x00007ff5bfd062e0 0x00007ff5bfd127cf No /usr/lib/libbz2.so.1
0x00007ff5bfbf44c0 0x00007ff5bfbf6865 Yes (*) /home/my_user/.tmp/_MEIhhcmlJ/lib-dynload/_lzma.cpython-311-x86_64-linux-musl.so
0x00007ff5bfbbc010 0x00007ff5bfbe15a3 No /usr/lib/liblzma.so.5
0x00007ff5bfb61870 0x00007ff5bfb6c1a9 Yes (*) /home/my_user/.tmp/_MEIhhcmlJ/lib-dynload/_pickle.cpython-311-x86_64-linux-musl.so
0x00007ff5bfa39830 0x00007ff5bfa41d26 Yes (*) /home/my_user/.tmp/_MEIhhcmlJ/lib-dynload/_socket.cpython-311-x86_64-linux-musl.so
0x00007ff5bfa1d4e0 0x00007ff5bfa219c2 Yes (*) /home/my_user/.tmp/_MEIhhcmlJ/lib-dynload/math.cpython-311-x86_64-linux-musl.so
0x00007ff5bfa13400 0x00007ff5bfa152a0 Yes (*) /home/my_user/.tmp/_MEIhhcmlJ/lib-dynload/select.cpython-311-x86_64-linux-musl.so
0x00007ff5bfa02590 0x00007ff5bfa06445 Yes (*) /home/my_user/.tmp/_MEIhhcmlJ/lib-dynload/array.cpython-311-x86_64-linux-musl.so
0x00007ff5bf9cf1d0 0x00007ff5bf9d04a5 Yes (*) /home/my_user/.tmp/_MEIhhcmlJ/lib-dynload/fcntl.cpython-311-x86_64-linux-musl.so
0x00007ff5bf9c9410 0x00007ff5bf9ca785 Yes (*) /home/my_user/.tmp/_MEIhhcmlJ/lib-dynload/_posixsubprocess.cpython-311-x86_64-linux-musl.so
0x00007ff5bfdb2050 0x00007ff5bfdb2175 Yes (*) /home/my_user/.tmp/_MEIhhcmlJ/lib-dynload/_typing.cpython-311-x86_64-linux-musl.so
0x00007ff5bfd9b160 0x00007ff5bfd9bc65 Yes (*) /home/my_user/.tmp/_MEIhhcmlJ/lib-dynload/_bisect.cpython-311-x86_64-linux-musl.so
0x00007ff5bf9a7270 0x00007ff5bf9a7ce5 Yes (*) /home/my_user/.tmp/_MEIhhcmlJ/lib-dynload/_random.cpython-311-x86_64-linux-musl.so
0x00007ff5bf8311a0 0x00007ff5bf834595 Yes (*) /home/my_user/.tmp/_MEIhhcmlJ/lib-dynload/_sha512.cpython-311-x86_64-linux-musl.so
0x00007ff5bf7bd0e0 0x00007ff5bf7bdb65 Yes (*) /home/my_user/.tmp/_MEIhhcmlJ/lib-dynload/_heapq.cpython-311-x86_64-linux-musl.so
0x00007ff5bf7af260 0x00007ff5bf7afbd5 Yes (*) /home/my_user/.tmp/_MEIhhcmlJ/lib-dynload/_queue.cpython-311-x86_64-linux-musl.so
0x00007ff5bf690220 0x00007ff5bf690995 Yes (*) /home/my_user/.tmp/_MEIhhcmlJ/lib-dynload/grp.cpython-311-x86_64-linux-musl.so
0x00007ff5bf4fc670 0x00007ff5bf4fffd5 Yes (*) /home/my_user/.tmp/_MEIhhcmlJ/lib-dynload/_hashlib.cpython-311-x86_64-linux-musl.so
0x00007ff5bf04c000 0x00007ff5bf30cbf7 Yes (*) /lib/libcrypto.so.3
0x00007ff5bf4d3260 0x00007ff5bf4d7cd5 Yes (*) /home/my_user/.tmp/_MEIhhcmlJ/lib-dynload/_blake2.cpython-311-x86_64-linux-musl.so
0x00007ff5bf486590 0x00007ff5bf48f436 Yes (*) /home/my_user/.tmp/_MEIhhcmlJ/lib-dynload/_datetime.cpython-311-x86_64-linux-musl.so
0x00007ff5beedf430 0x00007ff5beee8bc5 Yes (*) /home/my_user/.tmp/_MEIhhcmlJ/lib-dynload/_ssl.cpython-311-x86_64-linux-musl.so
0x00007ff5bee1b010 0x00007ff5bee8e137 No /lib/libssl.so.3
0x00007ff5bed973c0 0x00007ff5bed99755 Yes (*) /home/my_user/.tmp/_MEIhhcmlJ/lib-dynload/_csv.cpython-311-x86_64-linux-musl.so
0x00007ff5beae5590 0x00007ff5beae8e85 Yes (*) /home/my_user/.tmp/_MEIhhcmlJ/lib-dynload/pyexpat.cpython-311-x86_64-linux-musl.so
0x00007ff5beac1180 0x00007ff5bead600f No /usr/lib/libexpat.so.1
0x00007ff5bea9a500 0x00007ff5beaa5e7e Yes /home/my_user/.tmp/_MEIhhcmlJ/main.cpython-311-x86_64-linux-musl.so
0x00007ff5be9523e0 0x00007ff5be955f75 Yes (*) /home/my_user/.tmp/_MEIhhcmlJ/lib-dynload/_json.cpython-311-x86_64-linux-musl.so
0x00007ff5be2494e0 0x00007ff5be5336af Yes (*) /home/my_user/.tmp/_MEIhhcmlJ/pydantic_core/_pydantic_core.cpython-311-x86_64-linux-musl.so
0x00007ff5bde02dd0 0x00007ff5bde12ec5 No /home/my_user/.tmp/_MEIhhcmlJ/pydantic_core/../pydantic_core.libs/libgcc_s-1e52349c.so.1
0x00007ff5be8cc600 0x00007ff5be8e9a20 Yes (*) /home/my_user/.tmp/_MEIhhcmlJ/lib-dynload/_decimal.cpython-311-x86_64-linux-musl.so
0x00007ff5be1e4070 0x00007ff5be1e41e3 Yes (*) /home/my_user/.tmp/_MEIhhcmlJ/lib-dynload/_contextvars.cpython-311-x86_64-linux-musl.so
0x00007ff5be071090 0x00007ff5be071227 Yes (*) /home/my_user/.tmp/_MEIhhcmlJ/lib-dynload/_uuid.cpython-311-x86_64-linux-musl.so
0x00007ff5be061010 0x00007ff5be0643c0 No /lib/libuuid.so.1
0x00007ff5bd8862a0 0x00007ff5bd886f93 Yes (*) /home/my_user/.tmp/_MEIhhcmlJ/lib-dynload/_multiprocessing.cpython-311-x86_64-linux-musl.so
0x00007ff5bd73c2d0 0x00007ff5bd73d0b5 Yes (*) /home/my_user/.tmp/_MEIhhcmlJ/lib-dynload/termios.cpython-311-x86_64-linux-musl.so
0x00007ff5bd16b540 0x00007ff5bd16f775 Yes (*) /home/my_user/.tmp/_MEIhhcmlJ/lib-dynload/_asyncio.cpython-311-x86_64-linux-musl.so
0x00007ff5bcfdfb30 0x00007ff5bcfee987 Yes /home/my_user/.tmp/_MEIhhcmlJ/greenlet/_greenlet.cpython-311-x86_64-linux-musl.so
0x00007ff5bccbf978 0x00007ff5bcd41bd6 No /home/my_user/.tmp/_MEIhhcmlJ/greenlet/../greenlet.libs/libstdc++-a9383cce.so.6.0.28
0x00007ff5bcfc42b8 0x00007ff5bcfcf5fe No /home/my_user/.tmp/_MEIhhcmlJ/greenlet/../greenlet.libs/libgcc_s-a04fdf82.so.1
0x00007ff5bc802600 0x00007ff5bc830580 Yes (*) /home/my_user/.tmp/_MEIhhcmlJ/orjson/orjson.cpython-311-x86_64-linux-musl.so
0x00007ff5bcf85330 0x00007ff5bcf87e6c Yes (*) /home/my_user/.tmp/_MEIhhcmlJ/lib-dynload/_zoneinfo.cpython-311-x86_64-linux-musl.so
0x00007ff5bcf4d440 0x00007ff5bcf4f7e4 No /home/my_user/.tmp/_MEIhhcmlJ/psutil/_psutil_linux.abi3.so
0x00007ff5bcf46280 0x00007ff5bcf47947 No /home/my_user/.tmp/_MEIhhcmlJ/psutil/_psutil_posix.abi3.so
0x00007ff5bcf381f0 0x00007ff5bcf38ca5 Yes (*) /home/my_user/.tmp/_MEIhhcmlJ/lib-dynload/resource.cpython-311-x86_64-linux-musl.so
0x00007ff5bc64ba00 0x00007ff5bc7a7767 Yes /home/my_user/.tmp/_MEIhhcmlJ/uvloop/loop.cpython-311-x86_64-linux-musl.so
0x00007ff5bcf09610 0x00007ff5bcf0d250 Yes /home/my_user/.tmp/_MEIhhcmlJ/_yappi.cpython-311-x86_64-linux-musl.so
0x00007ff5bbab3940 0x00007ff5bc0f99d1 No /home/my_user/.tmp/_MEIhhcmlJ/cryptography/hazmat/bindings/_rust.abi3.so
0x00007ff5bced0040 0x00007ff5bcef1d4d No /home/my_user/.tmp/_MEIhhcmlJ/cryptography/hazmat/bindings/../../../cryptography.libs/libgcc_s-2298274a.so.1
0x00007ff5bcac7000 0x00007ff5bcaeb133 Yes /home/my_user/.tmp/_MEIhhcmlJ/_cffi_backend.cpython-311-x86_64-linux-musl.so
0x00007ff5bc37b4a0 0x00007ff5bc3848ef Yes /home/my_user/.tmp/_MEIhhcmlJ/multidict/_multidict.cpython-311-x86_64-linux-musl.so
0x00007ff5bb8bf7c0 0x00007ff5bb8d6f8f Yes /home/my_user/.tmp/_MEIhhcmlJ/yarl/_quoting_c.cpython-311-x86_64-linux-musl.so
0x00007ff5bb74c2d0 0x00007ff5bb74e8c6 Yes (*) /home/my_user/.tmp/_MEIhhcmlJ/lib-dynload/unicodedata.cpython-311-x86_64-linux-musl.so
0x00007ff5bcf8f6a0 0x00007ff5bcf9efa8 Yes /home/my_user/.tmp/_MEIhhcmlJ/propcache/_helpers_c.cpython-311-x86_64-linux-musl.so
0x00007ff5bb650300 0x00007ff5bb6b356c Yes /home/my_user/.tmp/_MEIhhcmlJ/_brotli.cpython-311-x86_64-linux-musl.so
0x00007ff5bb5395a0 0x00007ff5bb541d6d Yes /home/my_user/.tmp/_MEIhhcmlJ/aiohttp/_http_writer.cpython-311-x86_64-linux-musl.so
0x00007ff5bb4c9fc0 0x00007ff5bb516f95 Yes /home/my_user/.tmp/_MEIhhcmlJ/aiohttp/_http_parser.cpython-311-x86_64-linux-musl.so
0x00007ff5bb4af480 0x00007ff5bb4b29c9 Yes /home/my_user/.tmp/_MEIhhcmlJ/aiohttp/_websocket/mask.cpython-311-x86_64-linux-musl.so
0x00007ff5bb47e920 0x00007ff5bb4a09d0 Yes /home/my_user/.tmp/_MEIhhcmlJ/aiohttp/_websocket/reader_c.cpython-311-x86_64-linux-musl.so
0x00007ff5bb2dda80 0x00007ff5bb2f5377 Yes /home/my_user/.tmp/_MEIhhcmlJ/pycares/_cares.cpython-311-x86_64-linux-musl.so
0x00007ff5bb103750 0x00007ff5bb116a40 Yes /home/my_user/.tmp/_MEIhhcmlJ/frozenlist/_frozenlist.cpython-311-x86_64-linux-musl.so
0x00007ff5b956f450 0x00007ff5b9572244 Yes /home/my_user/.tmp/_MEIhhcmlJ/pytricia.cpython-311-x86_64-linux-musl.so
0x00007ff5b954b1a0 0x00007ff5b954c68b Yes /home/my_user/.tmp/_MEIhhcmlJ/lru/_lru.cpython-311-x86_64-linux-musl.so
0x00007ff5b951c3a0 0x00007ff5b951e0b5 Yes (*) /home/my_user/.tmp/_MEIhhcmlJ/lib-dynload/mmap.cpython-311-x86_64-linux-musl.so
0x00007ff5b90aa6b0 0x00007ff5b90b0792 Yes (*) /home/my_user/.tmp/_MEIhhcmlJ/lib-dynload/_elementtree.cpython-311-x86_64-linux-musl.so
0x00007ff5b62ea060 0x00007ff5b62ea1d5 Yes /home/my_user/.tmp/_MEIhhcmlJ/charset_normalizer/md.cpython-311-x86_64-linux-musl.so
0x00007ff5b62ba0a0 0x00007ff5b62d63b2 Yes /home/my_user/.tmp/_MEIhhcmlJ/charset_normalizer/md__mypyc.cpython-311-x86_64-linux-musl.so
0x00007ff5b62a44f0 0x00007ff5b62a7645 Yes (*) /home/my_user/.tmp/_MEIhhcmlJ/lib-dynload/_multibytecodec.cpython-311-x86_64-linux-musl.so
0x00007ff5b3dc3160 0x00007ff5b3dc359c Yes (*) /home/my_user/.tmp/_MEIhhcmlJ/lib-dynload/_posixshmem.cpython-311-x86_64-linux-musl.so
0x00007ff5b2e91300 0x00007ff5b2e923a0 No /usr/lib/python3.11/site-packages/memray/_inject.abi3.so
0x00007ff5b14b0040 0x00007ff5b15fb2ff No /usr/lib/python3.11/site-packages/memray/../memray.libs/libstdc++-5d72f927.so.6.0.33
0x00007ff5b232f040 0x00007ff5b2350d4d No /usr/lib/python3.11/site-packages/memray/../memray.libs/libgcc_s-0cd532bd.so.1
(*): Shared library is missing debugging information.
$2 = "MEMRAY: Checking if process is Python 3.7+."
$3 = {<text variable, no debug info>} 0x7ff5c0412f60 <PyMem_Malloc>
$4 = {<text variable, no debug info>} 0x7ff5c0413060 <PyMem_Calloc>
$5 = {<text variable, no debug info>} 0x7ff5c041b700 <PyMem_Realloc>
$6 = {<text variable, no debug info>} 0x7ff5c0413220 <PyMem_Free>
$7 = "MEMRAY: Process is Python 3.7+."
$8 = 0
Breakpoint 1 at 0x7ff5c09514f4
Breakpoint 2 at 0x7ff5c095135c
Breakpoint 3 at 0x7ff5c09537a6
Breakpoint 4 at 0x7ff5c0951423
Breakpoint 5 at 0x7ff5c0412f60
Breakpoint 6 at 0x7ff5c0413060
Breakpoint 7 at 0x7ff5c041b700
Breakpoint 8 at 0x7ff5c0413220
Breakpoint 9 at 0x7ff5c044e2b0
Breakpoint 10 at 0x7ff5c048f010
Thread 1 "my-compiled-app" hit Breakpoint 10, 0x00007ff5c048f010 in PyCallable_Check () from /home/my_user/.tmp/_MEIhhcmlJ/libpython3.11.so.1.0
#0 0x00007ff5c048f010 in PyCallable_Check () from /home/my_user/.tmp/_MEIhhcmlJ/libpython3.11.so.1.0
#1 0x00007ff5c0344c16 in ?? () from /home/my_user/.tmp/_MEIhhcmlJ/libpython3.11.so.1.0
#2 0x00007ff5c030528f in ?? () from /home/my_user/.tmp/_MEIhhcmlJ/libpython3.11.so.1.0
#3 0x00007ff5c0445221 in _PyFunction_Vectorcall () from /home/my_user/.tmp/_MEIhhcmlJ/libpython3.11.so.1.0
#4 0x00007ff5beaa49b7 in __Pyx_PyObject_FastCallDict (kwargs=0x0, _nargs=0, args=0x7fffd51b0cf8, func=0x7ff5b3e18ae0) at build/cython/central_ve/main.c:5424
#5 __pyx_pf_10central_ve_4main__cli_default_action (__pyx_self=<optimized out>) at build/cython/central_ve/main.c:2746
#6 __pyx_pw_10central_ve_4main_1_cli_default_action (__pyx_self=<optimized out>, unused=<optimized out>) at build/cython/central_ve/main.c:2596
#7 0x00007ff5beaa1729 in __Pyx_PyObject_CallMethO (arg=0x0, func=0x7ff5b4034ba0) at build/cython/central_ve/main.c:5354
#8 __Pyx_PyObject_FastCallDict (kwargs=0x0, _nargs=0, args=0x7fffd51b0d78, func=0x7ff5b4034ba0) at build/cython/central_ve/main.c:5388
#9 __pyx_pf_10central_ve_4main_2cli (__pyx_self=<optimized out>, __pyx_v_ctx=<optimized out>) at build/cython/central_ve/main.c:3141
#10 __pyx_pw_10central_ve_4main_3cli (__pyx_self=<optimized out>, __pyx_args=<optimized out>, __pyx_nargs=<optimized out>, __pyx_kwds=<optimized out>) at build/cython/central_ve/main.c:3057
#11 0x00007ff5c042ae7e in _PyEval_EvalFrameDefault () from /home/my_user/.tmp/_MEIhhcmlJ/libpython3.11.so.1.0
#12 0x00007ff5c044b769 in _PyObject_Call () from /home/my_user/.tmp/_MEIhhcmlJ/libpython3.11.so.1.0
#13 0x00007ff5c042ae7e in _PyEval_EvalFrameDefault () from /home/my_user/.tmp/_MEIhhcmlJ/libpython3.11.so.1.0
#14 0x00007ff5c045cba4 in ?? () from /home/my_user/.tmp/_MEIhhcmlJ/libpython3.11.so.1.0
#15 0x00007ff5c045c840 in ?? () from /home/my_user/.tmp/_MEIhhcmlJ/libpython3.11.so.1.0
#16 0x00007ff5c042ae7e in _PyEval_EvalFrameDefault () from /home/my_user/.tmp/_MEIhhcmlJ/libpython3.11.so.1.0
#17 0x00007ff5c045c78a in ?? () from /home/my_user/.tmp/_MEIhhcmlJ/libpython3.11.so.1.0
#18 0x00007ff5c042ae7e in _PyEval_EvalFrameDefault () from /home/my_user/.tmp/_MEIhhcmlJ/libpython3.11.so.1.0
#19 0x00007ff5c04219e9 in _PyObject_FastCallDictTstate () from /home/my_user/.tmp/_MEIhhcmlJ/libpython3.11.so.1.0
#20 0x00007ff5c044a6e4 in _PyObject_Call_Prepend () from /home/my_user/.tmp/_MEIhhcmlJ/libpython3.11.so.1.0
#21 0x00007ff5c04d0e4e in ?? () from /home/my_user/.tmp/_MEIhhcmlJ/libpython3.11.so.1.0
#22 0x00007ff5c041fb5c in _PyObject_MakeTpCall () from /home/my_user/.tmp/_MEIhhcmlJ/libpython3.11.so.1.0
#23 0x00007ff5c0428d4f in _PyEval_EvalFrameDefault () from /home/my_user/.tmp/_MEIhhcmlJ/libpython3.11.so.1.0
#24 0x00007ff5c0489186 in ?? () from /home/my_user/.tmp/_MEIhhcmlJ/libpython3.11.so.1.0
#25 0x00007ff5c0488f08 in PyEval_EvalCode () from /home/my_user/.tmp/_MEIhhcmlJ/libpython3.11.so.1.0
#26 0x0000000000403770 in ?? ()
#27 0x0000000000404142 in ?? ()
#28 0x00007ff5c094970a in ?? () from /lib/ld-musl-x86_64.so.1
#29 0x0000000000000000 in ?? ()
$9 = (void *) 0x7ff5b3f06eb0
$10 = 0x0
[New LWP 1868]
$11 = "SUCCESS"
[Inferior 1 (process 13599) detached]
Timed out waiting for connection from pid 13599
Hm. Both of those seem to indicate that it successfully attached to the remote process and spawned the thread that's going to start the Memray tracking session. It seems like whatever's going wrong is going wrong in the remote process, after the debugger thinks everything is done and the remote process has been successfully asked to do what it needs to do.
Does the process you're attaching to print anything to stderr or stdout after you try this? One possibility is that the background thread is waiting to acquire the GIL and it's not getting dropped, or something like that...
Thanks for the answer. Unfortunately nothing gets printed in stderr nor in stdout.
I think at this point you can try to work towards giving us a reproducer so we can investigate. Otherwise there isn't a lot we can do here unfortunately :(
The errors different, but maybe it will help you navigate:
app.py
import time
import signal
import sys
import os
_running = True
def _handle_signal(sig, frame):
global _running
print(f"[{os.getpid()}] Received signal {sig}. Shutting down...", flush=True)
_running = False
signal.signal(signal.SIGINT, _handle_signal)
signal.signal(signal.SIGTERM, _handle_signal)
def main():
print(f"[{os.getpid()}] Cythonized heartbeat app starting…", flush=True)
while _running:
print(f"[{os.getpid()}] heartbeat {int(time.time())}", flush=True)
time.sleep(5)
print(f"[{os.getpid()}] Bye.", flush=True)
if __name__ == "__main__":
main()
Dockerfile
FROM python:3.12-alpine
RUN apk add --no-cache build-base python3-dev musl-dev
ENV PYTHONDONTWRITEBYTECODE=1 \
PYTHONUNBUFFERED=1 \
PIP_NO_CACHE_DIR=1
WORKDIR /app
COPY app.py /app/
RUN apk add memray --repository http://dl-3.alpinelinux.org/alpine/edge/testing/
RUN apk add lldb gdb
# Build native binary from app.py with proper embed flags.
# Fallback to plain --ldflags if --embed isn't supported (older Pythons).
RUN pip install --no-cache-dir Cython && \
cython --embed -3 app.py -o app.c && \
LD_FLAGS="$(python3-config --embed --ldflags 2>/dev/null || python3-config --ldflags)" && \
gcc -O3 -pipe -fno-strict-aliasing $(python3-config --includes) app.c -o app ${LD_FLAGS} && \
strip /app/app
CMD ["./app"]
In one shell:
docker run --rm --name heartbeat cython-heartbeat
In the other:
docker exec -it heartbeat sh
memray attach 1 --verbose