Process attach for 3.11 on Mac

[gaogaotiantian]

Hi!

I was wondering if process attach is supported on 3.11 on Mac now? I saw some code checkins for it but I'm not sure if it's done yet. I was having problems on this specific combination. It works on 3.11 on Linux/Windows, and 3.10 on Mac. However, I'm kind of using a customized version with the process attach code.

If it's a known issue, then I can wait until it's fixed. If it is supposed to work, I can try to collect more information and see if I can provide a fast repro demo.

Thanks,

Tian

[fabioz]

It should work... can you send a screenshot showing the output you have when it fails?

[gaogaotiantian]

I tried to make a minimum demo to demonstrate the different behavior between 3.9 & 3.11 on Mac.

First of all, I have a very simply script running, basically a while loop. And yes I know without the debugger running on it, it is not expected to work. I used python attach_pydevd.py --pid <pid> to attach to it, and the output was:

Running: lldb --no-lldbinit --script-language Python -o 'process attach --pid 6688' -o 'command script import "/Users/gaotian/programs/PyDev.Debugger/pydevd_attach_to_process/linux_and_mac/lldb_prepare.py"' -o 'load_lib_and_attach "/Users/gaotian/programs/PyDev.Debugger/pydevd_attach_to_process/attach_x86_64.dylib" 0 "import sys;sys.path.append(\"/Users/gaotian/programs/PyDev.Debugger\");sys.path.append(\"/Users/gaotian/programs/PyDev.Debugger/pydevd_attach_to_process\");import attach_script;attach_script.attach(port=5678, host=\"127.0.0.1\", protocol=\"\");" 0' -o 'process detach' -o 'script import os; os._exit(1)'
(lldb) process attach --pid 6688
Process 6688 stopped
* thread #1, queue = 'com.apple.main-thread', stop reason = signal SIGSTOP
    frame #0: 0x00007fff20332656 libsystem_kernel.dylib`__select + 10
libsystem_kernel.dylib`__select:
->  0x7fff20332656 <+10>: jae    0x7fff20332660            ; <+20>
    0x7fff20332658 <+12>: movq   %rax, %rdi
    0x7fff2033265b <+15>: jmp    0x7fff2032b6bd            ; cerror
    0x7fff20332660 <+20>: retq   
Target 0: (Python) stopped.

Executable module set to "/usr/local/Cellar/[email protected]/3.9.13_1/Frameworks/Python.framework/Versions/3.9/Resources/Python.app/Contents/MacOS/Python".
Architecture set to: x86_64h-apple-macosx-.
(lldb) command script import "/Users/gaotian/programs/PyDev.Debugger/pydevd_attach_to_process/linux_and_mac/lldb_prepare.py"
(lldb) load_lib_and_attach "/Users/gaotian/programs/PyDev.Debugger/pydevd_attach_to_process/attach_x86_64.dylib" 0 "import sys;sys.path.append(\"/Users/gaotian/programs/PyDev.Debugger\");sys.path.append(\"/Users/gaotian/programs/PyDev.Debugger/pydevd_attach_to_process\");import attach_script;attach_script.attach(port=5678, host=\"127.0.0.1\", protocol=\"\");" 0
/Users/gaotian/programs/PyDev.Debugger/pydevd_attach_to_process/attach_x86_64.dylib
success
import sys;sys.path.append("/Users/gaotian/programs/PyDev.Debugger");sys.path.append("/Users/gaotian/programs/PyDev.Debugger/pydevd_attach_to_process");import attach_script;attach_script.attach(port=5678, host="127.0.0.1", protocol="");
success
(lldb) process detach
Process 6688 detached
(lldb) script import os; os._exit(1)
Traceback (most recent call last):
  File "/Users/gaotian/programs/PyDev.Debugger/pydevd_attach_to_process/attach_pydevd.py", line 73, in <module>
    main(process_command_line(sys.argv[1:]))
  File "/Users/gaotian/programs/PyDev.Debugger/pydevd_attach_to_process/attach_pydevd.py", line 68, in main
    add_code_to_python_process.run_python_code(
  File "/Users/gaotian/programs/PyDev.Debugger/pydevd_attach_to_process/add_code_to_python_process.py", line 528, in run_python_code_mac
    subprocess.check_call(' '.join(cmd), shell=True, env=env)
  File "/usr/local/Cellar/[email protected]/3.9.13_1/Frameworks/Python.framework/Versions/3.9/lib/python3.9/subprocess.py", line 373, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command 'lldb --no-lldbinit --script-language Python -o 'process attach --pid 6688' -o 'command script import "/Users/gaotian/programs/PyDev.Debugger/pydevd_attach_to_process/linux_and_mac/lldb_prepare.py"' -o 'load_lib_and_attach "/Users/gaotian/programs/PyDev.Debugger/pydevd_attach_to_process/attach_x86_64.dylib" 0 "import sys;sys.path.append(\"/Users/gaotian/programs/PyDev.Debugger\");sys.path.append(\"/Users/gaotian/programs/PyDev.Debugger/pydevd_attach_to_process\");import attach_script;attach_script.attach(port=5678, host=\"127.0.0.1\", protocol=\"\");" 0' -o 'process detach' -o 'script import os; os._exit(1)'' returned non-zero exit status 1.

which was expected - the attach and code injection succeeded but there's no debugger running on the process being attached. After trying to attach, the process being attached exited from an exception as well.

Then I did the exact same thing on 3.11.0 on Mac. The output was much simpler:

Running: lldb --no-lldbinit --script-language Python -o 'process attach --pid 6735' -o 'command script import "/Users/gaotian/programs/PyDev.Debugger/pydevd_attach_to_process/linux_and_mac/lldb_prepare.py"' -o 'load_lib_and_attach "/Users/gaotian/programs/PyDev.Debugger/pydevd_attach_to_process/attach_x86_64.dylib" 0 "import sys;sys.path.append(\"/Users/gaotian/programs/PyDev.Debugger\");sys.path.append(\"/Users/gaotian/programs/PyDev.Debugger/pydevd_attach_to_process\");import attach_script;attach_script.attach(port=5678, host=\"127.0.0.1\", protocol=\"\");" 0' -o 'process detach' -o 'script import os; os._exit(1)'
(lldb) process attach --pid 6735
error: attach failed: attach failed (Not allowed to attach to process.  Look in the console messages (Console.app), near the debugserver entries when the attached failed.  The subsystem that denied the attach permission will likely have logged an informative message about why it was denied.)
(lldb) 

Notice that I'm now in lldb after running the script, instead of my terminal. The process being attached kept running without interruption.

I believe this meant that at least the code injection was not successful. Not sure if you have a Mac around but this should be pretty easy to reproduce.

[fabioz]

From what you're describing, it says that lldb itself wasn't able to do the attach.

i.e.:

error: attach failed: attach failed (Not allowed to attach to process. Look in the console messages (Console.app), near the debugserver entries when the attached failed. The subsystem that denied the attach permission will likely have logged an informative message about why it was denied.)

is mentioned some log messages... do you think you can find those there?

I searched a bit and https://developer.apple.com/forums/thread/676028 also seems interesting... It says that it's possible that something as a hardened runtime may be set which disables the attach, so, maybe it'd be worth trying with a different Python build (conda-forge seems to have Python 3.11 already, can you try with that)?

Also can you confirm if this is a x86_64 or arm version of Python 3.11 (only x86_64 is currently supported).

As a note, the issue seems to be all before the debugger even has a chance to act as it's lldb which is not even connecting to the process -- so, there's some research needed on that side and in theory after that's done the debugger should work, but I don't know why lldb wouldn't really work in that case (and I don't really have a Mac around, so, it's a bit difficult to properly check it here...).

You can probably just try connecting lldb to the Python executable (to completely remove the python debugger from the scene) and try to get support from lldb folks... after lldb can connect, then it'd be time to go forward with the python debugger side.

[gaogaotiantian]

Okay I spent some time investigating this.

First of all, yes the error I had was not a debugger issue, it was Mac. I believe it's because the python app I was using used hardened runtime for some reason, so lldb can not attach to it. And it's on a Mac with intel CPU. I believe starting from python3.10, there's only the universal2 release version for Python on Mac.

Then the bad news, I used the official python package for Mac which was also the package setup-python uses. I tried to use 3.11-dev from py-env, which I believe was cloning the cpython repo and build locally, and it did not have the issue. I can attach to the process and my attaching code worked without any problem.

So, from one perspective, this issue is solved. However, from another perspective, it could mean that PyDev won't work on 3.11 + Mac anymore. The worse news is, this might be the case for all future python versions on Mac.

I'm not sure if this is something you want to follow up with, may or may not be an issue that the debuggers are not able to work on Mac anymore, but let me know if I there's any other information I could provide with.