beeware
Stub binary doesn't support `sys.executable` spawn/multiprocessing
Describe the bug When multiprocessing is used to fork a subprocess, the main program entrypoint is executed in the subprocess rather than the Process target.
To Reproduce
create a multiprocessing.Process() and start it using .start() The subprocess will call the parent process entry point.
Expected behavior The target function is called as the entrypoint of the subprocess
Screenshots If applicable, add screenshots to help explain your problem.
Environment:
- Operating System: macOS
- Python version: 3.10
Additional context Add any other context about the problem here.
Note this requires https://github.com/beeware/Python-Apple-support/pull/132 to include the _posixshmem package in the support package required by multiprocessing
The _posixshmem issue should have been resolved as part of beeware/Python-Apple-support#161; it is now verified as part of the support testbed.
However, the more general issue with multiprocessing remains.
The issue is that Briefcase doesn't guarantee that there will be a python executable; as a result, multiprocessing will fail in spawn mode (which is the default as of Python 3.8).
fork and forkserver mode should both work. However, spawn mode assumes that sys.executable can be invoked with the same arguments as the python interpreter, which isn't true of a macOS stub binary.
This is definitely something that should be addressed; some sort of logic in the stub app is required that verifies if the parent process is "self"; and if so, changes the interpretation of command line arguments so that spawn calls behave like a "normal" Python interpreter. This will need to be fixed on the Xcode template (see beeware/briefcase-macOS-Xcode-template#20).
forkserver mode doesn't work either, because that calls sys.executable to start the server, plus what the multiprocessing documentation calls a "resource tracker process".
For example, if you replace the __main__ module of a Briefcase app with the following:
if __name__ == '__main__':
import multiprocessing
import operator
import sys
print(f"{sys.argv=}")
if len(sys.argv) == 1:
multiprocessing.set_start_method("forkserver")
with multiprocessing.Pool(2) as p:
print(p.map(operator.neg, [1, 2, 3]))
briefcase run will log the following command lines:
sys.argv=['/Users/msmith/git/beeware/apps/helloworld/macOS/app/Hello World/Hello World.app/Contents/MacOS/Hello World']
sys.argv=['/Users/msmith/git/beeware/apps/helloworld/macOS/app/Hello World/Hello World.app/Contents/MacOS/Hello World', '-B', '-I', '-c', 'from multiprocessing.resource_tracker import main;main(7)']
sys.argv=['/Users/msmith/git/beeware/apps/helloworld/macOS/app/Hello World/Hello World.app/Contents/MacOS/Hello World', '-B', '-I', '-c', "from multiprocessing.forkserver import main; main(17, 18, ['__main__'], **{'sys_path': ['/Users/msmith/git/beeware/apps/helloworld/macOS/app/Hello World/Hello World.app/Contents/Resources/support/python38.zip', '/Users/msmith/git/beeware/apps/helloworld/macOS/app/Hello World/Hello World.app/Contents/Resources/support/python-stdlib', '/Users/msmith/git/beeware/apps/helloworld/macOS/app/Hello World/Hello World.app/Contents/Resources/support/python-stdlib/lib-dynload', '/Users/msmith/git/beeware/apps/helloworld/macOS/app/Hello World/Hello World.app/Contents/Resources/app_packages', '/Users/msmith/git/beeware/apps/helloworld/macOS/app/Hello World/Hello World.app/Contents/Resources/app']})"]
If you remove the len(sys.argv) check, the app will replicate itself exponentially and lock up the whole system.
fork mode worked fine for me on macOS. It's no longer the default on macOS because of https://github.com/python/cpython/issues/77906, but it looks like that only affects apps which call Objective-C APIs in the child processes.
Unfortunately, using fork mode isn't a cross-platform solution, because it isn't available on Windows. It's also unsafe to call fork in any process that has multiple threads (Python or otherwise), for the reasons explained here.
