billiard.Manager broken in billiard 3.5.0.3 in python 2.7, used to work in 3.3.0.21

[v-rfc]

Here's a trivial example of using a Manager that works with the original multiprocessing, and used to work in billiard 3.3.0.21, but no longer works with 3.5.0.3 running in linux on Python 2.7.5:

import billiard
m = billiard.Manager()
q = m.Queue()

Assume I'm not calling the above inside a celery task, I'm executing the above three statements in a new python process.

with billiard 3.5.0.3 running in linux on Python 2.7.5 , we immediately see a crash:

>>> import billiard
>>> m = billiard.Manager()
Process SyncManager-1:
Traceback (most recent call last):
  File "/usr/lib64/python2.7/site-packages/billiard/process.py", line 327, in _bootstrap
    self.run()
  File "/usr/lib64/python2.7/site-packages/billiard/process.py", line 114, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/lib64/python2.7/site-packages/billiard/managers.py", line 529, in _run_server
    server = cls._Server(registry, address, authkey, serializer)
  File "/usr/lib64/python2.7/site-packages/billiard/managers.py", line 158, in __init__
    self.listener = Listener(address=address, backlog=16)
  File "/usr/lib64/python2.7/site-packages/billiard/connection.py", line 483, in __init__
    address = address or arbitrary_address(family)
  File "/usr/lib64/python2.7/site-packages/billiard/connection.py", line 93, in arbitrary_address
    return tempfile.mktemp(prefix='listener-', dir=util.get_temp_dir())
  File "multiprocessing/util.py", line 141, in get_temp_dir
    current_process()._tempdir = tempdir
AttributeError: can't set attribute
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib64/python2.7/site-packages/billiard/context.py", line 94, in Manager
    m.start()
  File "/usr/lib64/python2.7/site-packages/billiard/managers.py", line 507, in start
    self._address = reader.recv()
  File "/usr/lib64/python2.7/site-packages/billiard/connection.py", line 280, in recv
    buf = self._recv_bytes()
  File "/usr/lib64/python2.7/site-packages/billiard/connection.py", line 458, in _recv_bytes
    buf = self._recv(4)
  File "/usr/lib64/python2.7/site-packages/billiard/connection.py", line 432, in _recv
    raise EOFError
EOFError

I tried to debug this. So far I can see the following issues. Each subsequent issue is revealed by patching to fix or work around the previous issues:

1. billiard.process.BaseProcess defines a read-only _tempdir property, but billiard re-uses multiprocessing.util.get_temp_dir() which assumes a processes' _tempdir is writeable. This is simple to fix.

2. billiard.connection's deliver_challenge and answer_challenge functions appear to break under python 2.7. The calls to hmac.new(authkey, message, 'md5').digest() fail in python 2.7 as passing a value of str type for the third argument is not supported. importing hashlib and passing hashlib.md5 resolves this in python 2.7 and reads as if it should work in python 3. Ref: https://docs.python.org/2/library/hmac.html#hmac.new ; https://docs.python.org/3/library/hmac.html#hmac.new .

3. calling manager.Queue() triggers a crash as billiard.managers tries to do with BaseProxy._mutex:, which fails, as the _mutex attribute is a multiprocessing.util.ForkAwareThreadLock object which doesn't implement the context manager protocol. See: https://github.com/celery/billiard/blame/78a5b4592446466afe1020b49b01918cdeaeb9f0/billiard/managers.py#L693 .

I stopped investigating after hitting the third issue, so there may be other regressions as well.

I'm not hitting these billiard issues directly with celery -- I have some legacy application code that was originally written to use multiprocessing Managers and Queues, then previously we replaced all of our dependencies on multiprocessing with dependencies on billiard to work around issues with celery tasks launching subprocesses after migrating the application from windows to linux (the old "Tasks are not allowed to start subprocesses" celery issue: https://github.com/celery/celery/issues/1709 ), but now these recent billiard changes prevent us from upgrading to newer versions of billiard and celery until we rewrite our application code to stop depending upon parts of the multiprocessing API that billiard breaks support for.

[miklagard]

I have still the same issue in billiard 3.5.0.4.

Value was not working as well in 3.5.0.3, but started to work in the new built. The problem is that I cannot share dictionaries with this and still need Manager.

[miklagard]

Small detail: Manager works without any problem with "Python 2.7.15+" but it fails in "Python 2.7.15rc1" and "Python 2.7.12" according my tests.

And it works again Python 3.5.2 and Python 3.6.7rc1 without any error.

[auvipy]

is it working with 3.6.0 version?

[miklagard]

I haven't tried.

[caioariede]

The multiprocessing API has changed in Python 3 and it seems like billiard didn't cover everything to keep the compatibility with Python 2 and 3. I'm running a Python 2 codebase, and here's how we monkey-patched the issue:

from billiard.process import BaseProcess

def _patched_billiard_process_tempdir(process, tempdir):
    process._config["tempdir"] = tempdir

BaseProcess._tempdir = BaseProcess._tempdir.setter(_patched_billiard_process_tempdir)