spawnv_passfds() doesn't work on Python 3.9

[vstinner]

spawnv_passfds() function is implemented in billiard/compat.py. On Pytho 3, it calls a private Python function: _posixsubprocess.fork_exec(). The problem is that the API of this function changed in Python 3.9, it got new required parameter.

umask:

• https://bugs.python.org/issue38417
• https://github.com/python/cpython/commit/f3751efb5c8b53b37efbbf75d9422c1d11c01646

user, group, extra_groups:

• https://bugs.python.org/issue36046
• https://github.com/python/cpython/commit/2b2ead74382513d0bb9ef34504e283a71e6a706f

You can pass default values:

• umask=-1
• user=None
• group=None
• extra_groups=None

Something like (you may use a different name than args to avoid reusing args name twice):

args = [
    args, [fsencode(path)], True, tuple(passfds), None, None,
    -1, -1, -1, -1, -1, -1, errpipe_read, errpipe_write,
    False, False, None]
if sys.version_info >= (3, 9):
    args.extend((None, None, None, -1))  # group, extra_groups, user, umask
args.append(None)  # preexec_fn
return _posixsubprocess.fork_exec(*args)

[vstinner]

Downstream Fedora issue with Python 3.9: https://bugzilla.redhat.com/show_bug.cgi?id=1792056

[auvipy]

ops! thanks for the report! I was planning to update the multiprocessing dependencies of billiard https://github.com/celery/billiard/pull/292 , would you mind sharing your insight about using which version of multiprocessing codes should billiard focused on? python 2.7 or from python3.9/master?

[vstinner]

My patch doesn't touch the Python 2 support which is kept: it only adds support for Python 3.9. It's up to you to continue to support Python 2 or not, but it's no longer supported upstream (will no longer get security fixes).

[auvipy]

billiard will get py27 support till Dec 2020[PyPY] and many old enterprise users. and for celery 5 we will not be using billiard. celery 4.x will be python2 compatible with billiard and amqp 1.0 support. celery 5will be based on trio and amqp 1.0 and python 3.6+ only.

[auvipy]

btw patch welcome.

[vstinner]

btw patch welcome.

I converted my inline patch into a concrete PR: PR #310.