@parallel doesn't work with Python 3.8

[maxmalysh]

Hi,

First of all, thanks for the fork and for the work done!

We're migrating from Fabric3==1.14.post1 to fab-classic.

Both packages have the same problem: parallel doesn't work with Python 3.8.

You can import it as: a. from fabric.api import parallel or b. from fabric.decorators import parallel

Here is an example:

@parallel
@roles('nginx')
def test():
    run('echo test!')

Result:

(venv) MacBook-Pro:my-website mmalysh$ fab test
[[email protected]] Executing task 'test'
Traceback (most recent call last):
  File "/Users/mmalysh/Development/my-website/venv/lib/python3.8/site-packages/fabric/main.py", line 759, in main
    execute(
  File "/Users/mmalysh/Development/my-website/venv/lib/python3.8/site-packages/fabric/tasks.py", line 412, in execute
    ran_jobs = jobs.run()
  File "/Users/mmalysh/Development/my-website/venv/lib/python3.8/site-packages/fabric/job_queue.py", line 138, in run
    _advance_the_queue()
  File "/Users/mmalysh/Development/my-website/venv/lib/python3.8/site-packages/fabric/job_queue.py", line 123, in _advance_the_queue
    job.start()
  File "/Users/mmalysh/.pyenv/versions/3.8.2/lib/python3.8/multiprocessing/process.py", line 121, in start
    self._popen = self._Popen(self)
  File "/Users/mmalysh/.pyenv/versions/3.8.2/lib/python3.8/multiprocessing/context.py", line 224, in _Popen
    return _default_context.get_context().Process._Popen(process_obj)
  File "/Users/mmalysh/.pyenv/versions/3.8.2/lib/python3.8/multiprocessing/context.py", line 283, in _Popen
    return Popen(process_obj)
  File "/Users/mmalysh/.pyenv/versions/3.8.2/lib/python3.8/multiprocessing/popen_spawn_posix.py", line 32, in __init__
    super().__init__(process_obj)
  File "/Users/mmalysh/.pyenv/versions/3.8.2/lib/python3.8/multiprocessing/popen_fork.py", line 19, in __init__
    self._launch(process_obj)
  File "/Users/mmalysh/.pyenv/versions/3.8.2/lib/python3.8/multiprocessing/popen_spawn_posix.py", line 47, in _launch
    reduction.dump(process_obj, fp)
  File "/Users/mmalysh/.pyenv/versions/3.8.2/lib/python3.8/multiprocessing/reduction.py", line 60, in dump
    ForkingPickler(file, protocol).dump(obj)
AttributeError: Can't pickle local object '_execute.<locals>.inner'

No problems with Python 3.7.7.

[ploxiln]

We have seen this kind of thing before ... the fabric-1 parallel mode forks separate processes, and this apparently requires pickling some objects into bytes to pass between processes, and in odd cases, some objects in the tree can't be pickled. This happened on windows before (possibly still). I'll take a look and see what can be done, maybe in a few days, unless you figure it out first :)

[ploxiln]

It looks like this is going to be quite a pickle. I untangled some wrappers in https://github.com/ploxiln/fab-classic/compare/master...ploxiln:python38_parallel and it seems that it's the task callables that are the real problem with Python-3.8:

_pickle.PicklingError: Can't pickle <function shellcmd at 0x10312c9d0>: it's not the same object as fabfile.shellcmd

This seems related to https://stackoverflow.com/questions/9336646/python-decorator-with-multiprocessing-fails and https://stackoverflow.com/questions/1412787/picklingerror-cant-pickle-class-decimal-decimal-its-not-the-same-object

I tried a few different hacks to get any kind of task working, as a proof of concept, with no luck.

[ploxiln]

I came up with a hacky way, which might only work for top-level @task decorated functions. see https://github.com/ploxiln/fab-classic/pull/32 and maybe try with:

pip install https://github.com/ploxiln/fab-classic/archive/python38_parallel.tar.gz

(not proper enough to merge though)

[Stealthii]

I've had great success with ensuring the default processing call uses fork() and not spawn() - pre 3.8 speed, behaviours and execution speed on macOS seem to be upheld:

diff --git a/fabric/tasks.py b/fabric/tasks.py
index 47a2b49f..c56c35a2 100644
--- a/fabric/tasks.py
+++ b/fabric/tasks.py
@@ -329,6 +329,9 @@ def execute(task, *args, **kwargs):
         # if it can't.
         try:
             import multiprocessing
+            ctx = multiprocessing.get_context('fork')
+            # Set up job queue for parallel cases
+            queue = ctx.Queue()
         except ImportError:
             import traceback
             tb = traceback.format_exc()
@@ -338,12 +341,11 @@ def execute(task, *args, **kwargs):
     traceback.) Please make sure the module is installed
     or that the above ImportError is fixed.""")
     else:
-        multiprocessing = None
+        ctx = None
+        queue = None

     # Get pool size for this task
     pool_size = task.get_pool_size(my_env['all_hosts'], state.env.pool_size)
-    # Set up job queue in case parallel is needed
-    queue = multiprocessing.Queue() if parallel else None
     jobs = JobQueue(pool_size, queue)
     if state.output.debug:
         jobs._debug = True
@@ -355,7 +357,7 @@ def execute(task, *args, **kwargs):
             try:
                 results[host] = _execute(
                     task, host, my_env, args, new_kwargs, jobs, queue,
-                    multiprocessing
+                    ctx
                 )
             except NetworkError as e:
                 results[host] = e

@ploxiln it might be worth making this change for non-Windows (or all if we don't maintain Windows support), until such a time as the multi-thread method can be rewritten as you suggest.