multitasking icon indicating copy to clipboard operation
multitasking copied to clipboard

Unable to pickle when using multiprocessing

Open mobone opened this issue 5 years ago • 5 comments

I'm using your example from the readme and while setting the engine to multiprocessing I receive the following error. I'm using python 3.8 64 bit.


Traceback (most recent call last):
  File "mp_test.py", line 23, in <module>
    hello(i+1)
  File "C:\Users\nbrei\AppData\Local\Programs\Python\Python38\lib\site-packages\multitasking\__init__.py", line 119, in async_method
    single.start()
  File "C:\Users\nbrei\AppData\Local\Programs\Python\Python38\lib\multiprocessing\process.py", line 121, in start
    self._popen = self._Popen(self)
  File "C:\Users\nbrei\AppData\Local\Programs\Python\Python38\lib\multiprocessing\context.py", line 224, in _Popen
    return _default_context.get_context().Process._Popen(process_obj)
  File "C:\Users\nbrei\AppData\Local\Programs\Python\Python38\lib\multiprocessing\context.py", line 326, in _Popen
    return Popen(process_obj)
  File "C:\Users\nbrei\AppData\Local\Programs\Python\Python38\lib\multiprocessing\popen_spawn_win32.py", line 93, in __init__
    reduction.dump(process_obj, to_child)
  File "C:\Users\nbrei\AppData\Local\Programs\Python\Python38\lib\multiprocessing\reduction.py", line 60, in dump
    ForkingPickler(file, protocol).dump(obj)
AttributeError: Can't pickle local object 'task.<locals>._run_via_pool'

C:\Users\nbrei\Documents\GitHub\pead_ml2>Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "C:\Users\nbrei\AppData\Local\Programs\Python\Python38\lib\multiprocessing\spawn.py", line 102, in spawn_main
    source_process = _winapi.OpenProcess(
OSError: [WinError 87] The parameter is incorrect

mobone avatar Jan 13 '20 03:01 mobone

It appears to work fine on my mac with python 3.7, traceback was on windows. I'll test on my windows machine with python 3.7 when I get home from work.

mobone avatar Jan 13 '20 17:01 mobone

Nope, didn't solve it. Still getting the same traceback on windows.

mobone avatar Jan 13 '20 22:01 mobone

So if I understand the issue, you used the code from README.rst exactly as typed and you are getting errors. As it is currently written there is no pickle function, please include a copy of your code.

datatalking avatar Jul 22 '21 13:07 datatalking

I have try define _run_via_pool staticmethod and it fix. Actually,I am unfamiliar with python and I dont the reason @staticmethod def _run_via_pool(*args, **kwargs):

wlxklyh avatar Dec 03 '22 18:12 wlxklyh

I just now clean installed Python 3.11.8 and am getting the same, parameter is incorrect. And again, it's literally copy and paste from the readme, and change from threads to process.

Edit: I just now had a friend try on his windows computer and he's getting the same exception.

# example.py
import multitasking
import time
import random
import signal

# kill all tasks on ctrl-c
signal.signal(signal.SIGINT, multitasking.killall)

multitasking.set_engine("process") # "process" or "thread"

# or, wait for task to finish on ctrl-c:
# signal.signal(signal.SIGINT, multitasking.wait_for_tasks)

@multitasking.task # <== this is all it takes :-)
def hello(count):
    sleep = random.randint(1,10)/2
    print("Hello %s (sleeping for %ss)" % (count, sleep))
    time.sleep(sleep)
    print("Goodbye %s (after for %ss)" % (count, sleep))

if __name__ == "__main__":
    for i in range(0, 10):
        hello(i+1)

mobone avatar Mar 14 '24 15:03 mobone