Basic Example broken.

[poleguy]

I'm trying this for the first time, and the simple example on the main page does not work: http://vmlaker.github.io/mpipe/

Tried on python 3.4.5 and 3.6.1 Windows 7 Anaconda 4.4.0 64-bit

Has this ever been tested on windows?

It errors out with: ... File "C:\Anaconda3\lib\site-packages\mpipe\Pipeline.py", line 9, in init self._input_stage.build() File "C:\Anaconda3\lib\site-packages\mpipe\Stage.py", line 99, in build self._do_stop_task, File "C:\Anaconda3\lib\site-packages\mpipe\OrderedWorker.py", line 93, in assemble worker.start() File "C:\Anaconda3\lib\multiprocessing\process.py", line 105, in start self._popen = self._Popen(self) File "C:\Anaconda3\lib\multiprocessing\context.py", line 223, in _Popen return _default_context.get_context().Process._Popen(process_obj) File "C:\Anaconda3\lib\multiprocessing\context.py", line 322, in _Popen return Popen(process_obj) File "C:\Anaconda3\lib\multiprocessing\popen_spawn_win32.py", line 65, in init reduction.dump(process_obj, to_child) File "C:\Anaconda3\lib\multiprocessing\reduction.py", line 60, in dump ForkingPickler(file, protocol).dump(obj) AttributeError: Can't pickle local object 'OrderedStage.init..wclass'

[vmlaker]

Sorry, hasn't been tested on Windows. Are you running this from the Python console? Try saving the code to a file and run with command:

python test.py

[anviloro]

same here, tried on windows didn't work, tried on ubuntu server, worked like a charm.

[AndrewMartchenko]

Hi, I think I have a solution to this problem. This problem is occurring because "wclass" class is dynamically defined inside the OrderedStage and UnorderedStage constructor. Python uses lambda magic to create this class and so it is not picklable in windows. To solve the problem you will need to pass the "target" function into the "wclass" constructor and possible define "wclass" globally. Hope this helps.

Edit: In windows functions can only be pickled if they are defined in the top level. So you will have to globally define 'wclass' for multiprocessing to pickle it.