Multiprocessing issues

[RussBaz]

Continuation of Issue #36 - threading issues.

So, does anyone know how to make it work with multiprocessing? I cannot grasp where exactly the issue is arising. Is it because of 'wrapt'?

[TheDataLeek]

Yeah, so for context let me paste my answer from the previous issue and clarify slightly

I am unconvinced that this is a problem on our end.

Dig into the SO page here: https://stackoverflow.com/questions/9336646/python-decorator-with-multiprocessing-fails

TL;DR ProcessPools can only accept pickleable things to spawn into processes. Decorated functions are inherently not pickleable, since it's a function wrapper and not an actual function.

Basically the problem is that when you use wrapt to apply a decorator to a function, it's not actually a function. It's some obtuse function representation. As a result, when you try to pass that "function" to a multiprocess instance, it explodes because you're not giving it any sort of function representation that it can deal with (which it references by complaining about it being "un-pickleable").

The best workaround that I can see is something like this:

def test_one_widget(widget_id: int) -> int:
    @enforce.runtime_validation
    def foo(widget_id: int) -> int:
        score = widget_inspector(widget_id, a='foo', b=4, c='bar')
        return  score
    return foo(widget_id)

Where, if you need to multiprocess on a certain function that enforce is applied to, you first wrap it in another function that does nothing but pass the value of the inner function through. It's a little more code, but then multiprocess won't complain about it being "not pickleable", since you're now passing it just a regular function.

EDIT: This is a problem that affects any decorator that uses wrapt or something like it to create decorators.

[RussBaz]

I might have an idea how to fix this but it will require a massive re-write of almost everything. It is not very feasible at this stage.

I think the best course of action for now is to add this workaround to the Readme.