ipyparallel icon indicating copy to clipboard operation
ipyparallel copied to clipboard

`dill` and `cloudpickle` don't work with `require`

Open basnijholt opened this issue 9 years ago • 1 comments

from ipyparallel import Client, require
client = Client()
lview = client.load_balanced_view()
dview = client[:]
dview.use_cloudpickle()

@require('types')
def func(x):
    return types.SimpleNamespace(n=x)

res = lview.map(func, range(10))

res.result()

gives

---------------------------------------------------------------------------NameError                                 Traceback (most recent call last)<string> in <module>()
/home/bnijholt/anaconda3/lib/python3.5/site-packages/ipyparallel/client/remotefunction.py in <lambda>(f, *sequences)
    248             if _mapping:
    249                 if sys.version_info[0] >= 3:
--> 250                     f = lambda f, *sequences: list(map(f, *sequences))
    251                 else:
    252                     f = map
/home/bnijholt/anaconda3/lib/python3.5/site-packages/ipyparallel/controller/dependency.py in __call__(self, *args, **kwargs)
     71 
     72     def __call__(self, *args, **kwargs):
---> 73         return self.f(*args, **kwargs)
     74 
     75     if not py3compat.PY3:
<ipython-input-8-293a3326374e> in func(x)
NameError: name 'types' is not defined

basnijholt avatar Nov 08 '16 20:11 basnijholt

These decorators shouldn't be necessary if you use cloudpickle, because they should detect imported names correctly, unlike the default serialization. That doesn't address the concern of writing code that works in both cases, but for snippets like this, several of these serialization decorators change behavior with dill/cloudpickle vs default.

minrk avatar Nov 21 '16 14:11 minrk