ipyparallel
ipyparallel copied to clipboard
`dill` and `cloudpickle` don't work with `require`
from ipyparallel import Client, require
client = Client()
lview = client.load_balanced_view()
dview = client[:]
dview.use_cloudpickle()
@require('types')
def func(x):
return types.SimpleNamespace(n=x)
res = lview.map(func, range(10))
res.result()
gives
---------------------------------------------------------------------------NameError Traceback (most recent call last)<string> in <module>()
/home/bnijholt/anaconda3/lib/python3.5/site-packages/ipyparallel/client/remotefunction.py in <lambda>(f, *sequences)
248 if _mapping:
249 if sys.version_info[0] >= 3:
--> 250 f = lambda f, *sequences: list(map(f, *sequences))
251 else:
252 f = map
/home/bnijholt/anaconda3/lib/python3.5/site-packages/ipyparallel/controller/dependency.py in __call__(self, *args, **kwargs)
71
72 def __call__(self, *args, **kwargs):
---> 73 return self.f(*args, **kwargs)
74
75 if not py3compat.PY3:
<ipython-input-8-293a3326374e> in func(x)
NameError: name 'types' is not defined
These decorators shouldn't be necessary if you use cloudpickle, because they should detect imported names correctly, unlike the default serialization. That doesn't address the concern of writing code that works in both cases, but for snippets like this, several of these serialization decorators change behavior with dill/cloudpickle vs default.