scoop
scoop copied to clipboard
Can't pickle decorated function
I am using decorated functions within the DEAP framework to limit the tree size and it seems SCOOP does not support pickling decorated functions.
@apply_decorator
def mutUniform(*args, **kwargs):
return gp.mutUniform(*args, **kwargs)
@apply_decorator
def mutUniform(*args, **kwargs):
return gp.mutUniform(*args, **kwargs)
toolbox.register("mutate", mutUniform, expr=toolbox.expr_mut, pset=pset)
output:
[2017-02-21 11:47:39,593] scoopzmq (b'127.0.0.1:57909') WARNING Pickling Error: Can't pickle <function mutUniform at 0x7f8dc2e7a730>: it's not the same object as __main__.mutUniform
scoop._comm.scoopexceptions.ReferenceBroken: This element could not be pickled: FutureId(worker=b'127.0.0.1:57909', rank=1):partial(0,)=None.
[2017-02-21 11:47:42,399] scoopzmq (b'127.0.0.1:57909') ERROR A worker exited unexpectedly. Read the worker logs for more information. SCOOP pool will now shutdown.
My code works fine with the built in Python 3 multiprocessing pool.map function btw.
hmm, without a mcve there's not really a good way to help you...
- how's
apply_decorator
,toolbox.expr_mut
andpset
defined? - did you intentionally list the first part of the code twice?
that said, from my experience it can help to statically define a composed function instead of loosely coupling it via toolbox... (this means use def myfunc(...)
and later toolbox.register('myfunc', myfunc)
instead of toolbox.register('myfunc', base_func, some_crazy_param_bindings, making_a_whole_new_function)
).