_pickle.PicklingError: Can't pickle <function <lambda> at ...>

[davified]

Hi there, first of all, thank you for the great work on metaflow!

We're using metaflow and we've encountered the following error:

[5f9e2b9f-a7fe-416d-8471-cdd6f265bd2b]   File "/app/metaflow/metaflow/datastore/datastore.py", line 455, in <lambda>
[5f9e2b9f-a7fe-416d-8471-cdd6f265bd2b]     transformable_obj.transform(lambda x: pickle.dumps(x, protocol=2))
[5f9e2b9f-a7fe-416d-8471-cdd6f265bd2b] _pickle.PicklingError: Can't pickle <function <lambda> at 0x7f91410cba70>:
 attribute lookup <lambda> on some_thing.my_function failed

The cause of this is that somewhere in our class, SomeThing, we define my_function as a python lambda. For now, we're using the workaround suggested in issue #391 , but it's causing our Flow to have lots of our custom serialisation / deserialisation using dill. We do this quite extensively in our codebase, and would prefer not to change that if possible.

We were wondering if metaflow could support serialisation of classes with lambda functions, for example by using dill instead of pickle for (de)serialising objects?

Thank you for reading, and thank you again for the great work on metaflow :-)

[calmacx]

I also ran into this problem. Editing task_datastore.py and replacing import pickle with import dill as pickle got things working fine for me without having to write lots of custom serialisation/deserialisation... just incase anyone else stumbles across this like I did and wants a temp fix.

[ksachdeva11]

I am facing this similar issue while using Lime with Metaflow.

self.explainer = lime.lime_tabular.LimeTabularExplainer(X.values, feature_names=X.columns.tolist())

Wondering how could this be resolved. Any help would be appreciated. Thanks