Kale Marshal not able to pickle SSLContext

[achal-wwt]

Hello,

I am using AWS EKS cluster and trying to use sagemaker API using Kale. I have created a variable called as tf_estimator using sagemaker api for training and it works fine when I run the pipeline using Kale. However, when I try to pass this object using Kale from one step to another. It fails while saving the model using KaleMarshal as show below

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-9-429d85a94558> in <module>
      3 _kale_marshal_utils.set_kale_data_directory("/marshal")
      4 _kale_marshal_utils.save(ROLE, "ROLE")
----> 5 _kale_marshal_utils.save(tf_estimator, "tf_estimator")
      6 # -----------------------DATA SAVING END-----------------------------------
      7 

/usr/local/lib/python3.6/dist-packages/kale/marshal/utils.py in save(obj, obj_name)
     37     resource_save(
     38         obj,
---> 39         os.path.join(KALE_DATA_DIRECTORY, obj_name))
     40 
     41 
/usr/local/lib/python3.6/dist-packages/kale/marshal/dispatchers.py in __call__(self, s, *args, **kwargs)
     81     def __call__(self, s, *args, **kwargs):
     82         """Run dispatch."""
---> 83         return self.dispatch(s)(s, *args, **kwargs)
     84 
     85 

Also the end of the stack trace is as shown below, shows that SSLContext can't be pickle.

/usr/lib/python3.6/pickle.py in save_dict(self, obj)
    819 
    820         self.memoize(obj)
--> 821         self._batch_setitems(obj.items())
    822 
    823     dispatch[dict] = save_dict
/usr/lib/python3.6/pickle.py in _batch_setitems(self, items)
    845                 for k, v in tmp:
    846                     save(k)
--> 847                     save(v)
    848                 write(SETITEMS)
    849             elif n:
/usr/lib/python3.6/pickle.py in save(self, obj, save_persistent_id)
    494             reduce = getattr(obj, "__reduce_ex__", None)
    495             if reduce is not None:
--> 496                 rv = reduce(self.proto)
    497             else:
    498                 reduce = getattr(obj, "__reduce__", None)
TypeError: can't pickle SSLContext objects

Can we resolve this issue? If we can, are you planning to get rid of these errors in future updates.

[StefanoFioravanzo]

Hi @achal-wwt sorry for replying so late. Could you please provide the simplest, self-contained snippet to create a similar Tensorflow estimator, so that I can try this myself and reproduce?

[yoavhalevi]

Hi @StefanoFioravanzo ,

I have ran into a similar issue: The error was: can't pickle SSLContext objects

the object I was trying to pass between pipeline steps have a connection to a mongoDB via a pymongo object called MongoClient (see it here) and a connection to Microsoft Azure blob via their sdk object BlockBlobService

i was trying to create a reproducing example with a notebook that looks like this: but received a different Kale error:

During data passing, Kale could not marshal the following object:
  - path: '/marshal/client'
  - type: '<class 'pymongo.mongo_client.MongoClient'>'
The error was: no default __reduce__ due to non-trivial __cinit__

which I believe is also related to the pickling part of the object serialization that Kale does.