sparkmagic icon indicating copy to clipboard operation
sparkmagic copied to clipboard

Support for passing values other than strings/pandas dataframes

Open dylanwilder opened this issue 4 years ago • 0 comments

Is your feature request related to a problem? Please describe. Currently, sparkmagic only supports transporting strings and dataframes back and forth. This is pretty limiting. It would be great to be able to push arbitrary objects e.g. trained scikit models. It's unclear why this hasn't been implemented. Perhaps it's considered risky if environments are not consistent on both sides, but we take care to maintain parity as we view this as best practice anyway. This issue seems preferable to not being able to do this at all.

Describe the solution you'd like use dill or cloud pickle to push arbitrary objects. remove the need to specify object type when calling

Describe alternatives you've considered implementing this ourselves in a local version. we've currently done this but would prefer to contribute back to the source repo. wanted to hear your thoughts.

dylanwilder avatar Aug 17 '20 18:08 dylanwilder