sparkmagic
sparkmagic copied to clipboard
Support for passing values other than strings/pandas dataframes
Is your feature request related to a problem? Please describe. Currently, sparkmagic only supports transporting strings and dataframes back and forth. This is pretty limiting. It would be great to be able to push arbitrary objects e.g. trained scikit models. It's unclear why this hasn't been implemented. Perhaps it's considered risky if environments are not consistent on both sides, but we take care to maintain parity as we view this as best practice anyway. This issue seems preferable to not being able to do this at all.
Describe the solution you'd like use dill or cloud pickle to push arbitrary objects. remove the need to specify object type when calling
Describe alternatives you've considered implementing this ourselves in a local version. we've currently done this but would prefer to contribute back to the source repo. wanted to hear your thoughts.