YouTokenToMe icon indicating copy to clipboard operation
YouTokenToMe copied to clipboard

Published 20 hours ago • verified publisher icon VKCOM

Is it possible to wrap the calls in pyspark UDFs?

Open AlexGidiotis opened this issue 5 years ago • 1 comments

This is a great tool for data preprocessing. I was wondering if it is possible to use it distributed by wrapping the inference function (not the training part) into spark (pyspark) UDFs.

AlexGidiotis avatar Dec 04 '19 08:12 AlexGidiotis

An update: I was trying to implement this myself and looks like this is not possible currently because the objects cannot be pickled. This is now related to issue #32

AlexGidiotis avatar Jan 14 '20 09:01 AlexGidiotis