Support for images from spark DataFrame

[andompesta]

Hi all,

I'm trying to train a recommendation system for user-items using Databricks as distributed computing platform. I'm applying proprocessing steps to my user related features as well as to the images, however, spark does not work well with numpy arrays. Petastorm approach to support image preprocessing on spark is to encode all numpy arrays as bytearray. Unfortunately, your dataloader does not have support for bytearray

is it possible to add support for custom unpacking function ?