[FEATURE REQUEST]: Easy way to convert Spark DataFrame to Analysis.DataFrame using Arrow

[bolcman]

Hi,

I'm looking for an easy way to convert Spark DF to Analysis DF using Arrow. I have some cases in which I need to "materialize" Spark DF as fast as possible, do some local transformations, then distribute the payload again. Based on description this can work pretty nice with Arrow batches between so the overhead can be minimized.

ideally a utility function would look like this: var fxDataFrame = sparkDataFrame.ToFxDF();

and maybe reversed function as well.

var sparkDataFrame= fxDataFrame .ToSparkDF();

Any thoughts ?

Thanks, Aleks