datafusion-comet icon indicating copy to clipboard operation
datafusion-comet copied to clipboard

Add support for Spark SQL `explode` expression

Open andygrove opened this issue 6 months ago • 1 comments

What is the problem the feature request solves?

Add support for explode:

https://spark.apache.org/docs/latest/api/sql/index.html#explode

explode(expr) - Separates the elements of array expr into multiple rows, or the elements of map expr into multiple rows and columns. Unless specified otherwise, uses the default column name col for elements of the array or key and value for the elements of the map.

Describe the potential solution

The Spark plan contains GenerateExec with a generator field, which will be an Explode class for this case. We can fall back to Spark for any other generator class for now.

Additional context

No response

andygrove avatar Jun 23 '25 23:06 andygrove

Working on it, this might be not the same as other builtin functions as the explode looks like a builtin function but the processing is different in Datafusion

comphead avatar Jun 26 '25 17:06 comphead