bebe

bebe copied to clipboard

Create PySpark interface

7

Will want to expose the "missing API functions" (e.g. regexp_extract_all) for the PySpark folks too. Think we'll be able to follow [this example](https://github.com/apache/spark/blob/master/python/pyspark/sql/functions.py#L1911-L1924) and expose these relatively easily. Something like...

MrPowers

Should be similar to [what's published for spark-daria](https://mrpowers.github.io/spark-daria/latest/api/#package).

MrPowers

These are functions that are not implemented in Spark, but commonly requested. They should be implemented as Catalyst Expressions so they're performant for the community: Datetime - [X] beginningOfDay -...

MrPowers

Functions that are in Spark SQL and not in Scala API to be implemented

3

We can use this issue to create a list of all the functions that are in Spark SQL, but not in the Scala API for whatever reason. Here's the list...

MrPowers

MrPowers

added copy of logic for bebe_regexp_extract_all to versions previous to 3.1.0

2

Not yet finished (cross spark version buid not inserted) But it allows to use the bebe function to (at least tested with 3.0.1 local) versions previous to 3.1.0. The source...

alfonsorr

Possibly make a Spark 2 release & talk about maintenance

1

I think we should bump to Spark 2.4.5 & see what can features can get added to Spark 2. This way they'll be some JAR files for the Spark 2...

MrPowers