Matthew Powers

Possibly refactor import statements in Python documentation

2

[The documentation](https://delta-io.github.io/delta-rs/python/usage.html#loading-a-delta-table) currently uses import statements like this: ```python from deltalake import DeltaTable from deltalake import DataCatalog ``` I usually see Python import statements like this: ```python import pandas as...

Create PySpark interface

7

Will want to expose the "missing API functions" (e.g. regexp_extract_all) for the PySpark folks too. Think we'll be able to follow [this example](https://github.com/apache/spark/blob/master/python/pyspark/sql/functions.py#L1911-L1924) and expose these relatively easily. Something like...

Should be similar to [what's published for spark-daria](https://mrpowers.github.io/spark-daria/latest/api/#package).

These are functions that are not implemented in Spark, but commonly requested. They should be implemented as Catalyst Expressions so they're performant for the community: Datetime - [X] beginningOfDay -...

Functions that are in Spark SQL and not in Scala API to be implemented

3

We can use this issue to create a list of all the functions that are in Spark SQL, but not in the Scala API for whatever reason. Here's the list...

Possibly make a Spark 2 release & talk about maintenance

1

I think we should bump to Spark 2.4.5 & see what can features can get added to Spark 2. This way they'll be some JAR files for the Spark 2...

Implement all the DataFrame comparison options for approx comparisons as well

1

* ignore_nullable * allow_nan_equality * ignore_column_order * ignore_row_order

This project is setup for Poetry development. Is there a way to setup this project for conda development as well? I want to keep using Poetry for my personal development...

Possibly set row comparison to true by default for DataFrame comparisons

1

Maybe I'm the outlier, but I consider the more intuitive check -- especially for testing purposes -- to ignore order. If some function produces a DataFrame that I want to...