Hyukjin Kwon

[SPARK-40027][PYTHON][SS][DOCS] Add self-contained examples for pyspark.sql.streaming.readwriter

1

### What changes were proposed in this pull request? This PR proposes to improve the examples in `pyspark.sql.streaming.readwriter` by making each example self-contained with a brief explanation and a bit...

[POC][PYTHON][SS] Arbitrary stateful processing in Structured Streaming with Python

1

### What changes were proposed in this pull request? This PR adds the Python version of `Dataset.groupByKey(...).flatMapGroupsWithState(...)` that is `DataFrame.groupby(...).applyInPandasWithState(...)` in PySpark. TBD Note that documentation will be done in...

We should implement: - [`DataFrame.plot.hexbin`](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.plot.hexbin.html#pandas.DataFrame.plot.hexbin) - [`DataFrame.boxplot`](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.boxplot.html#pandas.DataFrame.boxplot)

[ZEPPELIN-3551] Upgrade Scala to 2.11.12

6

### What is this PR for? This is just to update scala to 2.11.12 which to be consistent with spark (SPARK-24418). This PR takes over and closes #3033 There was...

### What changes were proposed in this pull request? This PR is a followup of https://github.com/apache/spark/pull/45126 that deduplicates the build code. ### Why are the changes needed? For better maintenance...

### What changes were proposed in this pull request? This PR removes outdated comment. We don't use branch-3.3 anymore ### Why are the changes needed? To remove obsolete information ###...

### What changes were proposed in this pull request? TBD ### Why are the changes needed? TBD ### Does this PR introduce _any_ user-facing change? No. ### How was this...

https://endoflife.date/python Python versions up to 3.7 are all EOL. This PR proposes to drop all of EOL versions, and updates its CI.

### What changes were proposed in this pull request? This PR proposes to fix the type hints of `functions.first(..., ignorenulls)` and `functions.last(..., ignorenulls)` to be properly `bool`s. ### Why are...

[SPARK-48960][CONNECT] Makes spark-submit works with Spark connect

1

### What changes were proposed in this pull request? This PR proposes to add the support of `--remote` at `bin/spark-submit` so it can use Spark Connect easily. This PR inclues:...