spark

spark copied to clipboard

[SPARK-39883][SQL][TESTS] Add DataFrame function parity check

1

### What changes were proposed in this pull request? This PR adds a test that compares the available list of DataFrame functions in org.apache.spark.sql.functions with the SQL function registry. This...

aray

[SPARK-37210][CORE][SQL] Allow forced use of staging directory

5

### What changes were proposed in this pull request? Add `forceUseStagingDir` config to force use of staging dir when writing. When setting `forceUseStagingDir` to true, I set `committerOutputPath` to staging...

wForget

[SPARK-38639][HIVE]Ignore the corrupted rows that failed to deserialize in hive sequence table

5

### What changes were proposed in this pull request? Original pr: https://github.com/apache/spark/pull/35963 When reading the hive sequence table, you can switch whether to skip corrupt records that fail to be...

caican00

[SPARK-39938][PYTHON][PS] Accept all inputs of prefix/suffix which implement __str__ in add_predix/add_suffix

2

We need to follow the pandas behavior of prefix/suffix parameter validation in add_prefix/add_suffix. Now, we force to validate it as a String type. But pandas looks all values which can...

bzhaoopenstack

[SPARK-39941][PYTHON][PS] window and min_periods should be integer in rolling func

2

window and min_periods parameters is not be validated in rolling function. ### What changes were proposed in this pull request? Validate the said 2 parameters to be a integer only...

bzhaoopenstack

[SPARK-39930][SQL] Introduce Cache Hints

2

### What changes were proposed in this pull request? This ticket intends to add query hints for cache behaviors, users can perform actions like the use/skip/cache/uncache, etc on the cached...

yaooqinn

[SPARK-39925][SQL] Add array_sort(column, comparator) overload to DataFrame operations

1

### What changes were proposed in this pull request? Adding a new `array_sort` overload to `org.apache.spark.sql.functions` that matches the new overload defined in [SPARK-29020](https://issues.apache.org/jira/browse/SPARK-29020) and added via #25728. ### Why...

brandondahler

[SPARK-39821][PYTHON][PS] Fix error during using DatetimeIndex

11

Pandas disallow conversion between datetime/timedelta and conversions for any datetimelike to float. This will raise error in PYSPARK, during we simply call a DatetimeIndex. So we need to avoid to...

bzhaoopenstack

[SPARK-39822][PYTHON][PS] Provide a good feedback to users

6

Provide a graceful error msg to users when they build Index with different dtypes. ### What changes were proposed in this pull request? Raise a graceful error when users create...

bzhaoopenstack

[POC][PYTHON][SS] Arbitrary stateful processing in Structured Streaming with Python

1

### What changes were proposed in this pull request? This PR adds the Python version of `Dataset.groupByKey(...).flatMapGroupsWithState(...)` that is `DataFrame.groupby(...).applyInPandasWithState(...)` in PySpark. TBD Note that documentation will be done in...

HyukjinKwon