spark

spark copied to clipboard

[SPARK-39931][PYTHON][WIP] Improve applyInPandas performance for very small groups

3

### What changes were proposed in this pull request? Given a batch size to `applyInPandas`, multiple groups are sent to Python UDF at once if they are very small. This...

EnricoMi

[SPARK-39607][SQL][DSV2] Distribution and ordering support V2 function in writing

4

### What changes were proposed in this pull request? Add new feature, make distribution and ordering support V2 function in writing. Currently, the rule `V2Writes` support converting `ApplyTransform` to `TransformExpression`...

pan3793

[SPARK-39635][SQL] Support driver metrics in DS v2 custom metric API

6

### What changes were proposed in this pull request? Expose custom metrics available on driver from DS v2 data sources on SQL UI. ### Why are the changes needed? https://github.com/apache/spark/commit/115ed89a3cd75faea3e6e29fb580da45309c0f31...

karuppayya

[SPARK-38474][CORE] Use error class in org.apache.spark.security

5

### Why are the changes needed? Refactored exceptions thrown in the package `org.apache.spark.security` in to be in inline Spark error class framework ### Does this PR introduce _any_ user-facing change?...

sandeep-katta

[SPARK-39710][SQL] Support push local topK through outer join

1

### What changes were proposed in this pull request? - Pull out the pattern of `TakeOrderedAndProjectExec` to `ExtractTopK` - Add a new rule `PushLocalTopKThroughOuterJoin` which matches the `ExtractTopK` pattern -...

ulysses-you

[SPARK-37980][SQL] Extend METADATA column to support row indexes

2

### What changes were proposed in this pull request? This change adds `row_index` column to `_metadata` struct. This column allows us to uniquely identify rows read from a given file...

ala

[SPARK-38909][CORE][YARN] Encapsulate `LevelDB` used to store remote/external shuffle state as `DB`

12

### What changes were proposed in this pull request? `ExternalShuffleBlockResolver`, `YarnShuffleService` and `RemoteBlockPushResolver` use `LevelDB` directly, this is not conducive to extending the use of `RocksDB` in this scenario. This...

LuciferYang

[SPARK-39866][SQL] Memory leak when closing a session of Spark ThriftServer

8

### What changes were proposed in this pull request? Fixed memory leak caused by not clearing the FileStatusCache created by calling FileStatusCache.getOrCreate when close a SparkSession of Spark Thrift Server....

Resol1992

[SPARK-25342][CORE][SQL]Support rolling back a result stage and rerunning all result tasks when writing files

3

### What changes were proposed in this pull request? From this pr:https://github.com/apache/spark/pull/22112, we learn that currently we can't rollback and rerun a result stage, and just fail. And this new...

caican00

[SPARK-39895][SQL][PYTHON][R] Support multiple column drop

5

### What changes were proposed in this pull request? Pyspark dataframe drop has following signature: `def drop(self, *cols: "ColumnOrName") -> "DataFrame":` However when we try to pass multiple Column types...

santosh-d3vpl3x