spark

spark copied to clipboard

[SPARK-39624][SQL] Support coalesce partition through CartesianProduct

3

### What changes were proposed in this pull request? Coalesce paritition for every group ### Why are the changes needed? With CartesianProduct, CoalesceShufflePartitions can not optimize it. Such as sql...

lsm1

[SPARK-38098][PYTHON] Add support for ArrayType of nested StructType to arrow-based conversion

7

### What changes were proposed in this pull request? This proposes to add support for ArrayType of nested StructType to arrow-based conversion. This allows Pandas UDFs, mapInArrow UDFs, and toPandas...

LucaCanali

[SPARK-39893][SQL] Push limit 1 to the aggregate's child plan if grouping expressions and aggregate expressions are foldable

1

### What changes were proposed in this pull request? If all group expressions are foldable, the result of this aggregate will always be OneRowRelation. And if all aggregate expressions are...

wankunde

### What changes were proposed in this pull request? The same `UnsupportedOperationException` is constructed in 3 places in `AccumulatorV2`, this pr extract an helper method to deduplicate code. ### Why...

LuciferYang

[SPARK-38932][SQL] Datasource v2 support report distinct keys

2

### What changes were proposed in this pull request? - Add a new mix in interface `SupportsReportDistinctKeys` for datasource v2 - Add a new method `reportDistinctKeysSet` in `LeafNode` - Override...

ulysses-you

[SPARK-33326][SQL] Fix the problem of writing hive partition table without updating metadata information

1

### What changes were proposed in this pull request? Fix the problem of writing hive partition table without updating metadata information ### Why are the changes needed? - This patch...

RS131419

[SPARK-39389]Filesystem closed should not be considered as corrupt files

9

### What changes were proposed in this pull request? Should throw the error if caused by Filesystem closed when we enable ignoreCorruptFiles ### Why are the changes needed? If we...

boneanxs

[SPARK-38699][SQL] Use error classes in the execution errors of dictionary encoding

1

…onary encoding#38699 ### What changes were proposed in this pull request? Migrate the following errors in QueryExecutionErrors: useDictionaryEncodingWhenDictionaryOverflowError -> DICTIONARY_OVERFLOW_ERROR ### Why are the changes needed? Porting execution errors of...

goutam-git

[SPARK-39894][SQL] Combine the similar binary comparison in boolean expression.

1

### What changes were proposed in this pull request? If boolean expression have two similar binary comparisons have the same symbol(e.g., >) and one side is literal and connected with...

beliefer

[SPARK-38213][STREAMING] Adding KafkaSink Metrics feature

9

### What changes were proposed in this pull request? KafkaSink Metrics feature is added ### Why are the changes needed? KafkaSink Metrics feature will be useful to collect and store...

senthh