spark

spark copied to clipboard

### What changes were proposed in this pull request? ### Why are the changes needed? ### Does this PR introduce _any_ user-facing change? ### How was this patch tested?

LuciferYang

[SPARK-40177][SQL] Simplify condition of form (a==b) || (a==null&&b==null) to a<=>b

3

### What changes were proposed in this pull request? New case is added in Boolean simplification to convert condition of form (a==b) || (a==null&&b==null) to ab. ### Why are the...

ayushi-agarwal

[SPARK-39799][SQL] DataSourceV2: View catalog interface

4

### What changes were proposed in this pull request? ViewCatalog API described in [SPIP](https://docs.google.com/document/d/1XOxFtloiMuW24iqJ-zJnDzHl2KMxipTjJoxleJFz66A/edit?usp=sharing). ### Why are the changes needed? First step towards DataSourceV2 view support. ### Does this PR...

jzhuge

[SPARK-38954][CORE] Implement sharing of cloud credentials among driver and executors

5

### What changes were proposed in this pull request? The PR implements a mechanism to get credentials for a cloud service like AWS from an external credentials provider, and share...

parthchandra

[MINOR][SQL] Remove an unnecessary parameter of the PartitionedFileUtil.splitFiles

3

### What changes were proposed in this pull request? Remove an unnecessary parameter of the` PartitionedFileUtil.splitFiles` ### Why are the changes needed? Make code clearer. ### Does this PR introduce...

weixiuli

[SPARK-40272][CORE]Support service port custom with range

2

### What changes were proposed in this pull request? Port range specification is supported by specifying the starting port and configuring the number of port retries.Introduction of a new parameters:...

chong0929

[WIP][SPARK-40427][SQL] Move LIMIT/OFFSET CheckAnalysis error messages to use the new error framework

1

### What changes were proposed in this pull request? Move LIMIT/OFFSET CheckAnalysis error messages to use the new error framework. This will help improve the usability of Apache Spark as...

dtenedor

[SPARK-34265][PYTHON][SQL] Instrument Python UDFs using SQL metrics

8

### What changes are proposed in this pull request? This proposes to add SQLMetrics instrumentation for Python UDF execution, including Pandas UDF, and related operations such as MapInPandas and MapInArrow....

LucaCanali

DO-NOT-MERGE! May need to split the PR to multiple reviewable size of PRs. Use this only for reference. ### What changes were proposed in this pull request? ### Why are...

HeartSaVioR

[SPARK-40377][SQL] Allow customize maxBroadcastTableBytes and maxBroadcastRows

2

### What changes were proposed in this pull request? Add two parameters to allow customizing maxBroadcastTableBytes and maxBroadcastRows ### Why are the changes needed? Recently, we encountered some driver OOM...

WickLiu