spark

spark copied to clipboard

[SPARK-40027][PYTHON][SS][DOCS] Add self-contained examples for pyspark.sql.streaming.readwriter

1

### What changes were proposed in this pull request? This PR proposes to improve the examples in `pyspark.sql.streaming.readwriter` by making each example self-contained with a brief explanation and a bit...

HyukjinKwon

[SPARK-39887][SQL] RemoveRedundantAliases should keep aliases that make the output of projection nodes unique

5

### What changes were proposed in this pull request? Keep the output attributes of a `Union` node's first child in the `RemoveRedundantAliases` rule to avoid correctness issues. ### Why are...

peter-toth

[SPARK-39984][CORE] Check workerLastHeartbeat with master before HeartbeatReceiver expires an executor

10

### What changes were proposed in this pull request? This PR aims to provide a method to lower the timeout. Our solution is to ask master for worker’s heartbeat when...

kevin85421

[SPARK-40028][SQL] Add binary examples for string expressions

1

### What changes were proposed in this pull request? Currently, Spark have many string expressions support binary type, but missing examples of binary. This PR will add examples of binary...

beliefer

[SPARK-39966][SQL] Use V2 Filter in SupportsDelete

2

### What changes were proposed in this pull request? Migrate SupportsDelete to use V2 Filter ### Why are the changes needed? this is part of the V2Filter migration work ###...

huaxingao

### What changes were proposed in this pull request? When the try_cast() syntax is equivalent to the non-ansi cast, convert TryCast as Cast without wrapping with "TryEval" expression. ### Why...

gengliangwang

[SPARK-39976][SQL] ArrayIntersect should handle null in left expression correctly

6

### What changes were proposed in this pull request? `ArrayInterscet` miss judge if null contains in right expression's hash set. ``` >>> a = [1, 2, 3] >>> b =...

AngersZhuuuu

### What changes were proposed in this pull request? This pr aims upgrade scala-maven-plugin to 4.7.1 ### Why are the changes needed? This version brings some bug fix related to...

LuciferYang

[SPARK-40020][SQL] Centralize the code of qualifying identifiers in SessionCatalog

2

### What changes were proposed in this pull request? This PR is a refactor of `SessionCatalog`. It centralizes the code of qualifying identifiers in one place: `SessionCatalog.qualifyIdentifier`. Then we can...

cloud-fan

[SPARK-39983][CORE][SQL] Do not cache unserialized broadcast relations on the driver

8

### What changes were proposed in this pull request? This PR addresses the issue raised in https://issues.apache.org/jira/browse/SPARK-39983 - broadcast relations should not be cached on the driver as they are...

alex-balikov