spark

spark copied to clipboard

[SPARK-39970][CORE] Introduce ThrottledLogger to prevent log message flooding caused by network issues

2

### What changes were proposed in this pull request? This PR implemented a ThrottledLogger, a logger with RateLimiters, to prevent log message flooding caused by network issues. In our ThrottledLogger,...

kevin85421

[SPARK-39950][SQL] It's unnecessary to materialize BroadcastQueryStage firstly, because the BroadcastQueryStage does not timeout in AQE.

4

### What changes were proposed in this pull request? In the https://issues.apache.org/jira/browse/SPARK-33933, it materializes BroadcastQueryStage firstly to try to avoid broadcast timeout in AQE, but the BroadcastQueryStage does not timeout...

weixiuli

[SPARK-39921][SQL] SkewJoin--Stream side skew in BroadcastHashJoin

3

### What changes were proposed in this pull request? Solve the data skew on the stream side in `BroadcastHashJoin` - When data skew needs to introduce additional shuffle, support forcibly...

wang-zhun

[SPARK-39942][PYTHON][PS] Need to verify the input nums is integer in nsmallest func

12

The input parameter of nsmallest should be validated as Integer. So I think we might miss this validation. And PySpark will raise Error when we input the strange types into...

bzhaoopenstack

[SPARK-39832][PYTHON] Support column arguments in regexp_replace

6

### What changes were proposed in this pull request? Support either literal Python strings or Column objects for the pattern and replacement arguments for `regexp_replace`. ### Why are the changes...

physinet

[SPARK-39957][CORE] Delay onDisconnected to enable Driver receives ExecutorExitCode

2

### What changes were proposed in this pull request? When onDisconnected is triggered, (1) Delay `RemoveExecutor` for 5 seconds to enable driver receives ExecutorExitCode from slow path (2) Prevent task...

kevin85421

[SPARK-39977][BUILD] Remove unnecessary guava exclusion from jackson-module-scala

1

### What changes were proposed in this pull request? Remove unnecessary guava exclusion from jackson-module-scala ### Why are the changes needed? The exclusion added in SPARK-6149, the recent versions of...

pan3793

[SPARK-39927][BUILD] Upgrade to Avro 1.11.1

7

### What changes were proposed in this pull request? Update the Avro version to 1.11.1 ### Why are the changes needed? To stay up to date with upstream ### Does...

iemejia

[SPARK-39357][SQL] Fix pmCache memory leak caused by IsolatedClassLoader

7

### What changes were proposed in this pull request? * Fixed memory leak caused `RawStore` cleanup mechanism not to take effect due to different `threadLocalMS` instances being manipulated. * Fixed...

tianshuang

[SPARK-38506][SQL] Push partial aggregation through join

5

### What changes were proposed in this pull request? 1. Add a new optimizer rule(PushPartialAggregationThroughJoin) to push the partial aggregation through join. It supports the following cases: - Push down...

wangyum