spark

spark copied to clipboard

[SPARK-49723][SQL] Add Variant metrics to the JSON File Scan node

1

### What changes were proposed in this pull request? This pull request adds the following metrics to JSON file scan nodes to collect metrics related to variants being constructed as...

harshmotw-db

[SPARK-49827][SQL] Fetching all partitions from hive metastore in batches

2

### What changes were proposed in this pull request? When there is any predicate missing in getPartitionsbyFilter and it tries to fetch all the partitions, the request is broken into...

Madhukar525722

[SPARK-49953][CORE] Require `errorClass` in `SparkException`

2

### What changes were proposed in this pull request? In the PR, I propose to remove the constructors of `SparkException` that accept a plan string as an error message, and...

MaxGekk

[SPARK-49942][SQL] Rename `errorClass` to `condition` in `classifyException()`

3

### What changes were proposed in this pull request? In the PR, I propose to rename the `errorClass` parameter to `condition` in `JdbcDialect.classifyException` and in `JdbcUtils.classifyException`. ### Why are the...

MaxGekk

### What changes were proposed in this pull request? Call `executeCollect` when the DataFrame is `CollectLimitExec` or `CollectTailExec`. ### Why are the changes needed? For limit execution, `SparkPlan.executeCollect` is more...

xieshuaihu

### What changes were proposed in this pull request? This PR is intended to improve analyzer performance by skip applying the DeduplicateRelations rule when there is a guarantee that there...

ahshahid

### What changes were proposed in this pull request? ### Why are the changes needed? ### Does this PR introduce _any_ user-facing change? ### How was this patch tested? ###...

WweiL

Call skipChildren for nested object

5

### What changes were proposed in this pull request? In `jsonObjectKeys`, we should call `skipChildren()` properly for nested object. ### Why are the changes needed? Currently if `jsonParser.currentName()` is null,...

tedyu

### What changes were proposed in this pull request? This PR adds support for users to provide a Dataframe that can be used to instantiate state for the query in...

jingz-db

[WIP][SPARK-49631][SQL] Remove the ANSI config suggestion in ARITHMETIC_OVERFLOW

2

### What changes were proposed in this pull request? This PR changes the message of ARITHMETIC_OVERFLOW error and adds relevant try expressions: - try_make_interval - try_conv - try_round - try_bround...

mihailom-db