spark

spark copied to clipboard

[SPARK-39696][CORE] Ensure Concurrent r/w `TaskMetrics` not throw Exception

17

### What changes were proposed in this pull request? This PR changes the declaration type of `TaskMetrics#externalAccums` from `s.c.mutable.ArrayBuffer` to `j.u.concurrent.CopyOnWriteArrayList` to ensure that errors described in SPARK-39696(`java.util.ConcurrentModificationException: mutation occurred...

LuciferYang

[SPARK-39743][DOCS] Updated some spark.io.compression configuration descriptions to clarify parameter applica…

5

…tion scope ### What changes were proposed in this pull request? Updated some `spark.io.compression` configuration descriptions to clarify parameter application scope. ### Why are the changes needed? Users are easily...

zzzzming95

[SPARK-40001][SQL] Add config to make DEFAULT values in JSON tables mutually exclusive with SQLConf.JSON_GENERATOR_IGNORE_NULL_FIELDS

2

### What changes were proposed in this pull request? Add config to make DEFAULT values in JSON tables mutually exclusive with SQLConf.JSON_GENERATOR_IGNORE_NULL_FIELDS. When this new config is true, allow DEFAULT...

dtenedor

### What changes were proposed in this pull request? Followup for https://github.com/apache/spark/pull/37294, to improve test coverage by adding more tests. ### Why are the changes needed? To improve the test...

itholic

[SPARK-40000][SQL] Add config to toggle whether to automatically add default values for INSERTs without user-specified fields

2

### What changes were proposed in this pull request? Add config to toggle whether to automatically add default values for INSERTs without user-specified fields. Example: ``` CREATE TABLE t (a...

dtenedor

[SPARK-39985][SQL] Enable implicit DEFAULT column values in inserts from DataFrames

2

### What changes were proposed in this pull request? Enable implicit DEFAULT column values in inserts from DataFrames. This mostly already worked since the DataFrame inserts already converted to LogicalPlans....

dtenedor

[SPARK-39997][SQL] Fix ParquetSchemaConverter fails match schema by id

3

currently, match parquet schema by id fails under certain case no new unit test added ### What changes were proposed in this pull request? in this PR, fixed cases where...

zinking

[SPARK-39876][SQL] Add UNPIVOT to SQL syntax

6

### What changes were proposed in this pull request? This adds UNPIVOT clause to SQL syntax. It follows the same syntax as [BigQuery](https://cloud.google.com/bigquery/docs/reference/standard-sql/query-syntax#unpivot_operator), [T-SQL](https://docs.microsoft.com/en-us/sql/t-sql/queries/from-using-pivot-and-unpivot?view=sql-server-ver15#unpivot-example), [Oracle](https://www.oracletutorial.com/oracle-basics/oracle-unpivot/): ``` FROM ... [ unpivot_clause...

EnricoMi

[SPARK-39991][SQL][AQE] Use available column statistics from completed query stages

1

### What changes were proposed in this pull request? AQE uses statistics from completed query stages and feeds them back into the logical optimizer. AQE currently only uses `dataSize` and...

andygrove

[SPARK-39854][SQL] replaceWithAliases should keep the original children for Generate

5

### What changes were proposed in this pull request? The current implementation of the `replaceWithAliases` method in `NestedColumnAliasing` replaces the children of the current plan with mapped aliases. When the...

jiaji-wu