spark

spark copied to clipboard

[SPARK-40485][SQL] Extend the partitioning options of the JDBC data source

1

### What changes were proposed in this pull request? This proposes to extend the available partitioning options for the JDBC data source. ### Why are the changes needed? Partitioning options...

LucaCanali

[SPARK-39399] [CORE] [K8S]: Fix proxy-user authentication for Spark on k8s in cluster deploy mode

2

### What changes were proposed in this pull request? The PR fixes the authentication failure of the proxy user on driver side while accessing kerberized hdfs through spark on k8s...

shrprasa

[SPARK-40199][SQL] Provide useful error when projecting a non-null column encounters null value

7

### What changes were proposed in this pull request? This modifies `GenerateUnsafeProjection` to wrap projections of non-null fields in try-catch blocks which swallow any `NullPointerException` that is thrown, and instead...

xkrogen

[SPARK-39069][SQL] Enhance ConstantPropagation to replace constants in inequality predicates

2

### What changes were proposed in this pull request? `ConstantPropagation` currently only supports replace constants in equality predicates. For example: `i = 5 AND j = i + 3` ->...

wangyum

### What changes were proposed in this pull request? Extend Catalyst's type system by a new decimal type: Decimal128Type with Int128. ### Why are the changes needed? Spark SQL today...

beliefer

[SPARK-39814] Use AmazonKinesisClientBuilder.withCredentials instead of new AmazonKinesisClient(credentials)

5

### What changes were proposed in this pull request? Use AmazonKinesisClientBuilder.withCredentials instead of new AmazonKinesisClient(credentials) to cleanup following compilation warnings: > [warn] /home/runner/work/spark/spark/connector/kinesis-asl/src/main/scala/org/apache/spark/examples/streaming/KinesisWordCountASL.scala:108:25: [deprecation @ org.apache.spark.examples.streaming.KinesisWordCountASL.main.kinesisClient | origin=com.amazonaws.services.kinesis.AmazonKinesisClient. | version=]...

panbingkun

### What changes were proposed in this pull request? Add create datasource table options docs in `docs/sql-ref-syntax-ddl-create-table-datasource.md`. ### Why are the changes needed? improve the missed docs and example. ###...

ulysses-you

[SPARK-40428][CORE][WIP] Fix shutdown hook in the CoarseGrainedSchedulerBackend

3

### What changes were proposed in this pull request? Fix the shutdown hook call through to CoarseGrainedSchedulerBackend ### Why are the changes needed? Sometimes if the driver shuts down abnormally...

holdenk

[SPARK-40469][CORE] Avoid creating directory failures

1

### What changes were proposed in this pull request? This PR replace `Files.createDirectory` with `Files.createDirectories`. ### Why are the changes needed? To avoid creating directory failures if the parent directory...

wangyum

[SPARK-40364][CORE] Use the unified `DBProvider#initDB ` method

13

### What changes were proposed in this pull request? There are 2 `initDB` in `DBProvider` 1. `DB initDB(DBBackend dbBackend, File dbFile, StoreVersion version, ObjectMapper mapper)` 2. `DB initDB(DBBackend dbBackend, File...

LuciferYang