spark

spark copied to clipboard

[SPARK-39851][SQL] Improve join stats estimation if one side can keep uniqueness

1

### What changes were proposed in this pull request? This PR improves join stats estimation if one side can keep uniqueness. A common case is: ```sql SELECT i_item_sk ss_item_sk FROM...

wangyum

[SPARK-39750][SQL] Enable `spark.sql.cbo.enabled` by default

11

### What changes were proposed in this pull request? This PR enable `spark.sql.cbo.enabled` by default. ### Why are the changes needed? 1. Enable CBO to get better performance, we've enabled...

wangyum

[SPARK-39850][YARN]Print applicationId once applied from yarn rm

1

### What changes were proposed in this pull request? Print the application id after getting application from yarn. ### Why are the changes needed? it's useful for users to find...

DongweiLee

[SPARK-39678][SQL] Improve stats estimation for v2 tables

13

### What changes were proposed in this pull request? We should propagate the row count stats in SizeInBytesOnlyStatsPlanVisitor if available. Row counts are propagated from connectors to spark in case...

singhpk234

spark-submit: throw an error when duplicate argument is provided

1

### What changes were proposed in this pull request? I propose this PR to warn the user when he is not using correctly the spark submit CLI. The idea here...

sybernatus

[SPARK-39838][SQL] Preserve explicit empty column metadata

1

### What changes were proposed in this pull request? This PR fixes the issue SPARK-39838 where an explicitly set empty Column Metadata object would be optimized away, making it impossible...

kupferk

[SPARK-39654][PYTHON][PS] Parameters quotechar and escapechar need limitation in read_csv

1

Currently, as doc description of pyspark pandas and pandas itself, these 2 options are only support 1-char input, so let's make pyspark pandas follow this behavior. Issue affected: https://issues.apache.org/jira/browse/SPARK-39654 ###...

bzhaoopenstack

[SPARK-39820][SQL] Add the Support for Custom Authentication for Thrift Server using Http Servlet Request

2

### What changes were proposed in this pull request? As per the CustomAuthentication(hive.server2.transport.mode=HTTP) in Thrift Server only String username and String password is supported in the authenticate method of PasswdAuthenticationProvider(https://github.com/apache/spark/blob/master/sql/hive-thriftserver/src/main/java/org/apache/hive/service/auth/PasswdAuthenticationProvider.java#L37)....

SaurabhChawla100

[SPARK-37019][SQL] Add codegen support to array higher-order functions

7

### What changes were proposed in this pull request? This PR adds codegen support to array based higher order functions except ArraySort. This is my first time playing around with...

Kimahriman

[SPARK-39091][SQL] Updating specific SQL Expression traits that don't compose when multiple are extended due to nodePatterns being final.

6

When creating classes that extend multiple `Expression` traits, they have an inheritance conflict due to `nodePatterns` being different values. `nodePatterns` is also final, which makes the inheritance conflict unavoidable. This...

neilagupta