Wenchen Fan

icon indicating an email [email protected]

icon company Databricks icon location Hangzhou, China

Results 245 comments of Wenchen Fan

[SPARK-40199][SQL] Provide useful error when projecting a non-null column encounters null value

Spark trusts data nullability in many places (expressions, projection generators, optimizer rules, etc.). It's a lot of efforts to improve error messages for all these places when data does not...

[SPARK-40199][SQL] Provide useful error when projecting a non-null column encounters null value

I made `SparkSession.internalCreateDataFrame` public to easily test the nullability mismatch bug. You can also use a data source to reproduce it. ``` val rdd = sc.makeRDD(Seq(InternalRow(null))) val df = spark.internalCreateDataFrame(rdd,...

[SPARK-40487][SQL] Make defaultJoin in BroadcastNestedLoopJoinExec running in parallel

makes sense, also cc @maryannxue @somani

[SPARK-34265][PYTHON][SQL] Instrument Python UDFs using SQL metrics

thanks, merging to master!

[SPARK-35242][SQL] Support changing session catalog's default database

we should also update `V2SessionCatalog.defaultNamespace`

[SPARK-35242][SQL] Support changing session catalog's default database

> override val defaultNamespace: Array[String] = Array(SQLConf.get.defaultDatabase) Yes

[SPARK-35242][SQL] Support changing session catalog's default database

SGTM!

[SPARK-35242][SQL] Support changing session catalog's default database

thanks, meriging to master!

[SPARK-40360] [WIP] ALREADY_EXISTS and NOT_FOUND exceptions

thanks, merging to master!

[SPARK-40296] Error class for DISTINCT function not found

cc @MaxGekk