Wenchen Fan comments

Results 245 comments of


                                            Wenchen Fan

[SPARK-40216][SQL] Extract common `ParquetUtils.prepareWrite` method to deduplicate code in `ParquetFileFormat` and `ParquetWrite`

@sadikovi can you take a look?

[SPARK-40216][SQL] Extract common `ParquetUtils.prepareWrite` method to deduplicate code in `ParquetFileFormat` and `ParquetWrite`

thanks, merging to master!

[SPARK-40355][SQL] Improve pushdown for orc & parquet when cast scenario

We have an optimizer rule `UnwrapCastInBinaryComparison`, does it solve your problem? And can you give some code pointers to explain why we have inconsistency between data and partition filters today?

[SPARK-40382][SQL] Group distinct aggregate expressions by semantically equivalent children in `RewriteDistinctAggregates`

thanks, merging to master!

[SPARK-40311][SQL][PYTHON] Add withColumnsRenamed to scala and pyspark API

looks fine, cc @viirya @dongjoon-hyun

[SPARK-40177][SQL] Simplify condition of form (a==b) || (a==null&&b==null) to a<=>b

cc @sigmod @wangyum

[SPARK-40474][SQL] Infer columns with mixed date and timestamp as String in CSV schema inference

There are many cases to consider here: 1) the CSV data is pure date, pure timestamp, or a mixture. 2) the user specifies datetime pattern or not. 1. pure date...

[SPARK-40474][SQL] Infer columns with mixed date and timestamp as String in CSV schema inference

Can we update PR description to mention that `prefersDate` is by default true now?

[SPARK-40474][SQL] Infer columns with mixed date and timestamp as String in CSV schema inference

thanks, merging to master!

[SPARK-40465][SQL] Refactor Decimal so as we can use other underlying implementation

The abstraction makes sense to me. The new `DecimalOperation` layer adds an indirection and may hurt performance, though I think the overhead is minor as Java decimal operation is already...