Andy Grove

icon indicating a website link https://andygrove.io icon indicating an email [email protected]

icon company @Apple icon location Broomfield, CO icon location Apache Arrow & Apache DataFusion PMC Member. Original creator of Apache DataFusion query engine. Author of https://www.howqueryengineswork.com/

Results 657 comments of Andy Grove

[EPIC] [DISCUSS] Comet timezone handling

> > when the data source is not Parquet: > > when is this true? For Comet sinks, such as: - LocalTableScanExec - CometSparkToColumnarExec - Exchanges - UnionExec - CoalesceExec

[EPIC] [DISCUSS] Comet timezone handling

Another related timezone issue: Cast from string to timestamp: ``` case DataTypes.TimestampType if timeZoneId.exists(tz => tz != "UTC") => Incompatible(Some(s"Cast will use UTC instead of $timeZoneId")) ```

[INTERNAL_ERROR] Custom columnar rules cannot transform shuffle node to something else

@coderfender is planning on working on this one

[INTERNAL_ERROR] Custom columnar rules cannot transform shuffle node to something else

Some notes from debugging this: `CometExecRule` performs the following transformation: ``` INPUT: Exchange hashpartitioning(date_id#5283, product_id#5284, units_sold#5285, store_id#5286, 5), ENSURE_REQUIREMENTS, [plan_id=860] +- HashAggregate(keys=[date_id#5283, product_id#5284, units_sold#5285, store_id#5286], functions=[], output=[date_id#5283, product_id#5284, units_sold#5285, store_id#5286])...

Spark SQL test failures due to timestamp mismatch when LocalTableScan is native

We see the same issue in https://github.com/apache/datafusion-comet/issues/2649

feat: rand expression support

Thanks @akupchinskiy. I plan on reviewing this after the holidays.

feat: rand expression support

I upmerged this PR and re-triggered the workflows. Sorry for the delay @akupchinskiy