Oleks V comments

Results 186 comments of


                                            Oleks V

DataFusion 37.0.0 upgrade

Related to https://github.com/apache/arrow-datafusion-comet/pull/239/

Parquet column with integer logical type cannot read as Spark date column

Thanks @okue for reporting this

Parquet column with integer logical type cannot read as Spark date column

Easy test to reproduce ``` Seq(15901).toDF("dt").write.parquet("/tmp/dt") spark.read.schema("dt date").parquet("/tmp/dt").createOrReplaceTempView("v") spark.sql("select min(dt) from v").show(false) ```

[EPIC] Native execution for Windows functions

Depends on https://github.com/apache/arrow-datafusion/issues/9055

Use stabalized aws-sdk and clap versions in `datafusion-cli`

ouch, MSRV failed. https://github.com/smithy-lang/smithy-rs/blob/main/rust-toolchain.toml they require 1.74 and we support 1.72. The latest rustc is 1.76 and we agreed I think to keep MSRV to be 4 versions behind

Test CometDriverPlugin in Kube/Yarn

Related to https://github.com/apache/datafusion-comet/issues/605

Test CometDriverPlugin in Kube/Yarn

Thanks @orthoxerox for running the experiment. Please correct me if I'm wrong so both Comet memory params increases the YARN container memory usage, at the same time `spark.executor.memoryOverhead` shows the...

Test CometDriverPlugin in Kube/Yarn

Historical server has slightly other implementation. Great, thanks for the help @orthoxerox I'm planning to document this today

fix: modulo op with negative zero divisor produces Nan

Filed https://github.com/apache/datafusion/issues/11051 lets see

fix: modulo op with negative zero divisor produces Nan

@kazuyukitanimura can we close it?