Andy Grove

Investigate OpenDAL features

1

### What is the problem the feature request solves? We would like to learn more about OpenDAL for supporting HDFS, S3, and other file stores. Some questions to answer: -...

### What is the problem the feature request solves? We need to support custom authentication with external file systems *(Access hdfs/hadoop-aws via JNI)* ## 1. HDFS support via `fs-hdfs` -...

[ANSI] Include original SQL in error messages

5

### What is the problem the feature request solves? Spark includes the original SQL query in error messages with hints to which part of the query failed. ``` scala> spark.sql("select...

### What is the problem the feature request solves? The Spark diffs contain logic for the following configs: - `ENABLE_COMET_SCAN_ONLY`: This is no longer set when running Spark SQL tests...

### Describe the bug Spark SQL test failure when I enable `spark.comet.sparkToColumnar.enabled`. ``` [info] - SPARK-7595: Window will cause resolve failed with self join *** FAILED *** (116 milliseconds) [info]...

Add support for Spark SQL `explode` expression

1

### What is the problem the feature request solves? Add support for `explode`: https://spark.apache.org/docs/latest/api/sql/index.html#explode > explode(expr) - Separates the elements of array expr into multiple rows, or the elements of...

chore: Add memory reservation debug logging and visualization

5

## Which issue does this PR close? Closes #. ## Rationale for this change Debugging. ``` [Task 486] MemoryPool[ExternalSorter[6]].try_grow(256232960) returning Ok [Task 486] MemoryPool[ExternalSorter[6]].try_grow(256375168) returning Ok [Task 486] MemoryPool[ExternalSorter[6]].try_grow(256899456) returning...

### Describe the bug When running `CometPlanStabilitySuite` with `native_datafusion` scan, I see query failures such as: ``` - check simplified (tpcds-v1.4/q6) - native_datafusion *** FAILED *** (122 milliseconds) java.lang.IllegalArgumentException: requirement...

chore: Enable plan stability tests for `native_datafusion`

1

## Which issue does this PR close? Closes #. Blocked by https://github.com/apache/datafusion-comet/issues/2672 ## Rationale for this change Note that DPP is disabled for `native_datafusion` for now due to https://github.com/apache/datafusion-comet/issues/2672 ##...

feat: [CometFuzz] Add support for literal arguments [WIP]

2

## Which issue does this PR close? Part of https://github.com/apache/datafusion-comet/issues/2628 ## Rationale for this change Increase test coverage by picking both columns and literals for arguments to scalar functions. ```sql...