Chong Gao

[FEA] Support First() in windowing context with double type

2

**Is your feature request related to a problem? Please describe.** This is a feature request to support first() function in windowing context with double type. For example: ``` import org.apache.spark.sql.Row...

Use alluxio native API to mount instead of cmd

2

Currently, we use an Alluxio command line to get the mount table and mount an s3 bucket. This PR introduced the Alluxio client jar to do: - Get the mount...

Closes #6779 ### Root cause There are 2 ways to create `InMemoryFileIndex`: [First](https://github.com/NVIDIA/spark-rapids/blob/branch-22.12/sql-plugin/src/main/scala/com/nvidia/spark/rapids/AlluxioUtils.scala#L634ToL640) ``` val fi = new InMemoryFileIndex( relation.sparkSession, inputFiles, // This is actually sub files under the root...

[BUG] Always read old data from alluxio regardless of S3 changes when using CONVERT_TIME replacement algorithm

2

**Describe the bug** When using the CONVERT_TIME Alluxio replacement algorithm, auto mount method, and adding new files into s3, Spark rapids can't read the new files. The query path is...

[BUG] Move the Python tests with large_data_test mark in PR #8825 to scale test tool/repo

1

**Describe the bug** This is a follow on issue. - [ ] Move the test in this [PR](https://github.com/NVIDIA/spark-rapids/pull/8825) to scale test tool/repo Refer to the [comment](https://github.com/NVIDIA/spark-rapids/pull/8825#discussion_r1276402512) - [ ] Move...

[FEA] [Follow on] Follow on issues for the checking non UTC test cases.

1

**Describe the bug** After https://github.com/NVIDIA/spark-rapids/pull/9482 is merged, should further finish the following items: - [ ] Test permerge with a time zone Env variable, by default it's UTC. ```python import...

[FEA] Fix Databricks failed cases when Time zone is non-UTC

2

**Is your feature request related to a problem? Please describe.** Spark-Rapids currently does not support legacy rebase mode when TZ is non-UTC. For non-UTC time zone, DB failed some cases...

[FEA] [follow-up] Parse just time to timestamp

1

**Is your feature request related to a problem? Please describe.** This is from: https://github.com/NVIDIA/spark-rapids-jni/pull/1539#issuecomment-1894854381 Implements just time format: // has no yyyy-MM-dd parts: * `[h]h:[m]m:[s]s.[ms][ms][ms][us][us][us][zone_id]` * `T[h]h:[m]m:[s]s.[ms][ms][ms][us][us][us][zone_id]` [Spark link](https://github.com/apache/spark/blob/v3.5.0/sql/api/src/main/scala/org/apache/spark/sql/catalyst/util/SparkDateTimeUtils.scala#L552C1-L555): val...

[FEA] [follow-up] match the Spark exceptions when cast string to timestamp

1

**Is your feature request related to a problem? Please describe.** It's from: [link](https://github.com/NVIDIA/spark-rapids-jni/pull/1539#discussion_r1459121981) **Describe the solution you'd like** Different Sparks have differrent behavior: - java.time.DateTimeException: Cannot cast ABC to TimestampType....

[FEA] Try to find some useful Alluxio Metrics and document them

2

**Is your feature request related to a problem? Please describe.** This issue is from: https://github.com/NVIDIA/spark-rapids/pull/7172#discussion_r1036170523 Alluxio has hundreds of metrics, refer to https://docs.alluxio.io/os/user/stable/en/reference/Metrics-List.html. Some of them are useful for monitoring...