Arnav Balyan

[VL] Can't run spark.plugins=io.glutenproject.GlutenPlugin with release jar

4

### Backend VL (Velox) ### Bug description bin/spark-shell \ --conf spark.plugins=io.glutenproject.GlutenPlugin \ --conf spark.memory.offHeap.enabled=true \ --conf spark.memory.offHeap.size=20g \ --conf spark.shuffle.manager=org.apache.spark.shuffle.sort.ColumnarShuffleManager \ --jars /home/user/gluten-velox-bundle-spark3.3_2.12-1.1.0.jar Output: ``` 24/06/23 05:27:23 WARN NativeCodeLoader: Unable...

- When reading parquet, hive converts the TZ to local timezone while querying which causes result mismatch against paimon - Add support to do the same when querying through paimon...

[core] Don't allow full compaction with lookup changelog producer

1

- Currently there is no check against full compaction on lookup changelog producer. - User reported unexpected behaviour in #6183 - Disable setting both the properties together instead of the...

[flink] Fail fast validation for delete with write-only mode enabled

1

- DeleteAction now checks CoreOptions.writeOnly() at the start of run. - If 'write-only' = 'true', it throws a clear UnsupportedOperationException. - avoids deep FileStoreCommitImpl manifest conflict errors. - Added testDeleteActionWriteOnlyRejected...

[core] Standardize flink versions across the modules

1

- Users building Paimon with Flink 1.18.1 encountered "Implementing class" errors due to version mismatches between modules - This patch standardizes the same flink version across modules, with a common...

- Emit worker performance, statistics and throughput to keystone and other third party metric integrations

Create data placement module which support intelligent routing based on current data placement and worker performance

Disk backend was replaced by iouring, remove unwanted code

Integrations with third party caches, and add external cache support to the tiered cache

Support a simple test bench which can run basic benchmarks on a new environment and provide detailed performance statistics.