Tengfei Huang

[SPARK-39853][CORE] Support stage level task resource profile for standalone cluster when dynamic allocation disabled

14

### What changes were proposed in this pull request? Currently stage level scheduling works for yarn/k8s/standalone cluster when dynamic allocation is enabled, and spark app will acquire executors with different...

[GLUTEN-4039][VL] Add array insert function for spark 3.4+

3

## What changes were proposed in this pull request? Add support of `array_insert` function for velox backend. Velox side changes: https://github.com/facebookincubator/velox/pull/9851 ## How was this patch tested? New UT added.

[GLUTEN-7532][CORE] Rewrite TakeOrderedAndProject as actually executed operators when do offload

9

## What changes were proposed in this pull request? Rewrite the `TakeOrderedAndProjectExecTransformer` to the operators which are actually executed when do offload. Currently, the logic is wrapped in the `RDD`...

[GLUTEN-7079][VL] Fix metrics for InputIteratorTransformer of broadcast exchange

8

## What changes were proposed in this pull request? Fix issue https://github.com/apache/incubator-gluten/issues/7079 collecting metrics for `InputIteratorTransformer` of broadcast exchange. ## How was this patch tested? UT added.

### Description Currently when compute `TakeOrderedAndProjectExecTransformer`, the logic within the RDD computing includes rewriting/building query plans and generate new RDDs. This is not a common practice processing sql operators. And...

[SPARK-52395][CORE] Fast fail when shuffle fetch failure happens

2

### What changes were proposed in this pull request? Currently for shuffle reading, [ShuffleBlockFetcherIterator](https://github.com/apache/spark/blob/efd81c5a46ce8e3dbddb287b18ebf527412b6261/core/src/main/scala/org/apache/spark/storage/ShuffleBlockFetcherIterator.scala#L702) will fetch local and host local blocks in task thread and send remote block fetch request...

[SPARK-54556][CORE] Rollback succeeding shuffle map stages when shuffle checksum mismatch detected

1

### What changes were proposed in this pull request? Rollback shuffle map stages when shuffle checksum mismatch detected: - cancel and resubmit the stage if it's running; - clean up...