Hongze Zhang

Thank you for reporting. IIUC The operator itself doesn't seem to be the one that slows down your query? Say the slowest tasks just took 438ms and 166ms. Would you...

@Surbhi-Vijay Was there a large difference on shuffle write size?

> Do we have a solution for this? Does #6670 solve this issue? I think so. Would you like to help have a try? If it works then we can...

If `Gluten - Filter applied on merged Parquet schema with new column should work` is removed, can we directly use vanilla Spark's `Filter applied on merged Parquet schema with new...

Besides, as a possible temporary solution, would you please try with [this method](https://github.com/apache/incubator-gluten/tree/main/tools/gluten-te/ubuntu/examples/buildhere-veloxbe-portable-libs) to create a build in docker container? It should be a more general way for building.

> When `spark.gluten.memory.dynamic.offHeap.sizing.enabled=true` it will not consider the configured off-heap size https://github.com/apache/incubator-gluten/blob/main/gluten-core/src/main/scala/org/apache/gluten/GlutenPlugin.scala#L167 Perhaps there was a tiny typo leading to a bug when PR #5439 was iterated? cc @supermem613 Would...

> > > When `spark.gluten.memory.dynamic.offHeap.sizing.enabled=true` it will not consider the configured off-heap size https://github.com/apache/incubator-gluten/blob/main/gluten-core/src/main/scala/org/apache/gluten/GlutenPlugin.scala#L167 > > > > > > Perhaps there was a tiny typo leading to a bug...

Why are you using the option `spark.gluten.memory.dynamic.offHeap.sizing.enabled`? @wenwj0

> `GlutenConfig.getConf().onHeapMemorySize()` was changed to `GlutenConfig.getConf().offHeapMemorySize()`, which doesn't look like intentional to me, but I am not sure. After reading more code I guess this was intentional... The code tended...

> onheap+offheap memory It sounds OK to me to change current approach to on-heap + off-heap, which may simplify the relevant configuration setting code by the way. cc @supermem613 @zhli1142015