Andy Grove

> Thank you for your reply. I will try when these features are merged. When I follow https://datafusion.apache.org/comet/contributor-guide/benchmarking.html the benchmark guide and set the same configuration, I still discover many...

Hi @DamonZhao-sfu. For query 72, are you enabling CBO in Spark or using any form of join reordering or are you using the official version of the query that joins...

@DamonZhao-sfu could you also provide the configs you used for the Spark run? I am seeing most queries running faster with Comet (but at 100GB) and would like to try...

Thanks @DamonZhao-sfu. We just updated our [benchmarking guide](https://datafusion.apache.org/comet/contributor-guide/benchmarking.html) with the currently recommended configs for the latest Comet code. Could you build with latest code and try with these settings? Here...

@DamonZhao-sfu We just released Comet 0.2.0 which provides some performance improvements for TPC-DS. We are now starting to test with 1TB data set as well. https://datafusion.apache.org/blog/2024/08/28/datafusion-comet-0.2.0/

This issue is from before the 0.1.0 release and there have been some improvements to TPC-DS since then. We have an epic for improving TPC-DS so I think we can...

The OOM is happening in native code in the Comet shuffle write processor

Closing this issue since it is vague and cannot reproduce. Native shuffle was re-implemented to be more memory efficient since this issue was filed.

> @andygrove @vaibhawvipul Could you please take a look? Apologies @sujithjay I had missed this ping. I will review this early next week.

This is looking good @sujithjay. CI is failing due to clippy warnings. If you run clippy locally you should be able to see the same warnings as well as suggestions...