Andy Grove
Andy Grove
I have run into a deadlock when running TPC-DS benchmarks with this feature, so I am moving to draft while I investigate. It is possibly related to the memory pool...
After upmerging, I no longer see the deadlock, but instead get an error if I have insufficient memory allocated, which is an improvement. ``` org.apache.comet.CometNativeException (External error: Internal error: Partition...
I have now marked the feature as experimental and explained in the tuning guide that there is no spill to disk so this could result in OOM.
Fresh benchmarks after upmerging. 
TPC-DS excluding q97 (OOM with ShuffledHashJoin). 
I think having separate jars for linux vs Apple makes sense
sort of related .. I think it would be best to stop publishing the fuzz test and integration test jars
Closing as duplicate of https://github.com/apache/datafusion-comet/issues/947
> @andygrove can I get a review please? Thank you in advance. Thanks for the PR @JensonChoi. The other change that will need to be made is to update the...
> @andygrove Following our discussions on Discord, I ran the following commands to update the golden files. > > ``` > make clean; make release PROFILES="-Pspark-3.4" > SPARK_GENERATE_GOLDEN_FILES=1 ./mvnw -pl...