Andy Grove
Andy Grove
I have not been able to reproduce this issue yet. I am using the same Comet commit and so far have tested on a single node cluster with these configs:...
I do wonder if the issue is related to specifying `spark.sql.extensions=org.apache.comet.CometSparkSessionExtensions` (which was originally the documented way to enable Comet) rather than using `spark.plugins=org.apache.spark.CometPlugin` which would set the executor memory...
I could not reproduce this issue in k8s either. Here is the spark-submit command that I used. ``` $SPARK_HOME/bin/spark-submit \ --master $SPARK_MASTER \ --conf spark.driver.memory=8G \ --conf spark.executor.instances=4 \ --conf...
I now see that I missed `spark.comet.exec.shuffle.mode=jvm`. Retesting...
I can reproduce this now.
Comet 0.7.0 supported 3.5.0 through 3.5.4 but not 3.5.5 which had breaking changes to internal apis. I am on vacation this week but it looks like 3.5.5 support was added...
> Update: spun up another cluster on Spark 3.5.3 and used the same prebuilt Comet JAR 0.7.0 > > Was able to get past the initial error documented here but...
I am assuming that the GCP version of Spark has some differences in these internal APIs
This issue was likely resolved https://github.com/apache/datafusion-comet/pull/693 so will close for now. @mkgada feel free to reopen if this is still an issue
It seems like this is a known issue in Iceberg 1.6.1 and not necessarily a Comet issue, so I will close this