[Bug] Modifying kyuubi-defaults.conf does not take effect when starting Spark for the second time

[A-little-bit-of-data]

Code of Conduct

• [X] I agree to follow this project's Code of Conduct

Search before asking

• [X] I have searched in the issues and found no similar issues.

Describe the bug

When kyuubi-defaults.conf is modified and spark is started again, the new configuration is not loaded

When kyuubi-defaults.conf is configured as follows，The spark task started on k8s still reads the old configuration.

trino.spark.executor.instances=2 trino.spark.driver.cores=1 trino.spark.executor.cores=2 trino.spark.kubernetes.driver.limit.cores=2 trino.spark.kubernetes.executor.limit.cores=3 trino.spark.driver.memory=1g trino.spark.executor.memory=2g

trino.spark.executor.instances=2 trino.spark.driver.cores=1 trino.spark.executor.cores=2 trino.spark.kubernetes.driver.limit.cores=2 trino.spark.kubernetes.executor.limit.cores=4 trino.spark.driver.memory=1g trino.spark.executor.memory=4g

The spark.kyuubi.kubernetes.spark.cleanupTerminatedDriverPod.kind=ALL parameter is configured, but after completion, the spark driver pod still exists and the status is completed. It needs to be manually deleted.

Affects Version(s)

1.9.1

Kyuubi Server Log Output

/opt/kyuubi/externals/spark-3.5.1-bin-hadoop3/bin/spark-submit \
 	--class org.apache.kyuubi.engine.spark.SparkSQLEngine \
 	--conf spark.hive.server2.thrift.resultset.default.fetch.size=1000 \
 	--conf spark.kyuubi.client.version=1.9.1 \
 	--conf spark.kyuubi.engine.engineLog.path=/opt/kyuubi/work/trino/kyuubi-spark-sql-engine.log.2 \
 	--conf spark.kyuubi.engine.share.level=USER \
 	--conf spark.kyuubi.engine.submit.time=1721112987920 \
 	--conf spark.kyuubi.engine.type=SPARK_SQL \
 	--conf spark.kyuubi.frontend.connection.url.use.hostname=false \
 	--conf spark.kyuubi.frontend.protocols=REST,THRIFT_BINARY \
 	--conf spark.kyuubi.ha.engine.ref.id=9653cc02-df96-4691-82b9-3452c5c2cdc4 \
 	--conf spark.kyuubi.ha.namespace=/kyuubi_1.9.1_USER_SPARK_SQL/trino/default \
 	--conf spark.kyuubi.ha.zookeeper.session.timeout=600000 \
 	--conf spark.kyuubi.kubernetes.spark.cleanupTerminatedDriverPod.kind=ALL \
 	--conf spark.kyuubi.kubernetes.terminatedApplicationRetainPeriod=PT1M \
 	--conf spark.kyuubi.metrics.enabled=true \
 	--conf spark.kyuubi.metrics.reporters= \
 	--conf spark.kyuubi.session.engine.check.interval=PT1M \
 	--conf spark.kyuubi.session.engine.idle.timeout=PT2M \
 	--conf spark.kyuubi.session.real.user=trino \
 	--conf spark.app.name=trino \
 	--conf spark.driver.cores=1 \
 	--conf spark.driver.memory=1g \
 	--conf spark.executor.cores=2 \
 	--conf spark.executor.instances=2 \
 	--conf spark.executor.memory=2g \
 	--conf spark.kubernetes.driver.label.kyuubi-unique-tag=9653cc02-df96-4691-82b9-3452c5c2cdc4 \
 	--conf spark.kubernetes.driver.limit.cores=2 \
 	--conf spark.kubernetes.driver.pod.name=kyuubi-trino-9653cc02-df96-4691-82b9-3452c5c2cdc4-driver \
 	--conf spark.kubernetes.executor.limit.cores=3 \
 	--conf spark.kubernetes.executor.podNamePrefix=kyuubi-trino-9653cc02-df96-4691-82b9-3452c5c2cdc4 \
 	--conf spark.user=trino \
 	--conf spark.kubernetes.driverEnv.SPARK_USER_NAME=trino \
 	--conf spark.executorEnv.SPARK_USER_NAME=trino \
 	--proxy-user trino /opt/kyuubi/externals/engines/spark/kyuubi-spark-sql-engine_2.12-1.9.1.jar




 /opt/kyuubi/externals/spark-3.5.1-bin-hadoop3/bin/spark-submit \
 	--class org.apache.kyuubi.engine.spark.SparkSQLEngine \
 	--conf spark.hive.server2.thrift.resultset.default.fetch.size=1000 \
 	--conf spark.kyuubi.client.version=1.9.1 \
 	--conf spark.kyuubi.engine.engineLog.path=/opt/kyuubi/work/trino/kyuubi-spark-sql-engine.log.3 \
 	--conf spark.kyuubi.engine.share.level=USER \
 	--conf spark.kyuubi.engine.submit.time=1721114348590 \
 	--conf spark.kyuubi.engine.type=SPARK_SQL \
 	--conf spark.kyuubi.frontend.connection.url.use.hostname=false \
 	--conf spark.kyuubi.frontend.protocols=REST,THRIFT_BINARY \
 	--conf spark.kyuubi.ha.engine.ref.id=f93fdc9d-402b-4d9c-ba6e-7f01a4451f19 \
 	--conf spark.kyuubi.ha.namespace=/kyuubi_1.9.1_USER_SPARK_SQL/trino/default \
 	--conf spark.kyuubi.ha.zookeeper.session.timeout=600000 \
 	--conf spark.kyuubi.kubernetes.spark.cleanupTerminatedDriverPod.kind=ALL \
 	--conf spark.kyuubi.kubernetes.terminatedApplicationRetainPeriod=PT1M \
 	--conf spark.kyuubi.metrics.enabled=true \
 	--conf spark.kyuubi.metrics.reporters= \
 	--conf spark.kyuubi.session.engine.check.interval=PT1M \
 	--conf spark.kyuubi.session.engine.idle.timeout=PT2M \
 	--conf spark.kyuubi.session.real.user=trino \
 	--conf spark.app.name=trino \
 	--conf spark.driver.cores=1 \
 	--conf spark.driver.memory=1g \
 	--conf spark.executor.cores=2 \
 	--conf spark.executor.instances=2 \
 	--conf spark.executor.memory=2g \
 	--conf spark.kubernetes.driver.label.kyuubi-unique-tag=f93fdc9d-402b-4d9c-ba6e-7f01a4451f19 \
 	--conf spark.kubernetes.driver.limit.cores=2 \
 	--conf spark.kubernetes.driver.pod.name=kyuubi-trino-f93fdc9d-402b-4d9c-ba6e-7f01a4451f19-driver \
 	--conf spark.kubernetes.executor.limit.cores=3 \
 	--conf spark.kubernetes.executor.podNamePrefix=kyuubi-trino-f93fdc9d-402b-4d9c-ba6e-7f01a4451f19 \
 	--conf spark.user=trino \
 	--conf spark.kubernetes.driverEnv.SPARK_USER_NAME=trino \
 	--conf spark.executorEnv.SPARK_USER_NAME=trino \
 	--proxy-user trino /opt/kyuubi/externals/engines/spark/kyuubi-spark-sql-engine_2.12-1.9.1.jar

Kyuubi Engine Log Output

No response

Kyuubi Server Configurations

No response

Kyuubi Engine Configurations

No response

Additional context

No response

Are you willing to submit PR?

• [ ] Yes. I would be willing to submit a PR with guidance from the Kyuubi community to fix.
• [X] No. I cannot submit a PR at this time.