spark [SPARK-54605][K8S] Override default spark env in driver pod

What changes were proposed in this pull request?

Override driver environment variables by appending custom driver env to the end of spark-env.sh in driver pod.

Why are the changes needed?

Within the Spark driver pod, the spark-submit script execution process sources the spark-env.sh configuration file. Environment variables defined by the user via spark.kubernetes.driverEnv.* configuration properties have the potential to be overridden by default variable definitions present in the spark-env.sh file.

Does this PR introduce any user-facing change?

No.

How was this patch tested?

UT.

Was this patch authored or co-authored using generative AI tooling?

No.

Dec 05 '25 07:12 littlexyw

@dongjoon-hyun Could you please review this PR when you have time? Many Thanks.

Dec 08 '25 06:12 littlexyw

BTW, the title looks misleading because the code is overriding instead of appending.

Dec 08 '25 18:12 dongjoon-hyun

Thank you for pinging me, @littlexyw .

This needs to proceed with Apache Spark 4.2.0 as an improvement because Apache Spark provides no spark-env.sh file by default. Could you adjust the PR once more?

Done.

Dec 10 '25 02:12 littlexyw