driver podTemplateFile not working

[PrathamCG]

"spark.kubernetes.driver.podTemplateFile" : "s3a://my-bucket/driver_template.yaml" "spark.kubernetes.executor.podTemplateFile" : "s3a://my-bucket/executor_template.yaml"

I am setting this in "sparkConf" but its not able to get file . In the driver template i am running node on-demand and in executor template I am running on-spot.

============================================================= driver_template.yaml:-

apiVersion: v1 kind: Pod metadata: labels: spark-role: driver namespace: dev spec: serviceAccountName: spark tolerations:

• effect: NoSchedule operator: Exists

============================================================== executor_template.yaml:-

apiVersion: v1 kind: Pod metadata: labels: spark-role: executer namespace: dev spec: serviceAccountName: spark affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: 'eks.amazonaws.com/capacityType' operator: In values: - SPOT

my main concern is I wanted to run driver on-demand and executors on-spot

[bscaleb]

Did you ever find a solution to this? I'm having the same problem.

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[github-actions[bot]]

This issue has been automatically closed because it has not had recent activity. Please comment "/reopen" to reopen it.