SparkApplication manifest file sparkConf azure storage account access key usage

[GurayCetin]

We are using SparkApplication manifest file with sparkConf section but we are giving access key in hard coded. We are aware that this is not best practice and want to change these hardcoded values access-key-1 and access-key-2 to get them from mounted k8s secret value path or k8s secret values.

So how can i achieve that? What is the approach? I could not give correct syntax for mounted path on sparkConf or secret environment variable with ${variable}. I am running it on Azure AKS and tried to key-vault integration and directly from k8s secret.

     sparkConf:
            spark.databricks.delta.optimizeWrite.enabled: false
            spark.kubernetes.executor.volumes.persistentVolumeClaim.spark-local-dir-1.options.sizeLimit: "500Gi"
            spark.hadoop.fs.azure.account.key.storage-account-1.blob.core.windows.net: "access-key-1"
            spark.hadoop.fs.azure.account.key.storage-account-2.blob.core.windows.net: "access-key-2"