Load the `spark.delight.accessToken.secret` using a K8s secret

[swill]

Is your feature request related to a problem? Please describe. Based on the documentation for integrating with the spark operator, the secret access token is specified in plain text within the body of the job manifest.

Describe the solution you'd like I would like it to be possible to specify the secret access token spark.delight.accessToken.secret via a Kubernetes secret. This is important because the spark application manifests are often tracked in git and shared between different departments / user groups. The current method for providing this configuration does not allow for the secrets to be scoped to the Kubernetes run context, as the access token is tracked via the manifest itself and does not reference a K8s secret to pull the secret information.

[jystephan]

Thank you Will. Indeed right now we're pulling the token from the spark conf and it's not easy to pass this using a Kubernetes secret. One simple solution would be to pull the token from the spark conf, and if it's missing lookup an environment variable like DELIGHT_ACCESS_TOKEN. This way you could mount define your token as a kubernetes secret, and then mount it as an environment variable in the driver pod.