How to assign environment variable in aztk cluster

[deepika087]

Hi I have a specific use case to assign environment variable within the aztk cluster is there any way to assign env variable ?

[jafreck]

You can create a custom script that sets an environment variable and add it to cluster.yaml configuration file. An aztk custom script is a bash script that is executed in the docker container that the spark environment is run in.

In your cluster.yaml:

custom_scripts: 
   - script: /path/to/your/script
     runOn: <master/worker/all-nodes>

[timotheeguerin]

thats something though we could simplify potentially too

[jafreck]

@timotheeguerin yeah, we could add a environment_variables block to cluster.yaml and job.yaml and then handle setting ourselves