metorikku
metorikku copied to clipboard
Control writing specific output in yaml file
I have some jobs that need to output multiple dataframes. However, I also want to control output by using environment variables. Is there any option available for controlling writing dataframes?
Rigth now you could force using environment varaibles such as boolean to create empty dataframes. For example, you could query your dataframe as
SELECT *
FROM <your_data_frame>
WHERE ${create_dataframe} = 'true'
And in job_config.yaml you could create this var as
variables:
create_dataframe: true
Finally, on output options set protectFromEmptyOutput to true