metorikku icon indicating copy to clipboard operation
metorikku copied to clipboard

Control writing specific output in yaml file

Open akizminet opened this issue 3 years ago • 1 comments

I have some jobs that need to output multiple dataframes. However, I also want to control output by using environment variables. Is there any option available for controlling writing dataframes?

akizminet avatar May 28 '21 16:05 akizminet

Rigth now you could force using environment varaibles such as boolean to create empty dataframes. For example, you could query your dataframe as

SELECT *
FROM <your_data_frame>
WHERE ${create_dataframe} = 'true'

And in job_config.yaml you could create this var as

variables:
  create_dataframe: true

Finally, on output options set protectFromEmptyOutput to true

lucabem avatar Aug 10 '22 19:08 lucabem