Variable support in jobs

[rclough]

By no means necessary, but another nice-to-have but not critical is having a job able to have a set of variables for execution that can be passed into the tasks as "arguments".

Use case:

• You have some process that does a bunch of processing on similar sales data
• Shop 1's files come in the morning
• Shop 2's files come in the evening
• Functionally, the tasks would be identical, except all files/tables are prefixed with Shop1* or Shop2*
• To do this curently, you would have 2 identical jobs, one with commands that pass in Shop1 as a parameter, the other Shop2. Dumbed down example:
  • process.sh Shop1.csv
  • mysql -e "set @shop_table=Shop1;source ./sql/update_shop_data.sql"
  • where the SQL file has something like:
  • LOAD DATA INFILE '@shop.csv' INTO TABLE @shop;
• If you could pass the Shop1/Shop2 prefix to the job as an argument, it would eliminate duplicate jobs. This would however complicate scheduling because you would need to support scheduling the same job with different arguments at different times

Another use case might be where you want to prevent "magic numbers" and/or have jobs that you might need to change over time, so you can easily set truncation numbers, maybe number of records you grab, etc.

[thieman]

Any work in this direction would require a slew of backend changes to support running multiple versions of the same Job at once. I think there may be some uses for this functionality, but we would need simultaneous execution before we can consider it.

[rclough]

I see. Its not super important, just an idea.