Add ability to run "Free-Style" jobs in Optimus

[sbchaos]

Is your feature request related to a problem? Please describe. Currently Optimus only provides a way to use a specific image in a job through the plugin, we require a way for people to run a free-style job where they can bring in their own image to be scheduled by the Optimus. This might address use-cases where creating a specific plugin might be an overkill or might not be feasible for Optimus team.

A typical use case might be to fetch logs after a job is run or run some other ad-hoc task, the new job can define a dependency on the job and fetch the logs after the previous job is complete.

Describe the solution you'd like Providing a plugin to create a free-style job by specifying an image for the job to execute. This can have configuration like

version: 1
name: cleanup
owner: [email protected]
schedule:
  start_date: "2022-02-26"
behavior:
  depends_on_past: false
  catch_up: false
task:
  name: generic
  config:
    IMAGE: docker.io/my-log-extractor:2.3
    JOB_LABELS: owner=optimus
    PROJECT: integration
    LOCATION: /var/logs/my-system/job.log
    USER_NAME: "{{.secret.user}}"
    USER_TOKEN: "{{.secret.token}}"
    FORMAT: JSON
dependencies:
  - load_to_bigquery

This will run the image as non-root user, with the required config available as environment variables and assets made available.

Describe alternatives you've considered Alternative might be to have a hook to external system to run these actions on completion of task. This might require people to manage their post jobs somewhere else.