Nomad CLI Monitor Dispatched Jobs

[evandam]

Proposal

Since the Nomad CLI supports monitoring deployments with nomad job run, it would be great to see similar behavior with nomad job dispatch.

Ideally it would dispatch the job and poll the dispatched job while it is running, and then exit 0/non-zero based on the exit status of the job (maybe some consideration with idempotency tokens needed).

Use-cases

Use case is mostly around a CI environment where we use dispatched jobs to prepare an environment, seed a database, etc.

Being able to shell out to the Nomad CLI rather than custom scripting with the Nomad API would be great.

Attempted Solutions

Current solution is pretty clunky, unless there's a better way to do it.

• Dispatch a parameterized job
• Get the dispatched job ID from the previous command
• Poll the Nomad API until the dispatched job is dead
• Check the latest alloc of the dispatched job to see if it completed successfully or failed.
  • /v1/job/<dispatched_job_id> will only say "dead" without indication of success/failure. Gets a bit tricky when the dispatched job has rescheduling, lost nodes, etc. so logic gets a little dicey.

[tgross]

Thanks for opening this issue @evandam! That makes sense to me, so we'll look into getting it on the roadmap.

[lgfa29]

#3753 also mentions tailing log output, which could be a useful feature as well.

[lgfa29]

#16898 also describes a use case for batch jobs with nomad job run command.