Scheduling job with a dependency on another job

[jason-riddle]

trafficstars

Is it possible to schedule a job only after another job is successful scheduled? A use case for example, would be to use one docker container to provide a REST API (https://github.com/begriffs/postgrest) for another docker container (postgres), but the REST API docker container should only come up once the database container is up and passed it's health check.

[dadgar]

Hey! Nomad currently does not have a way to express job dependencies but it is something on our roadmap. From your description it sounds like you need dependencies between tasks/task groups not jobs. A job would only be finished once all the task groups are dead. If you look at issue #419 I think it captures this issue, so I am going to mark as closed.

[zapman449]

I believe this should be re-opened.

Here's a use-case: I want to run hashicorp vault and docker-registry from within nomad. The other containers will need these to be available before they can start, but #419 argues for pushing all my apps into a single job.

Granted, this is less obvious for vault, but our private registry is foundational for everything.

[suslovsergey]

Any news?

[aaronhurt]

Similar use case here. We would like to be sure that things like our OAuth service, auditing service, etc.. are started and listed as dependencies before other backend applications start.

[DhashS]

Different use case here!

I'd like to have some control around precendence order in which my networking, services, and templates fire off. The suggestion of job-dependencies is limiting, unless there's some funky n-pass evaluation stuff, and execution in the same container.

My use case provisions container environment variables and config files based on the services running and registered with consul. Without a strict happens-before ordering, i get a non-deterministic template.

[scalp42]

See #419 for the same kind of issue.

[pznamensky]

It's quite obvious feature for the orchestrator, and at the moment, it's one of the most desired functionality for us. I still hope it would be implemented in Nomad.

[116davinder]

I believe, dependencies can be defined like this in nomad job spec.

# job spec example
dependency 1 {
  provider = consul/nomad/etc.
  check = wait for consul/nomad service / nomad-status to become health
    pattern = nomad/job/group/task [ if nomad service, here i want to see, if nomad can check if nomad service is up or not]
    pattern = name-of-consul-service [ if consul service, here i want to see, if nomad can check if consul service is up or not ]
    pattern = nomad/job/group/task-name [ if nomad-status, here, I want to see, what nomad know about the task, whether its up or not]

  interval = 10s
  retry = 100
  delay = 15s
}
dependency 2 {
  provider = consul/nomad/etc.
  check = wait for consul/nomad service / nomad-status to become health
    pattern = nomad/job/group/task [ if nomad service, here i want to see, if nomad can check if nomad service is up or not]
    pattern = name-of-consul-service [ if consul service, here i want to see, if nomad can check if consul service is up or not ]
    pattern = nomad/job/group/task-name [ if nomad-status, here, I want to see, what nomad know about the task/group/job, whether its up or not]

  interval = 10s
  retry = 100
  delay = 15s
}

[richard-tns]

I believe, dependencies can be defined like this in nomad job spec.

# job spec example
dependency 1 {
  provider = consul/nomad/etc.
  check = wait for consul/nomad service / nomad-status to become health
    pattern = nomad/job/group/task [ if nomad service, here i want to see, if nomad can check if nomad service is up or not]
    pattern = name-of-consul-service [ if consul service, here i want to see, if nomad can check if consul service is up or not ]
    pattern = nomad/job/group/task-name [ if nomad-status, here, I want to see, what nomad know about the task, whether its up or not]

  interval = 10s
  retry = 100
  delay = 15s
}
dependency 2 {
  provider = consul/nomad/etc.
  check = wait for consul/nomad service / nomad-status to become health
    pattern = nomad/job/group/task [ if nomad service, here i want to see, if nomad can check if nomad service is up or not]
    pattern = name-of-consul-service [ if consul service, here i want to see, if nomad can check if consul service is up or not ]
    pattern = nomad/job/group/task-name [ if nomad-status, here, I want to see, what nomad know about the task/group/job, whether its up or not]

  interval = 10s
  retry = 100
  delay = 15s
}

This is incorrect and is just the proposal in https://github.com/hashicorp/nomad/issues/19291

There is still no method to schedule dependencies between jobs

[DhashS]

I ended up using terraform for this. nomad_job and consul_health_check’s work together as a pair to make a job node that can be depended on for separate dependency nodes

[Material-Scientist]

Still nothing?

[HeinXavierSwart]

This issue has been open for nearly 10 years.

[jrasell]

Hi @Material-Scientist and @HeinXavierSwart, unfortunately you're both right in that this has not been assigned to an engineer for research. When this happens, the issue will become assigned and an update posted here but I cannot provide any timeline on that currently.

[idenkov]

This issue has been open for nearly 10 years.

I do not think anything will happen on that front, especially after IBM trying to acquire them, but even before that seems to me they have shifted their focus on other areas and not in the development of software.

So unless someone implement that feature and makes PR, we can't and shouldn't expect Hashicorp to implement it. But given what they were doing past 1-2 years best course of action will be to do what everyone else is doing - move away from their products.

[jrasell]

Hi @idenkov; please keep comments related to the issue as described with details of expected user experience that might help any future development effort.

shifted their focus on other areas and not in the development of software

I'd invite you to take a look at the Nomad changelog, commit history, or similar to see that this is incorrect.

[Juanadelacuesta]

Linking this issue here as a duplicate.

[Material-Scientist]

There is a hacky way to do it with templates:

template {
        destination = "local/test"
        #error_on_missing_key = true
        data = <<EOH
POSTGRES_ADDR="{{ range service "postgres" }}{{ .Address }}{{ end }}"
{{- $upstream_services := service "postgres" -}}
    
{{- if $upstream_services -}}
  {{- range $upstream_services }}
     server {{ .Address }}:{{ .Port }};
  {{- end }}
{{- else }}
  {{ key "fakeKey" }}
{{- end }}
EOH        
      }

E.g. this job won't start unless the postgres service is running.

"postgress" (misspelled on purpose) results in "missing: kv.block(fakeKey)":

But when spelling it correctly, and postgres is running, the task is started:

[maxadamo]

I believe, dependencies can be defined like this in nomad job spec.

job spec example

dependency 1 { provider = consul/nomad/etc. check = wait for consul/nomad service / nomad-status to become health pattern = nomad/job/group/task [ if nomad service, here i want to see, if nomad can check if nomad service is up or not] pattern = name-of-consul-service [ if consul service, here i want to see, if nomad can check if consul service is up or not ] pattern = nomad/job/group/task-name [ if nomad-status, here, I want to see, what nomad know about the task, whether its up or not]

interval = 10s retry = 100 delay = 15s } dependency 2 { provider = consul/nomad/etc. check = wait for consul/nomad service / nomad-status to become health pattern = nomad/job/group/task [ if nomad service, here i want to see, if nomad can check if nomad service is up or not] pattern = name-of-consul-service [ if consul service, here i want to see, if nomad can check if consul service is up or not ] pattern = nomad/job/group/task-name [ if nomad-status, here, I want to see, what nomad know about the task/group/job, whether its up or not]

interval = 10s retry = 100 delay = 15s }

This is incorrect and is just the proposal in #19291

There is still no method to schedule dependencies between jobs

I think it's not incorrect. It's a proposal, and as such "There is still no method to schedule dependencies between jobs" because it's proposed but not implemented. So, yapp, this way of structuring the code work if there is something underneath to handle this hypothetical block called dependency.

[maxadamo]

Hi @idenkov; please keep comments related to the issue as described with details of expected user experience that might help any future development effort.

shifted their focus on other areas and not in the development of software

I'd invite you to take a look at the Nomad changelog, commit history, or similar to see that this is incorrect.

@jrasell I can agree, but this is also a forever-standing ~issue~ request, which was raised 10 years ago. So, on one side I'd hope that Hashicorp embraces a GPL or AGPL license, but I know it is really off-topic here. On the other side, after 10 long years, a healthy amount of skepticism on this request should be allowed 😉