[feature] Defining and consuming custom resources

[sheerun]

This is one possible solution for #406. The idea is to implement custom resources similarly to mesos:

https://mesos.apache.org/documentation/attributes-resources/

When starting a client, specify resources it exposes

client {
  enabled = true

  resource "gpu" {
    type = "range"
    begin = 1
    end = 4
  }

  resource "ip" {
    type = "ip-range"
    begin = "192.168.1.0"
    end = "192.168.1.255"
  }

  resource "github_token" {
    type = "set"
    items = ["1234", "abcd"]
  }

  resource "network" {
    type = "enum"
    items {
      private = "eth2"
      overlay = "overlay1"
    }
  }
}

Allow to specify used resources in task specification

task {
  resources {
    gpu = 2
    ip = 1
    github_token = 1
    network = "private"
  }
}

Expose used resource to tasks (and custom drivers) via env variables

NOMAD_RESOURCE_gpu = '1,2'
NOMAD_RESOURCE_ip = '192.168.1.5'
NOMAD_RESOURCE_github_token = '1234'
NOMAD_RESOURCE_network = 'eth2'

[tolmanam]

This seems like a pretty useful feature. Is it on the roadmap or did something better come along?

[jbott]

Are there any plans to implement this or something similar? I have a use case where I am connecting hardware resources to individual nodes, and I would need to expose that info at the scheduler level to correctly place tasks.

[schmichael]

Device Plugins are planned but still in an internal design phase, so I don't want to make any promises regarding the timeline. Posting detailed use cases and implementation ideas here is definitely welcome and will be taken into account!

[andrewchambers]

I have a use case for jobs that involves mounting cloud provisioned disks, however each google compute node only supports 4 disks, so there is a resource limit of 4 per node.

[tgross]

Doing a bit of issue cleanup here. Since we last checked in on this issue, Nomad has shipped the devices, task driver, and storage plugin interfaces. I'm going to close this as resolved, and if we have interest for new kinds of plugins we can discuss that in a new issue.

[schmichael]

Let's keep this open as generic resources have yet to be implemented, but they're still something we're considering. While we have pretty good plugin coverage these days as @tgross noted, we lack the declarative custom resource definition approach outlined in the original issue. In the past people would even hijack the mostly-meaningless mbits resource to use as a custom resource, but now we've deprecated that as well!

If devices supported multi-tenancy they would be a solution to custom resources, albeit a lot more effort than the declarative format proposed above. However as of 1.0, device attributes are only used for constraints and a single device can only be used by a single allocation. This means you can't create custom devices just to have custom resources.

[andrewchambers]

Definitely not resolved, I feel like nomad has implemented everything BUT a simple way to define custom resources. It kind of feels like some of these other features are needlessly specific and complex compared to simple custom resources.

[mgdunn2]

We would love the ability to constrain allocations by available vram using the nvidia/cuda plugin with multi-tenancy but the proposal above would allow us to schedule around a generic resource that emulates that behavior.

[lattwood]

@schmichael what's the possibility of committing to not removing mbits from Nomad until there's generic resource support?

[benbuzbee]

We have 25 Gbps NICs and jobs that want 10 Gbps of it - we need nomad to know it can't schedule more than 3 of those jobs on a single machine.

[edigaryev]

However as of 1.0, device attributes are only used for constraints and a single device can only be used by a single allocation. This means you can't create custom devices just to have custom resources.

Assuming that:

• a single allocation consumes the device for the allocation's runtime duration
• a custom device plugin can programatically create/fake an arbitrary number of virtual non-existent devices

Isn't that the same as having custom integer-based resources? In a hacky way, of course.

[edigaryev]

Isn't that the same as having custom integer-based resources? In a hacky way, of course.

After quick searching through GitHub, found the nomad-generic-device-plugin.

It seems to achieve exactly that, without an automatic way to generate N devices in a specific (vendor, type, model) tuple, though.