dispatch: optional parameters with default values and description for labels

[shantanugadgil]

Proposal

For parameterized jobs we can only specify the parameter name for the required or optional parameters.

How about if we could specify more information about the parameters, like:

• a default value for optional parameters
• a description field, which would be used in the Nomad GUI (in the Dispatch section)
• possibly a label field which could be used in place of the default label.
• this thought process could be enhanced further to specify the data type like string, number, etc.

Use-cases

When an operator is using the Nomad GUI, I would like to have the "default" value of the optional parameter populated in the GUI. Also, on a mouseover on the paramter, I would like to see the description which would help the user to fill in correct values.

Attempted Solutions

N/A

*** I understand that this would essentially turn the parameter list from a "list-of-strings" to a "list-of-maps", which might be considered a big change.

[tgross]

Hi @shantanugadgil! This idea sounds interesting, can we dig into this a little further? Right now the parameterized block looks like this:

parameterized {
    payload       = "required" # can also be "optional" or "forbidden"
    meta_required = ["first"]
    meta_optional = ["second", "third"]
  }

And then when we dispatch we do so with something like: nomad job dispatch -meta first=1 second=2 third=3 example $payload_source. So are you suggesting that we expand the meta_required and meta_optional fields here? Maybe we could merge the two blocks into a single meta block that has subblocks. The names here are terrible but something like:

parameterized {
    payload       = "required" # can also be "optional" or "forbidden"

    metadata {

      meta "first" {
         required = true
         description = "the first thing"
      }

      meta "second" {
         required = false
         description = "the optional second thing"
         default = "example"
      }

    }
}

I'm not sure what you mean by the label field though. How would that work and what would it change to have it set?

Having non-string types for metadata might be possible but it gets hairy but then we need to store any types rather than the existing meta, and that'd have downstream effects when we start interpolating the metadata as meta blocks. Might be doable but something we'd need to discuss.

[shantanugadgil]

This is spot on. :+1:

The change in structure is exactly what I had in mind, though I have never understood why it was "meta_" and not "param_" (# nit-picking)

parameterized {
  payload = "required" # can also be "optional" or "forbidden"

  parameters {           # <----  seems more natural to me, but I don't really care what it's called
    meta "first" {
      required    = true
      description = "the first thing"
      label       = "First Parameter"
      type        = "string"    # <---  wishlist, not super critical
    }

    meta "second" {
      required    = false
      description = "the optional second thing"
      default     = "example"
      label       = "Second Parameter"
      type        = "number"    # <---  wishlist, not super critical
    }
  }
}

I'm not sure what you mean by the label field though. How would that work and what would it change to have it set?

The label is mostly a GUI thing which could be shown in place of Start * as shown in the below screenshot.

About the non-string data types ... I think I was getting carried away re-imagining the Dispatch UI in place of the Submit interface of a CI/CD tool like Jenkins. This is not essential to my use-case.

Wish: I was also wondering if we could configure in the job definition, to make the input editbox be shown with a spinner control for "number" type of parameters (only allow increment/decrement)

There are more things I could suggest from a UI perspective, but I think it would be good to limit the current scope only to the basic requirements of default . :smile:

[tgross]

though I have never understood why it was "meta_" and not "param_"

Probably because it literally gets added to the job's meta and not passed as parameters to the task. It's up to the task to interpolate that however it sees fit (or not!).

The label is mostly a GUI thing which could be shown in place of Start * as shown in the below screenshot.

Oh nice! I don't use the web UI enough, do I? :grinning:

Ok, I'm going to mark this for roadmapping. If folks have other use cases, we'd love to hear about them!

[shantanugadgil]

Probably because it literally gets added to the job's meta and not passed as parameters to the task. It's up to the task to interpolate that however it sees fit (or not!).

OK.

follow up thought: Could the required parameters also have a "default" value?

[jmnavarrol]

...and, since we are at it, how about the ability of validate parameters out of a list?

I have batch jobs that would heavily benefit from parameters' values selected from a drop-down list on the webui (and validated to belong to the list when job launched from command line)

[next-jesusmanuelnavarro]

Also, as a workaround for the description part, the ui block can be used now, i.e.:

	ui {
		description = <<-EOD
		**THIS IS A PARAMETERIZED DISPATCH**
		**Parameters:**
		* **FIRST_PARAMETER:** (mandatory) This sets FOO.
		* **OPTIONAL_PARAMETER:** (optional) Sets BAR to do something.

		More info about this job at [SOME URL](http://some.url/mydistpatch.html).
		EOD
	}

[jmnavarrol]

Hi @shantanugadgil! This idea sounds interesting, can we dig into this a little further? Having non-string types for metadata might be possible but it gets hairy but then we need to store any types rather than the existing meta, and that'd have downstream effects when we start interpolating the metadata as meta blocks. Might be doable but something we'd need to discuss.

It might look interesting, but let's not forget that params map to environment variables into the running container which are always to be strings with any further validation needed to be done within the running environment itself. So, either you control both the running environment and the jobspec, in which case this double checking may be redundant, or you only control one of them, in which case whatever the runtime does, it's the "source of authority". I mean, this is not a case where you can make use of all the validation capabilities that may HCL offer but a very limited subset of them. Is it worth it, or would it be more confusing? (I suppose I'm defending a case of "less is more")

I'd suggest tiering the development, from most valuable/easy to less:

1. Add the description field (as stated, this would mean parameters to move from a list to a list of dictionaries). This would also allow for "non-mandatory type hints", i.e.: description: Please, insure this looks like an octal number".
2. Allow for the parameter "window" to become a list selector, ideally the list validation would also work on command line invocations. This partially would also cover the "parameter type" scenario when the choices are limited.
3. I'm not so sure about the "label" concept: the parameter name already has to be a short string, thus acting already almost as a label, and the related environment variable can also be considered kind of a label too.
4. Same for validations: given a parameter's value must be a string and that it also should be validated by the runtime ¿does really add that much validating "a string that looks like a string" or "a string that looks like a number" or "a blob that can be rendered to the shell as a string", etc?

[tgross]

Yeah at the end of the day the task driver is going to get a bunch of strings. Given all the above bits I sort of wonder if we could leverage the existing mechanisms we have for HCL2 Variables to interpret the configuration of a dispatch meta block as well. That way there's not two different UXs for validation.