Dynamic field names in sideload()?

[graysonhead]

We have a environment that we are monitoring with a pretty high number of filesystems that presents a lot of variability in required threshold values, I've been using sideload with success for having host or group specific thresholds, but I am not sure I am going to be able to achieve my desired goal with device level overrides.

Here is what I have so far:

var window = 5m

var data = stream
|from()
    .measurement('disk')
    .where(lambda: "host" == 'test')
    .groupBy('host', 'device')
|window()
    .period(window)

|sideload()
    .source('file:///etc/kapacitor/sideload')
    .order('host/{{.host}}.yml', 'serverclass/{{.serverclass}}.yml')
    .field('disk_usage_warning_level', 95)
    .field('disk_usage_critical_level', 97)

|alert()
    .id('{{ .TaskName }}{{ index .Tags "host"}}//{{ index .Tags "device" }}')
    .message('[{{ .TaskName }}] {{ index .Tags "host" }} disk: {{ index .Tags "device" }} is at {{ index .Fields "used_percent" | printf "%0.2f" }}% utilization.')
    .warn(lambda: "used_percent" > "disk_usage_warning_level")
    .crit(lambda: "used_percent" > "disk_usage_critical_level")
    .pagerDuty2()

This works great, but it doesn't allow me to override at a path level.

Right now my 'test.yaml' override file for the test machine looks like:

#test.yaml
disk_usage_warning_level: 90
disk_usage_critical_level: 95

I want to be able to do something like this:

#test.yaml
disk_usage_warning:
    /var/local/docstore/index/docserviceD/rev2-complete: 30 #Matches a specific mountpoint
    /var.*: 99 #Regex matches any other mountpoints in /var
   default: 90 #Uses this if nothing more specific is found
disk_usage_critical: 95

Which I'm fairly certain is currently impossible.

I know normally I would be able to do something like adding to sideload.order: 'host/{{.host}}/{{.path}}.yaml', but the problem is that my paths have backslashes in them (of course) so I can't create a filename out of them anyways (plus, our environment has over 10,000 individual file systems....that is a lot of yaml files to manage).

Anyone have any suggestions on a better way to do this? Is it possible to dynamically name the sideload field to include the name of the path or something like that?

[mcassaniti]

You could use strReplace to build a sideload string and get around the fact you have / in your paths. That doesn't reduce the number of yaml files, but it does allow supporting paths properly.

[graysonhead]

I tried doing this, but it still grabs the host level override even though it is second in the list:

var window = 5m

var data = stream
|from()
    .measurement('disk')
    .where(lambda: "host" == 'tst-doc2')
    .groupBy('host', 'device', 'path')
|window()
    .period(window)
|eval(lambda: strReplace("path", '/', '_', -1))
    .as('pathsub')
    .keep()
|sideload()
    .source('file:///etc/kapacitor/sideload')
    .order('host/{{.host}}/{{.pathsub}}.yml', 'host/{{.host}}.yml', 'servergroup/{{.servergroup}}.yml')
    .field('disk_usage_warning_level', 95)
    .field('disk_usage_critical_level', 97)

|alert()
    .id('{{ .TaskName }}{{ index .Tags "host"}}//{{ index .Tags "device" }}')
    .message('[{{ .TaskName }}] {{ index .Tags "host" }} disk: {{ index .Tags "device" }} is at {{ index .Fields "used_percent" | printf "%0.2f" }}% utilization.')
    .warn(lambda: "used_percent" > "disk_usage_warning_level")
    .crit(lambda: "used_percent" > "disk_usage_critical_level")
    .pagerDuty2()
    .routingKey('e10d11f8888b44a7900cbb11d2689c98')

Should I be able to use multiple tag-names in a side-load file like that, or is it not supported?

[PeterSurda]

Should I be able to use multiple tag-names in a side-load file like that, or is it not supported?

This is a valid question, I'd also like to know the answer to that.

[PeterSurda]

Should I be able to use multiple tag-names in a side-load file like that, or is it not supported?

Based on my testing, it looks like you can use multiple tags inside one element of order, but only tags can be used. If you are having problems with tags containing slashes, try using a http(s) source instead of a file source.