state tag set under zpool_stats should be a field set instead

[toddhpoole]

Hi @richardelling, first off let me just say this is some great work. I've been following the original feature request for this on the telegraf side since late 2017 and I'm very excited to see it's now shipping in OpenZFS 2.1.0.

For the past 13 years, I've been using a Bash script to screen scrape the output of zpool status and feed it into various data stores and monitoring solutions. This has been brittle and given the sheer variety of useful metrics zpool_influxd is reporting, I'd like to now start using zpool_influxd.

I've taken a look at the InfluxDB line protocol emitted by zpool_influxdb and I'm curious about the design decision behind making state a tag set.

At the moment, I'm trying to find a way to extract status information about a given disk in a given vdev in a given pool (a use case which lends itself very well to templating in Grafana). Currently, zpool_influxdb embeds things like the state, path, and vdev of a disk into a tag set associated with a measurement named zpool_stats:

[root@testnas0 ~]# /usr/libexec/zfs/zpool_influxdb
zpool_stats,name=testtank0,state=ONLINE,vdev=root alloc=164827933913088u,free=27208643825664u,size=192036577738752u,read_bytes=33456128u,read_errors=0u,read_ops=5146u,write_bytes=32968704u,write_errors=0u,write_ops=2414u,checksum_errors=0u,fragmentation=0u 1631479854554473239
zpool_stats,name=testtank0,state=ONLINE,vdev=root/raidz-0 alloc=83201563766784u,free=12816725102592u,size=96018288869376u,read_bytes=16560128u,read_errors=0u,read_ops=2527u,write_bytes=17223680u,write_errors=0u,write_ops=1296u,checksum_errors=0u,fragmentation=0u 1631479854554473239
zpool_stats,name=testtank0,state=ONLINE,path=/dev/disk/by-id/wwn-0x5000c50087cfcfcc-part1,vdev=root/raidz-0/disk-0 alloc=0u,free=0u,size=0u,read_bytes=757760u,read_errors=0u,read_ops=69u,write_bytes=1495040u,write_errors=0u,write_ops=115u,checksum_errors=0u,fragmentation=0u 1631479854554473239
zpool_stats,name=testtank0,state=ONLINE,path=/dev/disk/by-id/wwn-0x5000cca252cef0ed-part1,vdev=root/raidz-0/disk-1 alloc=0u,free=0u,size=0u,read_bytes=684032u,read_errors=0u,read_ops=41u,write_bytes=1433600u,write_errors=0u,write_ops=104u,checksum_errors=0u,fragmentation=0u 1631479854554473239
zpool_stats,name=testtank0,state=ONLINE,path=/dev/disk/by-id/wwn-0x5000cca252cef787-part1,vdev=root/raidz-0/disk-2 alloc=0u,free=0u,size=0u,read_bytes=2568192u,read_errors=0u,read_ops=501u,write_bytes=1441792u,write_errors=0u,write_ops=106u,checksum_errors=0u,fragmentation=0u 1631479854554473239
... snip snip ...

Including path and vdev into the zpool_stats measurement as a tag set makes sense: InfluxDB tags are typically expected to be invariant over the life of a measurement, and so it's reasonable to keep properties that normally don't change (i.e., a disk's by-id path and its vdev location) stored as tags. But the state of a disk can be quite dynamic (off the top of my head I know there's OFFLINE, UNAVAILABLE, FAULTED, REMOVED, DEGRADED, and probably a few others I'm forgetting), so I believe it'd make more sense for state to be a field set just like what you've done with alloc, free, size, read_bytes, etc.

Shifting state from a tag to a field would not only be more in line with InfluxDB's world view, it would also help prevent InfluxDB instances consuming zpool_influxdb's output from creating unnecessary measurement entries whenever a disk's, vdev's, or zpool's state changes. See here for InfluxDB's perspective on tags sets and series cardinality, but basically: tags containing variable information will lead to a large number of unique series in an InfluxDB database each time those tags change. This in turn results in high series cardinality, which is a primary driver of high memory usage across many different types workloads, even low usage ones.

Lastly, shifting state to a field would also enable the new Stats panel in Grafana to display the state of each disk under an alias constructed from that disk's path or vdev location (which I don't believe is currently possible if state is stored in a tag since I'm not aware of a way to coax Grafana to convert a single tag set into a field set and display it under another tag set, though I could be wrong).

To be explicit, it'd be great if zpool_influxdb's output look a little something like this:

[root@testnas0 ~]# /usr/libexec/zfs/zpool_influxdb
zpool_stats,name=testtank0,vdev=root alloc=164827933913088u,free=27208643825664u,size=192036577738752u,read_bytes=33456128u,read_errors=0u,read_ops=5146u,write_bytes=32968704u,write_errors=0u,write_ops=2414u,checksum_errors=0u,fragmentation=0u,state=ONLINE 1631479854554473239
zpool_stats,name=testtank0,vdev=root/raidz-0 alloc=83201563766784u,free=12816725102592u,size=96018288869376u,read_bytes=16560128u,read_errors=0u,read_ops=2527u,write_bytes=17223680u,write_errors=0u,write_ops=1296u,checksum_errors=0u,fragmentation=0u,state=ONLINE 1631479854554473239
zpool_stats,name=testtank0,path=/dev/disk/by-id/wwn-0x5000c50087cfcfcc-part1,vdev=root/raidz-0/disk-0 alloc=0u,free=0u,size=0u,read_bytes=757760u,read_errors=0u,read_ops=69u,write_bytes=1495040u,write_errors=0u,write_ops=115u,checksum_errors=0u,fragmentation=0u,state=ONLINE 1631479854554473239
zpool_stats,name=testtank0,path=/dev/disk/by-id/wwn-0x5000cca252cef0ed-part1,vdev=root/raidz-0/disk-1 alloc=0u,free=0u,size=0u,read_bytes=684032u,read_errors=0u,read_ops=41u,write_bytes=1433600u,write_errors=0u,write_ops=104u,checksum_errors=0u,fragmentation=0u,state=ONLINE 1631479854554473239
zpool_stats,name=testtank0,path=/dev/disk/by-id/wwn-0x5000cca252cef787-part1,vdev=root/raidz-0/disk-2 alloc=0u,free=0u,size=0u,read_bytes=2568192u,read_errors=0u,read_ops=501u,write_bytes=1441792u,write_errors=0u,write_ops=106u,checksum_errors=0u,fragmentation=0u,state=ONLINE 1631479854554473239
... snip snip ...

(Note the elimination of the state tag set and the addition of the state field set at the end right before the timestamp.)

What do you think?

[richardelling]

There is a generic problem here: we want to know the state as a field, but also query the state. In influxdb we query on tags, not values. So we do need the state as a tag in order to make queries such as "count the number of unhealthy pools, vdevs, or devices" or "show the devices that are offline" in an efficient manner (because tags are indexed). But there are also cases where state makes sense as a field. I'm not opposed to adding it as a field.

NB, this is especially painful for prometheus because the only data type for a field (value) is double. For the zpool_prometheus implementation we put state as a tag (label) as well as a field (value). This is still quite painful because the data type must be a number and the state and other enums can change over the lifetime of the project. So it is only really feasible to do a state field query when it is text.

Thoughts?

[richardelling]

Also, the default interactive query generator for grafana is somewhat limited. When we want to see the state column, we hand edit the query. If you have a desired grafana dashboard can you share?

[mcondren]

Hello, so maybe I am missing something, but without STATE as a _field, how can I display "zpool state" in a grafana card? That was going to be my primary use case for this but using the influxDB flux query generator I can't output the state since it isn't a field. Any way with a flux query to do this? Looking for a card like "Backup zPool status: DEGRADED".

[teclabat]

... but without STATE as a _field, how can I display "zpool state" in a grafana card? ...

Would like to know this too. I am having the same problem.

[richardelling]

for grafana, there are two methods of creating an influxdb query: the query builder or edit by hand. In general, you can toggle back and forth between the two. Editing by hand will allow you to select a tag, but the grafana query builder only allows you to select a field. If you'd like to see an example, I can put together something over the weekend.

[teclabat]

Very much appreciated! I already tried myself, but without luck: SELECT last("state") FROM "zpool_stats" WHERE ("name" = 'sdstorage') AND $timeFilter also SELECT last("state") FROM "zpool_stats" WHERE ("name" = 'sdstorage') But there is no value in the table. I had a look into zpool_stats with the influx graph builder and there I see the ONLINE status.

[andrasg]

Following this topic as it wasn't straightforward at first for me either. The status info feels like a field to me rather than a tag. If you have it as a tag, then the ONLINE and not ONLINE values will be treated as a different series, and you will have to union and order two series together to see a common data set.

I would recommend considering making that a field.

[richardelling]

Comments on https://github.com/richardelling/zpool_influxdb/pull/12 appreciated. If this satisfies, then I'll make a similar PR to OpenZFS

[richardelling]

ping