Inconsistent disks' device label value (when using Device-mapper, LVM ...)

[finkr]

Host operating system:

Linux slxhmhs1 4.9.0-6-amd64 #1 SMP Debian 4.9.82-1+deb9u3 (2018-03-02) x86_64 GNU/Linux  

node_exporter version:

node_exporter, version 0.13.0+ds (branch: debian/sid, revision: 0.13.0+ds-1+b2)                                                  
  build user:       [email protected]                                                                   
  build date:       20170429-09:20:49                                                                                            
  go version:       go1.7.4                                                                                                      

node_exporter command line flags

/usr/bin/prometheus-node-exporter 
  -collector.diskstats.ignored-devices=^(ram|loop|fd)d+$
  -collector.filesystem.ignored-mount-points=^/(sys|proc|dev|run)($|/)
  -collector.textfile.directory=/var/lib/prometheus/node-exporter 

Are you running node_exporter in Docker?

No

What did you do that produced an error?

Just browse node-exporter's /metrics on a machine with LVM (or any other Device Mapper based storage)

What did you expect to see?

• Consistent label values for disk that use device-mapper (like LVM). And also consider:
• expose user-friendly device name (label values) for disk that use device-mapper.
• Expose more informations about LVM (LV -> VG mapping, PV->VG mapping ...)

What did you see instead?

The label value for device= aren't consistent for node_filesystem_.* and node_disk_.*. Also, node_disk_.* aren't user friendly / explicit.

node_filesystem_avail{device="/dev/mapper/vg1-lvroot",fstype="ext4",mountpoint="/"}

but:

node_disk_bytes_read{device="dm-0"} 
node_disk_bytes_written{device="dm-0"}
node_disk_io_now{device="dm-0"} 0

This is probably tough, since a device can have multiple names, see dmsetup info or

$find -L /dev -samefile /dev/dm-0                                                                                
/dev/vg1/lvroot
/dev/dm-0                                                                                                                        
/dev/mapper/vg1-lvroot
/dev/disk/by-uuid/620dd05c-beef-beef-beef-cafecafecafe
/dev/disk/by-id/dm-uuid-LVM-HB5blablablablahw                                     
/dev/disk/by-id/dm-name-vg1-lvroot 
/dev/block/254:0

(Keywords: LVM, volume group, logical volume, lv, vg, mirror, raid, multipath, crypt)

[finkr]

A given device-mapper device can have multiple logical names (multiple entries in /dev). Also, a given device can be mounted at multiple location (mountpoint), but two things are constants :

• The device's major:minor number
• The original device-mapper's name (in the case of LVM, it's based on the Volume group + Logical volume name)

It is actually easy to lookup the device-mapper's main name using Sysfs[1]. example using bash commands:

 $ stat -L -c "%t:%T %n" /dev/dm-0 /dev/mapper/vg1-lvroot
 fe:0 /dev/dm-0
 fe:0 /dev/mapper/vg1-lvroot

Then lookup in /sys:

$ cat /sys/dev/block/254:0/dm/name
vg1-lvroot

I also looked at the strace output of lsblk /dev/dm-0... it's exactly doing that !

$  strace lsblk /dev/dm-0
[..]
stat("/dev/dm-0", {st_mode=S_IFBLK|0660, st_rdev=makedev(254, 0), ...}) = 0
readlink("/sys/dev/block/254:0", "../../devices/virtual/block/dm-0", 4095) = 32
open("/sys/block/dm-0/dm/name", O_RDONLY|O_CLOEXEC) = 3
[..]

(This library may help: https://github.com/fntlnz/mountinfo to translate a device name into mountpoint)

[1] the dm/name in /sys/dev/block/254:0/dm/name exists at least since kernel 3.2 / Debian 7 (2013)

[SuperQ]

Thanks for writing this up. I agree, it would be good to have more consistent labels.

[tavyc]

I've been thinking about this issue.

I would like to have a mechanism to tell node_exporter which block device name to choose, because /dev/sdq2 means nothing to me on a machine with 20 disks. Device names should be stable across reboots, otherwise long-term graphs would be useless. This naming policy should be used by all collectors dealing with block device names.

Something like:

node_exporter \
  --blockdev-names="/dev/disk/by-label/.+" \
  --blockdev-names="/dev/mapper/.+" \
  --blockdev-names="/dev/disk/by-id/(?:ata|nvme|scsi|usb|dm-name|md-name)-.+" \

This would mean that for a device /dev/sda1 node_exporter will find all it's names by enumerating symlinks in /dev and pick the first matching regexp:

1. Prefer labels
2. Try device-mapper names (LVM, LUKS)
3. Try by-id
4. Fallback: use the original device name as-is

Someone else might like a different naming policy:

node_exporter --blockdev-names="/dev/disk/by-path/.+"  # Prefer physical device paths

I'd like to start working on a PR, but wanted to discuss the idea first. Is this the right approach?

[brian-brazil]

We currently work from /proc/diskstats. Any other joins would have to be done via PromQL with an info metric, otherwise this would get out of hand.

[SuperQ]

@tavyc This is an interesting idea, but I'm not sure how practical it is. For example, on one machine I have, /dev/disk/by-label/ only maps one device. So it might only be practical to have a separate metric that defines the relationship.

For example:

node_disk_label_info{device="/dev/sdb",device_label="data"} `

[finkr]

I have prototyped this with bash to create a textfile for node_exporter (snippet below). Sample output:

x_node_mountpoints_info{mountpoint="/", devname="/dev/mapper/vg-lv_root", device="dm-5"} 1
x_node_mountpoints_info{mountpoint="/boot", devname="/dev/sda1", device="sda1"} 1

Then I can write a query like:

    node_disk_bytes_read * on (instance, job,device) group_left(mountpoint) node_mountpoints_info{mountpoint="/"}

Which output looks like {device="dm-12",instance="localhost:9100",job="node",mountpoint="/"}

Note: Unfortunately, I had hard time to list all filesystem to create a graph : mountpoint= has to be a fixed value.. it can't be the regex .+ ( Error executing query: many-to-many matching not allowed: matching labels must be unique on one side). A workaround was to specify mountpoint=~"/.+" :

    node_disk_bytes_read * on (instance, job,device) group_left(mountpoint) x_node_mountpoints_info{mountpoint=~"/.+"}

---

prototype script

awk -F " "  '{if ($3 !~ /cgroup|devtmpfs|devpts|proc|sysfs|rpc_pipefs|debugfs|securityfs|binfmt_misc|fusectl/  ) print $0}' /proc/mounts 2>/dev/null  \
    | while read dev mp  fstype mntopts fs_freq fs_passno ; do
        if [ -e $dev ] ; then
            id="$(printf "%d:%d" $(stat -L  -c "0x%t 0x%T" "$dev"))"
            kname="$(basename $(dirname $(grep -lx "$id" /sys/block/*/dev /sys/block/*/*/dev)))"
            echo "x_node_mountpoints_info{mountpoint=\"$mp\", devname=\"$dev\", device=\"$kname\"} 1"
        fi;
    done > /var/lib/prometheus/node-exporter/x_node_mountpoints_info.$$
    mv /var/lib/prometheus/node-exporter/x_node_mountpoints_info.$$ /var/lib/prometheus/node-exporter/x_node_mountpoints_info.prom 

[discordianfish]

I have prototyped this with bash to create a textfile for node_exporter (snippet below). Sample output:

I think we should have something like this in the go collector. Contributions welcome! :)

Note: Unfortunately, I had hard time to list all filesystem to create a graph : mountpoint= has to be a fixed value.. it can't be the regex .+ ( Error executing query: many-to-many matching not allowed: matching labels must be unique on one side). A workaround was to specify mountpoint=~"/.+" :

Not sure I understand the problem. The error means that your selector returns multiple timeseries. It shouldn't matter if you used a regex or string to get there.

[rvalle]

@finkr how do you get LVM information via node_exporter at all? I cannot get any information about LVM mounts at all, while you mention that the information you receive is inconsistent. issue #543 metions that LVM requires root and that is the reason behind. Do you run node_exporter as root? or do you have any other solution?

[techc0de]

Hi,

I'm currently using prometheus/node exporter in a docker container, and after I added the following variables: --path.rootfs=/host, -v "/:/host:ro', it showing my luks drives, and main drive. How do I add the rslave option to -v "/:/host:ro in the portainer ?

[jessequinn]

@finkr how do you get LVM information via node_exporter at all? I cannot get any information about LVM mounts at all, while you mention that the information you receive is inconsistent. issue #543 metions that LVM requires root and that is the reason behind. Do you run node_exporter as root? or do you have any other solution?

mount your dev mappers and node export can pick them up.

for instance,

sudo mkdir -vp /mnt/lvm/{grafana,proxy,cortex}
sudo mount /dev/mapper/vg_1-containers_proxy /mnt/lvm/proxy/

Add to fstab if desired.

[dginhoux]

Hi,

I follow this issue since few time and now, i'm finding a way to translate xfs "{{ device }}" entry with LV Name or mountpoint... i think, my question'll find its answer in this issue when a solution will be validated.

Have a good day

[jcpunk]

I've also just run into this one...