Collected metrics were collected with same name and label values

[donjuanmon]

Hello,

One of my hosts running this exporter is showing collected metric "xxx" { label:<name:"server" value:"xxx" > label:<name:"share" value:"xxx" > gauge:<value:xxx > } was collected before with the same name and label values for all collected metrics.

All other hosts in the cluster are showing normal values. Any idea what may be causing this?

[shibumi]

Hi,

I never thought that somebody really uses this project :D it is some artifact from my bachelor thesis.

Can you provide an output of /proc/fs/cifs/Stats?

[andrew-backeby]

Just wanted to add, my company is also using this project, we have not seen this issue ourselves tho.

[shibumi]

@andrew-backeby ok cool! Such information is really important to me, because this means that I have to spend some more time into the project (cleanups, tests, etc) :) thanks for letting me know.

[donjuanmon]

hey @shibumi, just wanted to add we have it deployed to around 150 servers :) We will keep using it until node_exporter provides better cifs metrics.

Here is the output (sanitized)

[donjuanmon@host ~]$ sudo cat /proc/fs/cifs/Stats
Resources in use
CIFS Session: 2
Share (unique mount targets): 4
SMB Request/Response Buffer: 2 Pool size: 6
SMB Small Req/Resp Buffer: 2 Pool size: 30
Operations (MIDs): 0

0 session 0 share reconnects
Total vfs operations: 577118 maximum at one time: 4

1) \\server\NoSnap
SMBs: 864979
Negotiates: 0 sent 0 failed
SessionSetups: 0 sent 0 failed
Logoffs: 0 sent 0 failed
TreeConnects: 0 sent 0 failed
TreeDisconnects: 0 sent 0 failed
Creates: 0 sent 0 failed
Closes: 0 sent 0 failed
Flushes: 0 sent 0 failed
Reads: 0 sent 0 failed
Writes: 0 sent 0 failed
Locks: 0 sent 0 failed
IOCTLs: 0 sent 0 failed
Cancels: 0 sent 0 failed
Echos: 0 sent 0 failed
QueryDirectories: 0 sent 6 failed
ChangeNotifies: 0 sent 0 failed
QueryInfos: 0 sent 2 failed
SetInfos: 0 sent 0 failed
OplockBreaks: 0 sent 0 failed
2) \\server\NoSnap
SMBs: 866873
Negotiates: 0 sent 0 failed
SessionSetups: 0 sent 0 failed
Logoffs: 0 sent 0 failed
TreeConnects: 0 sent 0 failed
TreeDisconnects: 0 sent 0 failed
Creates: 0 sent 0 failed
Closes: 0 sent 0 failed
Flushes: 0 sent 0 failed
Reads: 0 sent 0 failed
Writes: 0 sent 0 failed
Locks: 0 sent 0 failed
IOCTLs: 0 sent 0 failed
Cancels: 0 sent 0 failed
Echos: 0 sent 0 failed
QueryDirectories: 0 sent 6 failed
ChangeNotifies: 0 sent 0 failed
QueryInfos: 0 sent 2 failed
SetInfos: 0 sent 0 failed
OplockBreaks: 0 sent 0 failed

Are the identical names of the mounted cifs causing problems?

[shibumi]

Hi @donjuanmon this project was actually a proof of concept for node_exporter. The reason why I made a separate exporter out of this was that the node_exporter maintainers were not happy about the regex. They told me that there must be another solution for parsing the NFS statistics, but they did not come up with a better solution either. I am not sure if there ever will be an integration into the node_exporter. CC: @superq (one of the prometheus devs)

Are the identical names of the mounted cifs causing problems?

They might be a problem. I will try to debug this asap, my company doesn't let me do open source during work hours :(

[donjuanmon]

@shibumi something must have been wrong with the host, after a reboot /Stats shows the a single mount point and the exporter is working normally. Thank you for the response though and please let me know if I can help test any future releases!

[shibumi]

@donjuanmon ah, so you never configured two mount points with the same name? I wonder if I can mitigate such bugs in the future

[shibumi]

Re-opened, because this is worth investigating.

[donjuanmon]

@shibumi correct, I only have one configured on this host which led to some confusion when I saw the duplicate. Here's the output for a host with multiple cifs mounts configured:

Resources in use
CIFS Session: 2
Share (unique mount targets): 4
SMB Request/Response Buffer: 2 Pool size: 6
SMB Small Req/Resp Buffer: 2 Pool size: 30
Operations (MIDs): 0

12 session 8 share reconnects
Total vfs operations: 1486057 maximum at one time: 7

1) \\server\NoSnap
SMBs: 2227436
Negotiates: 0 sent 0 failed
SessionSetups: 0 sent 0 failed
Logoffs: 0 sent 0 failed
TreeConnects: 0 sent 0 failed
TreeDisconnects: 0 sent 0 failed
Creates: 0 sent 0 failed
Closes: 0 sent 0 failed
Flushes: 0 sent 0 failed
Reads: 0 sent 0 failed
Writes: 0 sent 0 failed
Locks: 0 sent 0 failed
IOCTLs: 0 sent 0 failed
Cancels: 0 sent 0 failed
Echos: 0 sent 0 failed
QueryDirectories: 0 sent 0 failed
ChangeNotifies: 0 sent 0 failed
QueryInfos: 0 sent 2 failed
SetInfos: 0 sent 0 failed
OplockBreaks: 0 sent 0 failed
2) \\server\DataNonProd
SMBs: 2227400
Negotiates: 0 sent 0 failed
SessionSetups: 0 sent 0 failed
Logoffs: 0 sent 0 failed
TreeConnects: 0 sent 0 failed
TreeDisconnects: 0 sent 0 failed
Creates: 0 sent 0 failed
Closes: 0 sent 0 failed
Flushes: 0 sent 0 failed
Reads: 0 sent 0 failed
Writes: 0 sent 0 failed
Locks: 0 sent 0 failed
IOCTLs: 0 sent 0 failed
Cancels: 0 sent 0 failed
Echos: 0 sent 0 failed
QueryDirectories: 0 sent 0 failed
ChangeNotifies: 0 sent 0 failed
QueryInfos: 0 sent 2 failed
SetInfos: 0 sent 0 failed
OplockBreaks: 0 sent 0 failed

The exporter works with two different shares defined.

[donjuanmon]

hey @shibumi, I can confirm that two cifs mounts pointed to the same share will create the duplicated metric error originally described. Not sure how the exporter can parse for duplicate entries in /proc/fs/cifs/Stats but that seems to be primary cause of the problem.

[shibumi]

@donjuanmon Interesting. I might have an idea how to fix this. Do you know if both sections are the same? Are all values equal? Or are the values of one section wrong?

I wonder if this is a bug in the kernel cifs stats.