node_exporter icon indicating copy to clipboard operation
node_exporter copied to clipboard

Published 20 hours ago • verified publisher icon prometheus

add avgRTT to NFS operation stats

Open ksankeerth opened this issue 1 year ago • 3 comments

Hi Team/ @discordianfish,

Please check this PR and let me know if anything has to be changed. Fixes #2550

We have to merge https://github.com/prometheus/procfs/pull/487 before merging this PR.

Thanks

ksankeerth avatar Mar 11 '23 12:03 ksankeerth

Build and tests failed because of missing field in struct. Once we merge procfs PR, I guess these errors won't come.

ksankeerth avatar Mar 11 '23 12:03 ksankeerth

Hi Team/ @discordianfish,

Since this PR is merged https://github.com/prometheus/procfs/pull/487, Can you please run CI again?

ksankeerth avatar May 11 '23 02:05 ksankeerth

@ksankeerth You need to update the procfs dependency here first

discordianfish avatar May 17 '23 12:05 discordianfish

@ksankeerth @discordianfish @SuperQ Sorry to bother you, any plan to merge this?

ethercflow avatar Mar 19 '24 08:03 ethercflow

I've rebased, let see

discordianfish avatar Mar 21 '24 12:03 discordianfish

@ksankeerth Looks like this still needs work

discordianfish avatar Mar 21 '24 12:03 discordianfish

@ksankeerth Looks like this still needs work

Sorry for missing this. I'll update tonight

ksankeerth avatar Mar 21 '24 12:03 ksankeerth

It seems, there are some e2e test failures. I'm looking at them.

ksankeerth avatar Mar 25 '24 03:03 ksankeerth

Hi @discordianfish / @SuperQ ,

I tested locally and can see NFS mountstats that I've added. But e2e tests fail because of old fixtures. Can You help me to update fixtures?

I've tried to update fixtures using command make update_fixtures but it doesn't update fixtures with new metrics.

 # HELP node_mountstats_nfs_operations_average_round_trip_time_seconds Duration from the time that the client's kernel sends the RPC request until the time it receives the reply.
# TYPE node_mountstats_nfs_operations_average_round_trip_time_seconds counter
node_mountstats_nfs_operations_average_round_trip_time_seconds{export="192.168.205.25:/var/nfs/general",mountaddr="192.168.205.25",operation="ACCESS",protocol="tcp"} 0.003
node_mountstats_nfs_operations_average_round_trip_time_seconds{export="192.168.205.25:/var/nfs/general",mountaddr="192.168.205.25",operation="ALLOCATE",protocol="tcp"} 0
node_mountstats_nfs_operations_average_round_trip_time_seconds{export="192.168.205.25:/var/nfs/general",mountaddr="192.168.205.25",operation="BIND_CONN_TO_SESSION",protocol="tcp"} 0
node_mountstats_nfs_operations_average_round_trip_time_seconds{export="192.168.205.25:/var/nfs/general",mountaddr="192.168.205.25",operation="CLONE",protocol="tcp"} 0
node_mountstats_nfs_operations_average_round_trip_time_seconds{export="192.168.205.25:/var/nfs/general",mountaddr="192.168.205.25",operation="CLOSE",protocol="tcp"} 0.00075
node_mountstats_nfs_operations_average_round_trip_time_seconds{export="192.168.205.25:/var/nfs/general",mountaddr="192.168.205.25",operation="COMMIT",protocol="tcp"} 0
node_mountstats_nfs_operations_average_round_trip_time_seconds{export="192.168.205.25:/var/nfs/general",mountaddr="192.168.205.25",operation="COPY",protocol="tcp"} 0
node_mountstats_nfs_operations_average_round_trip_time_seconds{export="192.168.205.25:/var/nfs/general",mountaddr="192.168.205.25",operation="COPY_NOTIFY",protocol="tcp"} 0
node_mountstats_nfs_operations_average_round_trip_time_seconds{export="192.168.205.25:/var/nfs/general",mountaddr="192.168.205.25",operation="CREATE",protocol="tcp"} 0
node_mountstats_nfs_operations_average_round_trip_time_seconds{export="192.168.205.25:/var/nfs/general",mountaddr="192.168.205.25",operation="CREATE_SESSION",protocol="tcp"} 0.001
node_mountstats_nfs_operations_average_round_trip_time_seconds{export="192.168.205.25:/var/nfs/general",mountaddr="192.168.205.25",operation="DEALLOCATE",protocol="tcp"} 0
node_mountstats_nfs_operations_average_round_trip_time_seconds{export="192.168.205.25:/var/nfs/general",mountaddr="192.168.205.25",operation="DELEGRETURN",protocol="tcp"} 0.0006666666666666666
node_mountstats_nfs_operations_average_round_trip_time_seconds{export="192.168.205.25:/var/nfs/general",mountaddr="192.168.205.25",operation="DESTROY_CLIENTID",protocol="tcp"} 0
node_mountstats_nfs_operations_average_round_trip_time_seconds{export="192.168.205.25:/var/nfs/general",mountaddr="192.168.205.25",operation="DESTROY_SESSION",protocol="tcp"} 0
node_mountstats_nfs_operations_average_round_trip_time_seconds{export="192.168.205.25:/var/nfs/general",mountaddr="192.168.205.25",operation="EXCHANGE_ID",protocol="tcp"} 0.0015
node_mountstats_nfs_operations_average_round_trip_time_seconds{export="192.168.205.25:/var/nfs/general",mountaddr="192.168.205.25",operation="FREE_STATEID",protocol="tcp"} 0
node_mountstats_nfs_operations_average_round_trip_time_seconds{export="192.168.205.25:/var/nfs/general",mountaddr="192.168.205.25",operation="FSID_PRESENT",protocol="tcp"} 0
node_mountstats_nfs_operations_average_round_trip_time_seconds{export="192.168.205.25:/var/nfs/general",mountaddr="192.168.205.25",operation="FSINFO",protocol="tcp"} 0.001
node_mountstats_nfs_operations_average_round_trip_time_seconds{export="192.168.205.25:/var/nfs/general",mountaddr="192.168.205.25",operation="FS_LOCATIONS",protocol="tcp"} 0.001
node_mountstats_nfs_operations_average_round_trip_time_seconds{export="192.168.205.25:/var/nfs/general",mountaddr="192.168.205.25",operation="GETACL",protocol="tcp"} 0
node_mountstats_nfs_operations_average_round_trip_time_seconds{export="192.168.205.25:/var/nfs/general",mountaddr="192.168.205.25",operation="GETATTR",protocol="tcp"} 0.0009210526315789473
node_mountstats_nfs_operations_average_round_trip_time_seconds{export="192.168.205.25:/var/nfs/general",mountaddr="192.168.205.25",operation="GETDEVICEINFO",protocol="tcp"} 0
node_mountstats_nfs_operations_average_round_trip_time_seconds{export="192.168.205.25:/var/nfs/general",mountaddr="192.168.205.25",operation="GETDEVICELIST",protocol="tcp"} 0
node_mountstats_nfs_operations_average_round_trip_time_seconds{export="192.168.205.25:/var/nfs/general",mountaddr="192.168.205.25",operation="GETXATTR",protocol="tcp"} 0
node_mountstats_nfs_operations_average_round_trip_time_seconds{export="192.168.205.25:/var/nfs/general",mountaddr="192.168.205.25",operation="GET_LEASE_TIME",protocol="tcp"} 0
node_mountstats_nfs_operations_average_round_trip_time_seconds{export="192.168.205.25:/var/nfs/general",mountaddr="192.168.205.25",operation="LAYOUTCOMMIT",protocol="tcp"} 0
node_mountstats_nfs_operations_average_round_trip_time_seconds{export="192.168.205.25:/var/nfs/general",mountaddr="192.168.205.25",operation="LAYOUTERROR",protocol="tcp"} 0
node_mountstats_nfs_operations_average_round_trip_time_seconds{export="192.168.205.25:/var/nfs/general",mountaddr="192.168.205.25",operation="LAYOUTGET",protocol="tcp"} 0
node_mountstats_nfs_operations_average_round_trip_time_seconds{export="192.168.205.25:/var/nfs/general",mountaddr="192.168.205.25",operation="LAYOUTRETURN",protocol="tcp"} 0
node_mountstats_nfs_operations_average_round_trip_time_seconds{export="192.168.205.25:/var/nfs/general",mountaddr="192.168.205.25",operation="LAYOUTSTATS",protocol="tcp"} 0
node_mountstats_nfs_operations_average_round_trip_time_seconds{export="192.168.205.25:/var/nfs/general",mountaddr="192.168.205.25",operation="LINK",protocol="tcp"} 0
node_mountstats_nfs_operations_average_round_trip_time_seconds{export="192.168.205.25:/var/nfs/general",mountaddr="192.168.205.25",operation="LISTXATTRS",protocol="tcp"} 0
node_mountstats_nfs_operations_average_round_trip_time_seconds{export="192.168.205.25:/var/nfs/general",mountaddr="192.168.205.25",operation="LOCK",protocol="tcp"} 0
node_mountstats_nfs_operations_average_round_trip_time_seconds{export="192.168.205.25:/var/nfs/general",mountaddr="192.168.205.25",operation="LOCKT",protocol="tcp"} 0
node_mountstats_nfs_operations_average_round_trip_time_seconds{export="192.168.205.25:/var/nfs/general",mountaddr="192.168.205.25",operation="LOCKU",protocol="tcp"} 0
node_mountstats_nfs_operations_average_round_trip_time_seconds{export="192.168.205.25:/var/nfs/general",mountaddr="192.168.205.25",operation="LOOKUP",protocol="tcp"} 0.001
node_mountstats_nfs_operations_average_round_trip_time_seconds{export="192.168.205.25:/var/nfs/general",mountaddr="192.168.205.25",operation="LOOKUPP",protocol="tcp"} 0
node_mountstats_nfs_operations_average_round_trip_time_seconds{export="192.168.205.25:/var/nfs/general",mountaddr="192.168.205.25",operation="LOOKUP_ROOT",protocol="tcp"} 0
node_mountstats_nfs_operations_average_round_trip_time_seconds{export="192.168.205.25:/var/nfs/general",mountaddr="192.168.205.25",operation="NULL",protocol="tcp"} 0
node_mountstats_nfs_operations_average_round_trip_time_seconds{export="192.168.205.25:/var/nfs/general",mountaddr="192.168.205.25",operation="OFFLOAD_CANCEL",protocol="tcp"} 0
node_mountstats_nfs_operations_average_round_trip_time_seconds{export="192.168.205.25:/var/nfs/general",mountaddr="192.168.205.25",operation="OPEN",protocol="tcp"} 0.0038333333333333336
node_mountstats_nfs_operations_average_round_trip_time_seconds{export="192.168.205.25:/var/nfs/general",mountaddr="192.168.205.25",operation="OPEN_CONFIRM",protocol="tcp"} 0
node_mountstats_nfs_operations_average_round_trip_time_seconds{export="192.168.205.25:/var/nfs/general",mountaddr="192.168.205.25",operation="OPEN_DOWNGRADE",protocol="tcp"} 0
node_mountstats_nfs_operations_average_round_trip_time_seconds{export="192.168.205.25:/var/nfs/general",mountaddr="192.168.205.25",operation="OPEN_NOATTR",protocol="tcp"} 0
node_mountstats_nfs_operations_average_round_trip_time_seconds{export="192.168.205.25:/var/nfs/general",mountaddr="192.168.205.25",operation="PATHCONF",protocol="tcp"} 0.001
node_mountstats_nfs_operations_average_round_trip_time_seconds{export="192.168.205.25:/var/nfs/general",mountaddr="192.168.205.25",operation="READ",protocol="tcp"} 0
node_mountstats_nfs_operations_average_round_trip_time_seconds{export="192.168.205.25:/var/nfs/general",mountaddr="192.168.205.25",operation="READDIR",protocol="tcp"} 0.002
node_mountstats_nfs_operations_average_round_trip_time_seconds{export="192.168.205.25:/var/nfs/general",mountaddr="192.168.205.25",operation="READLINK",protocol="tcp"} 0
node_mountstats_nfs_operations_average_round_trip_time_seconds{export="192.168.205.25:/var/nfs/general",mountaddr="192.168.205.25",operation="READ_PLUS",protocol="tcp"} 0
node_mountstats_nfs_operations_average_round_trip_time_seconds{export="192.168.205.25:/var/nfs/general",mountaddr="192.168.205.25",operation="RECLAIM_COMPLETE",protocol="tcp"} 0.009
node_mountstats_nfs_operations_average_round_trip_time_seconds{export="192.168.205.25:/var/nfs/general",mountaddr="192.168.205.25",operation="RELEASE_LOCKOWNER",protocol="tcp"} 0
node_mountstats_nfs_operations_average_round_trip_time_seconds{export="192.168.205.25:/var/nfs/general",mountaddr="192.168.205.25",operation="REMOVE",protocol="tcp"} 0.0036666666666666666
node_mountstats_nfs_operations_average_round_trip_time_seconds{export="192.168.205.25:/var/nfs/general",mountaddr="192.168.205.25",operation="REMOVEXATTR",protocol="tcp"} 0
node_mountstats_nfs_operations_average_round_trip_time_seconds{export="192.168.205.25:/var/nfs/general",mountaddr="192.168.205.25",operation="RENAME",protocol="tcp"} 0
node_mountstats_nfs_operations_average_round_trip_time_seconds{export="192.168.205.25:/var/nfs/general",mountaddr="192.168.205.25",operation="RENEW",protocol="tcp"} 0
node_mountstats_nfs_operations_average_round_trip_time_seconds{export="192.168.205.25:/var/nfs/general",mountaddr="192.168.205.25",operation="SECINFO",protocol="tcp"} 0
node_mountstats_nfs_operations_average_round_trip_time_seconds{export="192.168.205.25:/var/nfs/general",mountaddr="192.168.205.25",operation="SECINFO_NO_NAME",protocol="tcp"} 0
node_mountstats_nfs_operations_average_round_trip_time_seconds{export="192.168.205.25:/var/nfs/general",mountaddr="192.168.205.25",operation="SEEK",protocol="tcp"} 0
node_mountstats_nfs_operations_average_round_trip_time_seconds{export="192.168.205.25:/var/nfs/general",mountaddr="192.168.205.25",operation="SEQUENCE",protocol="tcp"} 0.003142857142857143
node_mountstats_nfs_operations_average_round_trip_time_seconds{export="192.168.205.25:/var/nfs/general",mountaddr="192.168.205.25",operation="SERVER_CAPS",protocol="tcp"} 0.001
node_mountstats_nfs_operations_average_round_trip_time_seconds{export="192.168.205.25:/var/nfs/general",mountaddr="192.168.205.25",operation="SETACL",protocol="tcp"} 0
node_mountstats_nfs_operations_average_round_trip_time_seconds{export="192.168.205.25:/var/nfs/general",mountaddr="192.168.205.25",operation="SETATTR",protocol="tcp"} 0.00425
node_mountstats_nfs_operations_average_round_trip_time_seconds{export="192.168.205.25:/var/nfs/general",mountaddr="192.168.205.25",operation="SETCLIENTID",protocol="tcp"} 0
node_mountstats_nfs_operations_average_round_trip_time_seconds{export="192.168.205.25:/var/nfs/general",mountaddr="192.168.205.25",operation="SETCLIENTID_CONFIRM",protocol="tcp"} 0
node_mountstats_nfs_operations_average_round_trip_time_seconds{export="192.168.205.25:/var/nfs/general",mountaddr="192.168.205.25",operation="SETXATTR",protocol="tcp"} 0
node_mountstats_nfs_operations_average_round_trip_time_seconds{export="192.168.205.25:/var/nfs/general",mountaddr="192.168.205.25",operation="STATFS",protocol="tcp"} 0.00175
node_mountstats_nfs_operations_average_round_trip_time_seconds{export="192.168.205.25:/var/nfs/general",mountaddr="192.168.205.25",operation="SYMLINK",protocol="tcp"} 0
node_mountstats_nfs_operations_average_round_trip_time_seconds{export="192.168.205.25:/var/nfs/general",mountaddr="192.168.205.25",operation="TEST_STATEID",protocol="tcp"} 0
node_mountstats_nfs_operations_average_round_trip_time_seconds{export="192.168.205.25:/var/nfs/general",mountaddr="192.168.205.25",operation="WRITE",protocol="tcp"} 0.0065

Thanks, Sankeerthan

ksankeerth avatar Mar 26 '24 04:03 ksankeerth

Hi Team,

Updated PR with required changes. Please check and let me know if more changes have to be done.

Thanks

ksankeerth avatar Mar 27 '24 03:03 ksankeerth

I was just reviewing this, and the original procfs PR.

This is not a metric we can accept, as it is a computation against two metrics we already expose.

node_mountstats_operations_response_time_seconds_total / node_mountstats_operations_requests_total

Worse, this is actually an invalid and misleading new metric, because it produces a "since beginning of time" average, rather than the current average response time which is provided when you do the correct PromQL:

rate(node_mountstats_operations_response_time_seconds_total[1m]) / rate(node_mountstats_operations_requests_total[1m])

SuperQ avatar Mar 28 '24 07:03 SuperQ