glusterfs icon indicating copy to clipboard operation
glusterfs copied to clipboard

Published 20 hours ago • verified publisher icon gluster

io-stats: Introduce FOP_HITS_PER_SEC in profile data

Open mohit84 opened this issue 2 years ago • 44 comments

  1. Currently, glusterfs profile does not provide any metrics to measure what is the fops handling rate per second? Sometimes a user/developer can be interested to monitor this value.
  2. During testing i have observed the client profile is printing latency in microsec even time difference is captured in nanosecond.

Solution: 1) Analyze the FOP_HITS_PER_SEC during profile capture based on the last time duration interval divided by a total number of fop hits during that duration. 2) Correct the format to print the latency in nanosecond.

Fixes: #3734 Change-Id: Id0f043493854724d32d81dbe09e254d4d19ffdbb Signed-off-by: Mohit Agrawal [email protected]

mohit84 avatar Aug 23 '22 11:08 mohit84

/run regression

mohit84 avatar Aug 23 '22 11:08 mohit84

CLANG-FORMAT FAILURE: Before merging the patch, this diff needs to be considered for passing clang-format

index ffafdda20..38d44c922 100644
--- a/cli/src/cli-rpc-ops.c
+++ b/cli/src/cli-rpc-ops.c
@@ -967,9 +967,9 @@ xml_output:
                                       replica_count, disperse_count,
                                       redundancy_count, arbiter_count);
 
-        cli_out("Transport-type: %s",
-                ((transport == 0) ? "tcp"
-                                  : (transport == 1) ? "rdma" : "tcp,rdma"));
+        cli_out("Transport-type: %s", ((transport == 0)   ? "tcp"
+                                       : (transport == 1) ? "rdma"
+                                                          : "tcp,rdma"));
         j = 1;
 
         GF_FREE(local->get_vol.volname);

gluster-ant avatar Aug 23 '22 12:08 gluster-ant

1 test(s) failed ./tests/basic/afr/afr-read-hash-mode.t

1 test(s) generated core ./tests/basic/afr/afr-read-hash-mode.t

4 test(s) needed retry ./tests/000-flaky/basic_mount-nfs-auth.t ./tests/000-flaky/bugs_nfs_bug-1116503.t ./tests/000-flaky/glusterd-restart-shd-mux.t ./tests/basic/afr/afr-read-hash-mode.t

2 flaky test(s) marked as success even though they failed ./tests/000-flaky/basic_mount-nfs-auth.t ./tests/000-flaky/bugs_nfs_bug-1116503.t https://build.gluster.org/job/gh_centos7-regression/2724/

gluster-ant avatar Aug 23 '22 12:08 gluster-ant

/run regression

mohit84 avatar Aug 23 '22 12:08 mohit84

/run full regression

mohit84 avatar Aug 23 '22 12:08 mohit84

/run regression

mohit84 avatar Aug 23 '22 12:08 mohit84

/run regression

mohit84 avatar Aug 23 '22 12:08 mohit84

CLANG-FORMAT FAILURE: Before merging the patch, this diff needs to be considered for passing clang-format

index ffafdda20..38d44c922 100644
--- a/cli/src/cli-rpc-ops.c
+++ b/cli/src/cli-rpc-ops.c
@@ -967,9 +967,9 @@ xml_output:
                                       replica_count, disperse_count,
                                       redundancy_count, arbiter_count);
 
-        cli_out("Transport-type: %s",
-                ((transport == 0) ? "tcp"
-                                  : (transport == 1) ? "rdma" : "tcp,rdma"));
+        cli_out("Transport-type: %s", ((transport == 0)   ? "tcp"
+                                       : (transport == 1) ? "rdma"
+                                                          : "tcp,rdma"));
         j = 1;
 
         GF_FREE(local->get_vol.volname);

gluster-ant avatar Aug 23 '22 13:08 gluster-ant

/run regression

mohit84 avatar Aug 23 '22 15:08 mohit84

1 test(s) failed ./tests/bugs/distribute/bug-882278.t

0 test(s) generated core

5 test(s) needed retry ./tests/000-flaky/basic_afr_split-brain-favorite-child-policy.t ./tests/000-flaky/basic_mount-nfs-auth.t ./tests/000-flaky/bugs_nfs_bug-1116503.t ./tests/basic/hardlink-limit.t ./tests/bugs/distribute/bug-882278.t

2 flaky test(s) marked as success even though they failed ./tests/000-flaky/basic_mount-nfs-auth.t ./tests/000-flaky/bugs_nfs_bug-1116503.t https://build.gluster.org/job/gh_centos7-regression/2728/

gluster-ant avatar Aug 23 '22 17:08 gluster-ant

/run regression

mohit84 avatar Aug 24 '22 01:08 mohit84

1 test(s) failed ./tests/bugs/distribute/bug-882278.t

0 test(s) generated core

4 test(s) needed retry ./tests/000-flaky/basic_afr_split-brain-favorite-child-policy.t ./tests/000-flaky/basic_mount-nfs-auth.t ./tests/000-flaky/bugs_nfs_bug-1116503.t ./tests/bugs/distribute/bug-882278.t

2 flaky test(s) marked as success even though they failed ./tests/000-flaky/basic_mount-nfs-auth.t ./tests/000-flaky/bugs_nfs_bug-1116503.t https://build.gluster.org/job/gh_centos7-regression/2729/

gluster-ant avatar Aug 24 '22 05:08 gluster-ant

The test case ./tests/bugs/distribute/bug-882278.t is failing because of not being able to resolve a hostname, I think we need to add an entry in /etc/hosts. I have updated the same to the infra team.

mohit84 avatar Aug 24 '22 06:08 mohit84

/run regression

mscherer avatar Aug 24 '22 12:08 mscherer

Related to https://github.com/gluster/project-infrastructure/issues/170

mscherer avatar Aug 24 '22 12:08 mscherer

/run regression

mscherer avatar Aug 24 '22 14:08 mscherer

1 test(s) failed ./tests/basic/distribute/file-create.t

0 test(s) generated core

9 test(s) needed retry ./tests/000-flaky/basic_changelog_changelog-snapshot.t ./tests/000-flaky/basic_distribute_rebal-all-nodes-migrate.t ./tests/000-flaky/basic_mount-nfs-auth.t ./tests/000-flaky/bugs_distribute_bug-1117851.t ./tests/000-flaky/bugs_glusterd_bug-857330/normal.t ./tests/000-flaky/bugs_glusterd_bug-857330/xml.t ./tests/000-flaky/bugs_nfs_bug-1116503.t ./tests/basic/afr/ta-shd.t ./tests/basic/distribute/file-create.t

7 flaky test(s) marked as success even though they failed ./tests/000-flaky/basic_changelog_changelog-snapshot.t ./tests/000-flaky/basic_distribute_rebal-all-nodes-migrate.t ./tests/000-flaky/basic_mount-nfs-auth.t ./tests/000-flaky/bugs_distribute_bug-1117851.t ./tests/000-flaky/bugs_glusterd_bug-857330/normal.t ./tests/000-flaky/bugs_glusterd_bug-857330/xml.t ./tests/000-flaky/bugs_nfs_bug-1116503.t https://build.gluster.org/job/gh_centos7-regression/2734/

gluster-ant avatar Aug 24 '22 23:08 gluster-ant

/run regression

mohit84 avatar Aug 25 '22 00:08 mohit84

/run full regression

mohit84 avatar Aug 25 '22 00:08 mohit84

1 test(s) failed ./tests/00-geo-rep/georep-basic-tarssh-ec.t

0 test(s) generated core

11 test(s) needed retry ./tests/000-flaky/basic_changelog_changelog-snapshot.t ./tests/000-flaky/basic_distribute_rebal-all-nodes-migrate.t ./tests/000-flaky/basic_mount-nfs-auth.t ./tests/000-flaky/bugs_distribute_bug-1117851.t ./tests/000-flaky/bugs_glusterd_bug-857330/normal.t ./tests/000-flaky/bugs_glusterd_bug-857330/xml.t ./tests/000-flaky/bugs_nfs_bug-1116503.t ./tests/000-flaky/glusterd-restart-shd-mux.t ./tests/00-geo-rep/bug-1600145.t ./tests/00-geo-rep/georep-basic-dr-rsync-arbiter.t ./tests/00-geo-rep/georep-basic-tarssh-ec.t

7 flaky test(s) marked as success even though they failed ./tests/000-flaky/basic_distribute_rebal-all-nodes-migrate.t ./tests/000-flaky/basic_mount-nfs-auth.t ./tests/000-flaky/bugs_distribute_bug-1117851.t ./tests/000-flaky/bugs_glusterd_bug-857330/normal.t ./tests/000-flaky/bugs_glusterd_bug-857330/xml.t ./tests/000-flaky/bugs_nfs_bug-1116503.t ./tests/000-flaky/glusterd-restart-shd-mux.t https://build.gluster.org/job/gh_centos7-regression/2737/

gluster-ant avatar Aug 25 '22 02:08 gluster-ant

/run regression

mohit84 avatar Aug 25 '22 02:08 mohit84

1 test(s) failed ./tests/00-geo-rep/georep-basic-tarssh-ec.t

0 test(s) generated core

9 test(s) needed retry ./tests/000-flaky/basic_distribute_rebal-all-nodes-migrate.t ./tests/000-flaky/basic_mount-nfs-auth.t ./tests/000-flaky/bugs_distribute_bug-1117851.t ./tests/000-flaky/bugs_glusterd_bug-857330/normal.t ./tests/000-flaky/bugs_glusterd_bug-857330/xml.t ./tests/000-flaky/bugs_nfs_bug-1116503.t ./tests/000-flaky/glusterd-restart-shd-mux.t ./tests/00-geo-rep/georep-basic-dr-rsync-arbiter.t ./tests/00-geo-rep/georep-basic-tarssh-ec.t

7 flaky test(s) marked as success even though they failed ./tests/000-flaky/basic_distribute_rebal-all-nodes-migrate.t ./tests/000-flaky/basic_mount-nfs-auth.t ./tests/000-flaky/bugs_distribute_bug-1117851.t ./tests/000-flaky/bugs_glusterd_bug-857330/normal.t ./tests/000-flaky/bugs_glusterd_bug-857330/xml.t ./tests/000-flaky/bugs_nfs_bug-1116503.t ./tests/000-flaky/glusterd-restart-shd-mux.t https://build.gluster.org/job/gh_centos7-regression/2738/

gluster-ant avatar Aug 25 '22 04:08 gluster-ant

/run regression

mohit84 avatar Aug 25 '22 04:08 mohit84

1 test(s) failed ./tests/00-geo-rep/georep-basic-tarssh-ec.t

0 test(s) generated core

10 test(s) needed retry ./tests/000-flaky/basic_changelog_changelog-snapshot.t ./tests/000-flaky/basic_distribute_rebal-all-nodes-migrate.t ./tests/000-flaky/basic_mount-nfs-auth.t ./tests/000-flaky/bugs_distribute_bug-1117851.t ./tests/000-flaky/bugs_glusterd_bug-857330/normal.t ./tests/000-flaky/bugs_glusterd_bug-857330/xml.t ./tests/000-flaky/bugs_nfs_bug-1116503.t ./tests/000-flaky/glusterd-restart-shd-mux.t ./tests/00-geo-rep/georep-basic-dr-rsync-arbiter.t ./tests/00-geo-rep/georep-basic-tarssh-ec.t

7 flaky test(s) marked as success even though they failed ./tests/000-flaky/basic_distribute_rebal-all-nodes-migrate.t ./tests/000-flaky/basic_mount-nfs-auth.t ./tests/000-flaky/bugs_distribute_bug-1117851.t ./tests/000-flaky/bugs_glusterd_bug-857330/normal.t ./tests/000-flaky/bugs_glusterd_bug-857330/xml.t ./tests/000-flaky/bugs_nfs_bug-1116503.t ./tests/000-flaky/glusterd-restart-shd-mux.t https://build.gluster.org/job/gh_centos7-regression/2739/

gluster-ant avatar Aug 25 '22 06:08 gluster-ant

/run regression

mscherer avatar Aug 25 '22 12:08 mscherer

1 test(s) failed ./tests/00-geo-rep/georep-basic-tarssh-ec.t

0 test(s) generated core

9 test(s) needed retry ./tests/000-flaky/basic_distribute_rebal-all-nodes-migrate.t ./tests/000-flaky/basic_mount-nfs-auth.t ./tests/000-flaky/bugs_distribute_bug-1117851.t ./tests/000-flaky/bugs_glusterd_bug-857330/normal.t ./tests/000-flaky/bugs_glusterd_bug-857330/xml.t ./tests/000-flaky/bugs_nfs_bug-1116503.t ./tests/000-flaky/glusterd-restart-shd-mux.t ./tests/00-geo-rep/georep-basic-dr-rsync-arbiter.t ./tests/00-geo-rep/georep-basic-tarssh-ec.t

7 flaky test(s) marked as success even though they failed ./tests/000-flaky/basic_distribute_rebal-all-nodes-migrate.t ./tests/000-flaky/basic_mount-nfs-auth.t ./tests/000-flaky/bugs_distribute_bug-1117851.t ./tests/000-flaky/bugs_glusterd_bug-857330/normal.t ./tests/000-flaky/bugs_glusterd_bug-857330/xml.t ./tests/000-flaky/bugs_nfs_bug-1116503.t ./tests/000-flaky/glusterd-restart-shd-mux.t https://build.gluster.org/job/gh_centos7-regression/2742/

gluster-ant avatar Aug 25 '22 14:08 gluster-ant

/run regression

mohit84 avatar Aug 26 '22 11:08 mohit84

1 test(s) failed ./tests/bugs/rpc/bug-884452.t

0 test(s) generated core

8 test(s) needed retry ./tests/000-flaky/basic_mount-nfs-auth.t ./tests/000-flaky/bugs_nfs_bug-1116503.t ./tests/000-flaky/glusterd-restart-shd-mux.t ./tests/00-geo-rep/georep-basic-dr-tarssh-arbiter.t ./tests/bugs/glusterd/brick-order-check-add-brick.t ./tests/bugs/glusterfs-server/bug-887145.t ./tests/bugs/quota/bug-1035576.t ./tests/bugs/rpc/bug-884452.t

2 flaky test(s) marked as success even though they failed ./tests/000-flaky/basic_mount-nfs-auth.t ./tests/000-flaky/bugs_nfs_bug-1116503.t https://build.gluster.org/job/gh_centos7-regression/2748/

gluster-ant avatar Aug 26 '22 16:08 gluster-ant

/run regression

mohit84 avatar Aug 27 '22 05:08 mohit84

1 test(s) failed ./tests/basic/ec/ec-data-heal.t

0 test(s) generated core

3 test(s) needed retry ./tests/000-flaky/basic_mount-nfs-auth.t ./tests/000-flaky/bugs_nfs_bug-1116503.t ./tests/basic/ec/ec-data-heal.t

2 flaky test(s) marked as success even though they failed ./tests/000-flaky/basic_mount-nfs-auth.t ./tests/000-flaky/bugs_nfs_bug-1116503.t https://build.gluster.org/job/gh_centos7-regression/2749/

gluster-ant avatar Aug 27 '22 06:08 gluster-ant