glusterfs icon indicating copy to clipboard operation
glusterfs copied to clipboard

Published 20 hours ago • verified publisher icon gluster

Georep: make passive worker sync self-heal traffic

Open schaffung opened this issue 4 years ago • 19 comments

Problem: When the worker corresponding to a brick which was down becomes Active after the brick comes up, there is a race window where the I/O which happened when the brick was down would not sync to slave. This can result in data loss at slave.

Fix: As the lost fop comes as self heal traffic, we will make the passive workers to sync self heal traffic. This way, we ensure the brick which turned passive from active has received its fops.

Steps:

  1. make passive worker sync MKNOD
  2. the time for this sync is updated as stime. When made into an active worker we use this stime to resume sync.
  3. make clusterstime as a new xattr and store it.
  4. to avoid passive's stime crossing active stime, check the cluster stime and then proceed.

Change-Id: I96a703744cb211df03c69152594c812b1af87635 fixes: #1049 Signed-off-by: Hari Gowtham [email protected] Signed-off-by: srijan-sivakumar [email protected]

schaffung avatar Oct 12 '20 06:10 schaffung

Can one of the admins verify this patch?

gluster-ant avatar Oct 12 '20 06:10 gluster-ant

Can one of the admins verify this patch?

gluster-ant avatar Oct 12 '20 06:10 gluster-ant

Can one of the admins verify this patch?

gluster-ant avatar Oct 12 '20 06:10 gluster-ant

/recheck smoke

schaffung avatar Oct 12 '20 06:10 schaffung

/run regression

schaffung avatar Nov 05 '20 07:11 schaffung

/run regression

schaffung avatar Nov 17 '20 07:11 schaffung

/run regression

schaffung avatar Nov 17 '20 07:11 schaffung

/run regression

schaffung avatar Nov 17 '20 08:11 schaffung

/run regression

schaffung avatar Nov 17 '20 09:11 schaffung

Thank you for your contributions. Noticed that this issue is not having any activity in last ~6 months! We are marking this issue as stale because it has not had recent activity. It will be closed in 2 weeks if no one responds with a comment here.

stale[bot] avatar Jul 20 '21 09:07 stale[bot]

Closing this issue as there was no update since my last update on issue. If this is an issue which is still valid, feel free to open it.

stale[bot] avatar Aug 04 '21 17:08 stale[bot]

/run regression

Shwetha-Acharya avatar Nov 29 '21 10:11 Shwetha-Acharya

CLANG-FORMAT FAILURE: Before merging the patch, this diff needs to be considered for passing clang-format

index 3c4458c3b..7679c9794 100644
--- a/xlators/cluster/dht/src/dht-rebalance.c
+++ b/xlators/cluster/dht/src/dht-rebalance.c
@@ -2938,7 +2938,6 @@ gf_defrag_task(void *opaque)
     set_lk_owner_from_ptr(&lkowner, &lkowner);
     syncopctx_setfslkowner(&lkowner);
 
-
     q_head = &(defrag->queue[0].list);
 
     /* The following while loop will dequeue one entry from the defrag->queue
diff --git a/xlators/debug/io-stats/src/io-stats.c b/xlators/debug/io-stats/src/io-stats.c
index 0eac50117..a36336736 100644
--- a/xlators/debug/io-stats/src/io-stats.c
+++ b/xlators/debug/io-stats/src/io-stats.c
@@ -3765,10 +3765,10 @@ reconfigure(xlator_t *this, dict_t *options)
     GF_OPTION_RECONF("ios-dump-format", dump_format_str, options, str, out);
     ios_set_log_format_code(conf, dump_format_str);
     if (conf->ios_sample_interval) {
-        GF_OPTION_RECONF("ios-sample-buf-size", conf->ios_sample_buf_size, options,
-                         int32, out);
+        GF_OPTION_RECONF("ios-sample-buf-size", conf->ios_sample_buf_size,
+                         options, int32, out);
     } else {
-        ios_sample_buf_size_configure (this->name, conf);
+        ios_sample_buf_size_configure(this->name, conf);
     }
 
     GF_OPTION_RECONF("sys-log-level", sys_log_str, options, str, out);
@@ -3951,7 +3951,7 @@ init(xlator_t *this)
         GF_OPTION_INIT("ios-sample-buf-size", conf->ios_sample_buf_size, int32,
                        out);
     } else {
-        ios_sample_buf_size_configure (this->name, conf);
+        ios_sample_buf_size_configure(this->name, conf);
     }
 
     ret = ios_init_sample_buf(conf);
diff --git a/xlators/mgmt/glusterd/src/glusterd-volume-set.c b/xlators/mgmt/glusterd/src/glusterd-volume-set.c
index 5652b7e69..a9f7353f7 100644
--- a/xlators/mgmt/glusterd/src/glusterd-volume-set.c
+++ b/xlators/mgmt/glusterd/src/glusterd-volume-set.c
@@ -1460,12 +1460,10 @@ struct volopt_map_entry glusterd_volopt_map[] = {
      .voltype = "performance/md-cache",
      .option = "pass-through",
      .op_version = GD_OP_VERSION_4_1_0},
-    {
-        .key = "performance.write-behind-pass-through",
-        .voltype = "performance/write-behind",
-        .option = "pass-through",
-        .op_version = GD_OP_VERSION_9_0
-    },
+    {.key = "performance.write-behind-pass-through",
+     .voltype = "performance/write-behind",
+     .option = "pass-through",
+     .op_version = GD_OP_VERSION_9_0},
     {.key = "performance.md-cache-timeout",
      .voltype = "performance/md-cache",
      .option = "md-cache-timeout",
diff --git a/xlators/performance/write-behind/src/write-behind.c b/xlators/performance/write-behind/src/write-behind.c
index c46ffb19e..243ed511c 100644
--- a/xlators/performance/write-behind/src/write-behind.c
+++ b/xlators/performance/write-behind/src/write-behind.c
@@ -3191,15 +3191,13 @@ struct volume_options options[] = {
         .op_version = {GD_OP_VERSION_6_0},
         .flags = OPT_FLAG_SETTABLE,
     },
-    {
-        .key = {"pass-through"},
-        .type = GF_OPTION_TYPE_BOOL,
-        .default_value = "false",
-        .op_version = {GD_OP_VERSION_9_0},
-        .flags = OPT_FLAG_SETTABLE | OPT_FLAG_DOC | OPT_FLAG_CLIENT_OPT,
-        .tags = {"write-behind"},
-        .description = "Enable/disable write-behind pass-through"
-    },
+    {.key = {"pass-through"},
+     .type = GF_OPTION_TYPE_BOOL,
+     .default_value = "false",
+     .op_version = {GD_OP_VERSION_9_0},
+     .flags = OPT_FLAG_SETTABLE | OPT_FLAG_DOC | OPT_FLAG_CLIENT_OPT,
+     .tags = {"write-behind"},
+     .description = "Enable/disable write-behind pass-through"},
     {.key = {"flush-behind"},
      .type = GF_OPTION_TYPE_BOOL,
      .default_value = "on",

gluster-ant avatar Nov 29 '21 10:11 gluster-ant

1 test(s) failed ./tests/00-geo-rep/passive_sync.t

0 test(s) generated core

3 test(s) needed retry ./tests/000-flaky/basic_distribute_rebal-all-nodes-migrate.t ./tests/000-flaky/basic_ec_ec-quorum-count-partial-failure.t ./tests/00-geo-rep/passive_sync.t https://build.gluster.org/job/gh_centos7-regression/1876/

gluster-ant avatar Nov 29 '21 10:11 gluster-ant

CLANG-FORMAT FAILURE: Before merging the patch, this diff needs to be considered for passing clang-format

index 4d068dbe4..b417792a7 100644
--- a/cli/src/cli-rpc-ops.c
+++ b/cli/src/cli-rpc-ops.c
@@ -917,9 +917,9 @@ xml_output:
             type, brick_count, dist_count, stripe_count, replica_count,
             disperse_count, redundancy_count, arbiter_count);
 
-        cli_out("Transport-type: %s",
-                ((transport == 0) ? "tcp"
-                                  : (transport == 1) ? "rdma" : "tcp,rdma"));
+        cli_out("Transport-type: %s", ((transport == 0)   ? "tcp"
+                                       : (transport == 1) ? "rdma"
+                                                          : "tcp,rdma"));
         j = 1;
 
         GF_FREE(local->get_vol.volname);
diff --git a/xlators/cluster/dht/src/dht-rebalance.c b/xlators/cluster/dht/src/dht-rebalance.c
index 3c4458c3b..7679c9794 100644
--- a/xlators/cluster/dht/src/dht-rebalance.c
+++ b/xlators/cluster/dht/src/dht-rebalance.c
@@ -2938,7 +2938,6 @@ gf_defrag_task(void *opaque)
     set_lk_owner_from_ptr(&lkowner, &lkowner);
     syncopctx_setfslkowner(&lkowner);
 
-
     q_head = &(defrag->queue[0].list);
 
     /* The following while loop will dequeue one entry from the defrag->queue
diff --git a/xlators/debug/io-stats/src/io-stats.c b/xlators/debug/io-stats/src/io-stats.c
index 0eac50117..a36336736 100644
--- a/xlators/debug/io-stats/src/io-stats.c
+++ b/xlators/debug/io-stats/src/io-stats.c
@@ -3765,10 +3765,10 @@ reconfigure(xlator_t *this, dict_t *options)
     GF_OPTION_RECONF("ios-dump-format", dump_format_str, options, str, out);
     ios_set_log_format_code(conf, dump_format_str);
     if (conf->ios_sample_interval) {
-        GF_OPTION_RECONF("ios-sample-buf-size", conf->ios_sample_buf_size, options,
-                         int32, out);
+        GF_OPTION_RECONF("ios-sample-buf-size", conf->ios_sample_buf_size,
+                         options, int32, out);
     } else {
-        ios_sample_buf_size_configure (this->name, conf);
+        ios_sample_buf_size_configure(this->name, conf);
     }
 
     GF_OPTION_RECONF("sys-log-level", sys_log_str, options, str, out);
@@ -3951,7 +3951,7 @@ init(xlator_t *this)
         GF_OPTION_INIT("ios-sample-buf-size", conf->ios_sample_buf_size, int32,
                        out);
     } else {
-        ios_sample_buf_size_configure (this->name, conf);
+        ios_sample_buf_size_configure(this->name, conf);
     }
 
     ret = ios_init_sample_buf(conf);
diff --git a/xlators/mgmt/glusterd/src/glusterd-volume-set.c b/xlators/mgmt/glusterd/src/glusterd-volume-set.c
index 5652b7e69..a9f7353f7 100644
--- a/xlators/mgmt/glusterd/src/glusterd-volume-set.c
+++ b/xlators/mgmt/glusterd/src/glusterd-volume-set.c
@@ -1460,12 +1460,10 @@ struct volopt_map_entry glusterd_volopt_map[] = {
      .voltype = "performance/md-cache",
      .option = "pass-through",
      .op_version = GD_OP_VERSION_4_1_0},
-    {
-        .key = "performance.write-behind-pass-through",
-        .voltype = "performance/write-behind",
-        .option = "pass-through",
-        .op_version = GD_OP_VERSION_9_0
-    },
+    {.key = "performance.write-behind-pass-through",
+     .voltype = "performance/write-behind",
+     .option = "pass-through",
+     .op_version = GD_OP_VERSION_9_0},
     {.key = "performance.md-cache-timeout",
      .voltype = "performance/md-cache",
      .option = "md-cache-timeout",
diff --git a/xlators/performance/write-behind/src/write-behind.c b/xlators/performance/write-behind/src/write-behind.c
index c46ffb19e..243ed511c 100644
--- a/xlators/performance/write-behind/src/write-behind.c
+++ b/xlators/performance/write-behind/src/write-behind.c
@@ -3191,15 +3191,13 @@ struct volume_options options[] = {
         .op_version = {GD_OP_VERSION_6_0},
         .flags = OPT_FLAG_SETTABLE,
     },
-    {
-        .key = {"pass-through"},
-        .type = GF_OPTION_TYPE_BOOL,
-        .default_value = "false",
-        .op_version = {GD_OP_VERSION_9_0},
-        .flags = OPT_FLAG_SETTABLE | OPT_FLAG_DOC | OPT_FLAG_CLIENT_OPT,
-        .tags = {"write-behind"},
-        .description = "Enable/disable write-behind pass-through"
-    },
+    {.key = {"pass-through"},
+     .type = GF_OPTION_TYPE_BOOL,
+     .default_value = "false",
+     .op_version = {GD_OP_VERSION_9_0},
+     .flags = OPT_FLAG_SETTABLE | OPT_FLAG_DOC | OPT_FLAG_CLIENT_OPT,
+     .tags = {"write-behind"},
+     .description = "Enable/disable write-behind pass-through"},
     {.key = {"flush-behind"},
      .type = GF_OPTION_TYPE_BOOL,
      .default_value = "on",

gluster-ant avatar Dec 22 '21 17:12 gluster-ant

CLANG-FORMAT FAILURE: Before merging the patch, this diff needs to be considered for passing clang-format

index 4d068dbe4..b417792a7 100644
--- a/cli/src/cli-rpc-ops.c
+++ b/cli/src/cli-rpc-ops.c
@@ -917,9 +917,9 @@ xml_output:
             type, brick_count, dist_count, stripe_count, replica_count,
             disperse_count, redundancy_count, arbiter_count);
 
-        cli_out("Transport-type: %s",
-                ((transport == 0) ? "tcp"
-                                  : (transport == 1) ? "rdma" : "tcp,rdma"));
+        cli_out("Transport-type: %s", ((transport == 0)   ? "tcp"
+                                       : (transport == 1) ? "rdma"
+                                                          : "tcp,rdma"));
         j = 1;
 
         GF_FREE(local->get_vol.volname);
diff --git a/xlators/cluster/dht/src/dht-rebalance.c b/xlators/cluster/dht/src/dht-rebalance.c
index 3c4458c3b..7679c9794 100644
--- a/xlators/cluster/dht/src/dht-rebalance.c
+++ b/xlators/cluster/dht/src/dht-rebalance.c
@@ -2938,7 +2938,6 @@ gf_defrag_task(void *opaque)
     set_lk_owner_from_ptr(&lkowner, &lkowner);
     syncopctx_setfslkowner(&lkowner);
 
-
     q_head = &(defrag->queue[0].list);
 
     /* The following while loop will dequeue one entry from the defrag->queue
diff --git a/xlators/debug/io-stats/src/io-stats.c b/xlators/debug/io-stats/src/io-stats.c
index 0eac50117..a36336736 100644
--- a/xlators/debug/io-stats/src/io-stats.c
+++ b/xlators/debug/io-stats/src/io-stats.c
@@ -3765,10 +3765,10 @@ reconfigure(xlator_t *this, dict_t *options)
     GF_OPTION_RECONF("ios-dump-format", dump_format_str, options, str, out);
     ios_set_log_format_code(conf, dump_format_str);
     if (conf->ios_sample_interval) {
-        GF_OPTION_RECONF("ios-sample-buf-size", conf->ios_sample_buf_size, options,
-                         int32, out);
+        GF_OPTION_RECONF("ios-sample-buf-size", conf->ios_sample_buf_size,
+                         options, int32, out);
     } else {
-        ios_sample_buf_size_configure (this->name, conf);
+        ios_sample_buf_size_configure(this->name, conf);
     }
 
     GF_OPTION_RECONF("sys-log-level", sys_log_str, options, str, out);
@@ -3951,7 +3951,7 @@ init(xlator_t *this)
         GF_OPTION_INIT("ios-sample-buf-size", conf->ios_sample_buf_size, int32,
                        out);
     } else {
-        ios_sample_buf_size_configure (this->name, conf);
+        ios_sample_buf_size_configure(this->name, conf);
     }
 
     ret = ios_init_sample_buf(conf);
diff --git a/xlators/mgmt/glusterd/src/glusterd-volume-set.c b/xlators/mgmt/glusterd/src/glusterd-volume-set.c
index 5652b7e69..a9f7353f7 100644
--- a/xlators/mgmt/glusterd/src/glusterd-volume-set.c
+++ b/xlators/mgmt/glusterd/src/glusterd-volume-set.c
@@ -1460,12 +1460,10 @@ struct volopt_map_entry glusterd_volopt_map[] = {
      .voltype = "performance/md-cache",
      .option = "pass-through",
      .op_version = GD_OP_VERSION_4_1_0},
-    {
-        .key = "performance.write-behind-pass-through",
-        .voltype = "performance/write-behind",
-        .option = "pass-through",
-        .op_version = GD_OP_VERSION_9_0
-    },
+    {.key = "performance.write-behind-pass-through",
+     .voltype = "performance/write-behind",
+     .option = "pass-through",
+     .op_version = GD_OP_VERSION_9_0},
     {.key = "performance.md-cache-timeout",
      .voltype = "performance/md-cache",
      .option = "md-cache-timeout",
diff --git a/xlators/performance/write-behind/src/write-behind.c b/xlators/performance/write-behind/src/write-behind.c
index c46ffb19e..243ed511c 100644
--- a/xlators/performance/write-behind/src/write-behind.c
+++ b/xlators/performance/write-behind/src/write-behind.c
@@ -3191,15 +3191,13 @@ struct volume_options options[] = {
         .op_version = {GD_OP_VERSION_6_0},
         .flags = OPT_FLAG_SETTABLE,
     },
-    {
-        .key = {"pass-through"},
-        .type = GF_OPTION_TYPE_BOOL,
-        .default_value = "false",
-        .op_version = {GD_OP_VERSION_9_0},
-        .flags = OPT_FLAG_SETTABLE | OPT_FLAG_DOC | OPT_FLAG_CLIENT_OPT,
-        .tags = {"write-behind"},
-        .description = "Enable/disable write-behind pass-through"
-    },
+    {.key = {"pass-through"},
+     .type = GF_OPTION_TYPE_BOOL,
+     .default_value = "false",
+     .op_version = {GD_OP_VERSION_9_0},
+     .flags = OPT_FLAG_SETTABLE | OPT_FLAG_DOC | OPT_FLAG_CLIENT_OPT,
+     .tags = {"write-behind"},
+     .description = "Enable/disable write-behind pass-through"},
     {.key = {"flush-behind"},
      .type = GF_OPTION_TYPE_BOOL,
      .default_value = "on",

gluster-ant avatar Dec 23 '21 06:12 gluster-ant

/run regression

Shwetha-Acharya avatar Dec 27 '21 16:12 Shwetha-Acharya

1 test(s) failed ./tests/00-geo-rep/passive_sync.t

0 test(s) generated core

6 test(s) needed retry ./tests/000-flaky/basic_afr_split-brain-favorite-child-policy.t ./tests/000-flaky/basic_mount-nfs-auth.t ./tests/000-flaky/bugs_nfs_bug-1116503.t ./tests/000-flaky/glusterd-restart-shd-mux.t ./tests/00-geo-rep/georep-basic-dr-rsync-arbiter.t ./tests/00-geo-rep/passive_sync.t

2 flaky test(s) marked as success even though they failed ./tests/000-flaky/basic_mount-nfs-auth.t ./tests/000-flaky/bugs_nfs_bug-1116503.t https://build.gluster.org/job/gh_centos7-regression/1991/

gluster-ant avatar Dec 27 '21 17:12 gluster-ant

Thank you for your contributions. Noticed that this issue is not having any activity in last ~6 months! We are marking this issue as stale because it has not had recent activity. It will be closed in 2 weeks if no one responds with a comment here.

stale[bot] avatar Sep 20 '22 19:09 stale[bot]

Closing this issue as there was no update since my last update on issue. If this is an issue which is still valid, feel free to open it.

stale[bot] avatar Oct 16 '22 03:10 stale[bot]