glusterfs icon indicating copy to clipboard operation
glusterfs copied to clipboard

No documented procedure to replace a failed brick when brick path doesn't change

Open handrea2009 opened this issue 1 year ago • 1 comments

Description of problem:

Scenario: I have a distributed replicated volume: replica3, 3 nodes, 1 brick per node, brick is XFS on a disks). One disk on a node failed, then the disk has been replaced, XFS created and mounted on same mount-point (so same path).

How I can make gluster to bring the brick back online and heal?

The documentation mention only how to replace that brick with another one with different path (so different mount-point)

https://docs.gluster.org/en/main/Administrator-Guide/Managing-Volumes/#replace-faulty-brick

If I use the "gluster volume replace-brick ...." command using same path for the SOURCE-BRICK and NEW-BRICK fail and returns an odd message:

gluster volume replace-brick read-cache cluster-node1:/brick_read_cache cluster-node1:/brick_read_cache commit force volume replace-brick: failed: Brick: cluster-node1:/brick_read_cache not available. Brick may be containing or be contained by an existing brick.

It seems the following sequence of commands achieve it, but I can't find any documentation about that and what exactly they do:

gluster volume reset-brick volume node1:/brick_read_cache start gluster volume reset-brick volume node1:/brick_read_cache node1:/brick_read_cache commit

Mandatory info:

`gluster volume status read-cache Status of volume: read-cache Gluster process TCP Port RDMA Port Online Pid

Brick cluster-node1:/brick_sdc 60057 0 Y 1818449 Brick cluster-node2:/brick_sdc 54042 0 Y 2153842 Brick cluster-node3:/brick_sdc 60074 0 Y 2616705 Brick cluster-node1:/brick_sde N/A N/A N N/A Brick cluster-node2:/brick_sde 56469 0 Y 2153857 Brick cluster-node3:/brick_sde 60156 0 Y 2616720 Self-heal Daemon on localhost N/A N/A Y 1818304 Quota Daemon on localhost N/A N/A Y 1818500 Self-heal Daemon on cluster-node3 N/A N/A Y 2616652 Quota Daemon on cluster-node3 N/A N/A Y 2616748 Self-heal Daemon on cluster-node2 N/A N/A Y 2153760 Quota Daemon on cluster-node2 N/A N/A Y 2153887 `

- The operating system / glusterfs version:

`uname -a Linux cluster-node1 4.18.0-513.9.1.el8_9.x86_64 #1 SMP Wed Nov 29 18:55:19 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux

gluster --version glusterfs 10.5 Repository revision: git://git.gluster.org/glusterfs.git Copyright (c) 2006-2016 Red Hat, Inc. https://www.gluster.org/ GlusterFS comes with ABSOLUTELY NO WARRANTY. It is licensed to you under your choice of the GNU Lesser General Public License, version 3 or any later version (LGPLv3 or later), or the GNU General Public License, version 2 (GPLv2), in all cases as published by the Free Software Foundation. `

handrea2009 avatar Dec 15 '23 15:12 handrea2009

Looks like missing documentation. Refer this doc about Reset-brick command: https://access.redhat.com/documentation/en-us/red_hat_gluster_storage/3.5/html/administration_guide/sect-migrating_volumes#sect-Migrating_Volumes-Reconfigure_Brick

aravindavk avatar Dec 15 '23 16:12 aravindavk