glusterfs
glusterfs copied to clipboard
gluster
Files not working
We have a Gluster cluster with 15 server and total of 180 disk on centos 7 , the cluster was working fine for ling time but we had a fulty disk with one of the serves.
What we have done is to power off the server and changing the the disk that is not functioning but after we powerd it on again that server showing connected but its rejected , we followed a document below to solve the problem https://gluster-documentations.readthedocs.io/en/latest/Administrator%20Guide/Resolving%20Peer%20Rejected/ and after that the server is showing “ peer in cluster” but the total size from the mount point is showing the capacity of one server only.
We search the web for a solution and it was something about shared bricks or something like that and we just set the "cluster.min-free-disk: 11%" and that solve the problem and the size back to normal BUT not all the data is working.
Now the data is there but some of the data not working
Can you please help us to solve that problem
Below is some information about the cluster
[root@gfs2 ~]# gluster vol info
Volume Name: dvol
Type: Distributed-Disperse
Volume ID: 9123a316-7519-46e5-ba95-7aa1b9c5124b
Status: Started
Snapshot Count: 0
Number of Bricks: 30 x (4 + 2) = 180
Transport-type: tcp
Bricks:
Brick1: gfs12:/1/brick_a
Brick2: gfs8:/a1/brick
Brick3: gfs2:/1/brick
Brick4: gfs12:/2/brick_b
Brick5: gfs8:/a2/brick
Brick6: gfs2:/2/brick
Brick7: gfs12:/3/brick_c
Brick8: gfs8:/a3/brick
Brick9: gfs2:/3/brick
Brick10: gfs12:/4/brick_d
Brick11: gfs8:/a4/brick
Brick12: gfs2:/4/brick
Brick13: gfs12:/5/brick_e
Brick14: gfs8:/a5/brick
Brick15: gfs2:/5/brick
Brick16: gfs12:/6/brick_f
Brick17: gfs8:/a6/brick
Brick18: gfs2:/6/brick
Brick19: gfs12:/7/brick_g
Brick20: gfs8:/a7/brick
Brick21: gfs2:/7/brick_b
Brick22: gfs12:/8/brick_h
Brick23: gfs8:/a8/brick
Brick24: gfs2:/8/brick
Brick25: gfs12:/9/brick_i
Brick26: gfs8:/a9/brick
Brick27: gfs2:/9/brick
Brick28: gfs12:/10/brick_j
Brick29: gfs8:/a10/brick
Brick30: gfs2:/10/brick
Brick31: gfs12:/11/brick_k
Brick32: gfs8:/a11/brick
Brick33: gfs2:/11/brick
Brick34: gfs12:/12/brick_l
Brick35: gfs8:/a12/brick
Brick36: gfs2:/12/brick_b
Brick37: gfs3:/1/brick
Brick38: gfs4:/1/brick
Brick39: gfs11:/1/brick
Brick40: gfs3:/2/brick
Brick41: gfs4:/2/brick
Brick42: gfs11:/2/brick
Brick43: gfs3:/3/brick
Brick44: gfs4:/3/brick
Brick45: gfs11:/3/brick
Brick46: gfs3:/4/brick
Brick47: gfs4:/4/brick
Brick48: gfs11:/4/brick
Brick49: gfs3:/5/brick
Brick50: gfs4:/5/brick
Brick51: gfs11:/5/brick
Brick52: gfs3:/6/brick
Brick53: gfs4:/6/brick
Brick54: gfs11:/6/brick
Brick55: gfs3:/7/brick
Brick56: gfs4:/7/brick
Brick57: gfs11:/7/brick
Brick58: gfs3:/8/brick
Brick59: gfs4:/8/brick
Brick60: gfs11:/8/brick
Brick61: gfs3:/9/brick
Brick62: gfs4:/9/brick
Brick63: gfs11:/9/brick
Brick64: gfs3:/10/brick
Brick65: gfs4:/10/brick
Brick66: gfs11:/10/brick
Brick67: gfs3:/11/brick
Brick68: gfs4:/11/brick
Brick69: gfs11:/11/brick
Brick70: gfs3:/12/brick
Brick71: gfs4:/12/brick
Brick72: gfs11:/12/brick
Brick73: gfs5:/1/brick_a
Brick74: gfs5:/2/brick_b
Brick75: gfs10:/1/brick_a
Brick76: gfs10:/2/brick_b
Brick77: gfs6:/1/brick_a
Brick78: gfs6:/2/brick_b
Brick79: gfs5:/3/brick_c
Brick80: gfs5:/4/brick_d
Brick81: gfs10:/3/brick_c
Brick82: gfs10:/4/brick_d
Brick83: gfs6:/3/brick_c
Brick84: gfs6:/4/brick_d
Brick85: gfs5:/5/brick_e
Brick86: gfs5:/6/brick_f
Brick87: gfs10:/5/brick_e
Brick88: gfs10:/6/brick_f
Brick89: gfs6:/5/brick_e
Brick90: gfs6:/6/brick_f
Brick91: gfs5:/7/brick_g
Brick92: gfs5:/8/brick_h
Brick93: gfs10:/7/brick_g
Brick94: gfs10:/8/brick_h
Brick95: gfs6:/7/brick_g
Brick96: gfs6:/8/brick_h
Brick97: gfs5:/9/brick_i
Brick98: gfs5:/10/brick_j
Brick99: gfs10:/9/brick_i
Brick100: gfs10:/10/brick_j
Brick101: gfs6:/9/brick_i
Brick102: gfs6:/10/brick_j
Brick103: gfs5:/11/brick_k
Brick104: gfs5:/12/brick_l
Brick105: gfs10:/11/brick_k
Brick106: gfs10:/12/brick_l
Brick107: gfs6:/11/brick_k
Brick108: gfs6:/12/brick_l
Brick109: gfs1:/1/brick
Brick110: gfs7:/1/brick
Brick111: gfs9:/1/brick
Brick112: gfs1:/2/brick
Brick113: gfs7:/2/brick
Brick114: gfs9:/2/brick
Brick115: gfs1:/3/brick
Brick116: gfs7:/3/brick
Brick117: gfs9:/3/brick
Brick118: gfs1:/4/brick
Brick119: gfs7:/4/brick
Brick120: gfs9:/4/brick
Brick121: gfs1:/5/brick
Brick122: gfs7:/5/brick
Brick123: gfs9:/5/brick
Brick124: gfs1:/6/brick
Brick125: gfs7:/6/brick
Brick126: gfs9:/6/brick
Brick127: gfs1:/7/brick
Brick128: gfs7:/7/brick
Brick129: gfs9:/7/brick
Brick130: gfs1:/8/brick
Brick131: gfs7:/8/brick
Brick132: gfs9:/8/brick
Brick133: gfs1:/9/brick
Brick134: gfs7:/9/brick
Brick135: gfs9:/9/brick
Brick136: gfs1:/10/brick
Brick137: gfs7:/10/brick
Brick138: gfs9:/10/brick
Brick139: gfs1:/11/brick
Brick140: gfs7:/11/brick
Brick141: gfs9:/11/brick
Brick142: gfs1:/12/brick
Brick143: gfs7:/12/brick
Brick144: gfs9:/12/brick
Brick145: gfs13:/1/brick
Brick146: gfs14:/1/brick
Brick147: gfs15:/1/brick
Brick148: gfs13:/2/brick
Brick149: gfs14:/2/brick
Brick150: gfs15:/2/brick
Brick151: gfs13:/3/brick
Brick152: gfs14:/3/brick
Brick153: gfs15:/3/brick
Brick154: gfs13:/4/brick
Brick155: gfs14:/4/brick
Brick156: gfs15:/4/brick
Brick157: gfs13:/5/brick
Brick158: gfs14:/5/brick
Brick159: gfs15:/5/brick
Brick160: gfs13:/6/brick
Brick161: gfs14:/6/brick
Brick162: gfs15:/6/brick
Brick163: gfs13:/7/brick
Brick164: gfs14:/7/brick
Brick165: gfs15:/7/brick
Brick166: gfs13:/8/brick
Brick167: gfs14:/8/brick
Brick168: gfs15:/8/brick
Brick169: gfs13:/9/brick
Brick170: gfs14:/9/brick
Brick171: gfs15:/9/brick
Brick172: gfs13:/10/brick
Brick173: gfs14:/10/brick
Brick174: gfs15:/10/brick
Brick175: gfs13:/11/brick
Brick176: gfs14:/11/brick
Brick177: gfs15:/11/brick_b
Brick178: gfs13:/12/brick
Brick179: gfs14:/12/brick
Brick180: gfs15:/12/brick
Options Reconfigured:
cluster.disperse-self-heal-daemon: on
disperse.background-heals: 14
performance.client-io-threads: on
features.scrub-throttle: aggressive
performance.stat-prefetch: on
features.scrub: pause
features.bitrot: on
storage.build-pgfid: on
performance.io-thread-count: 64
performance.cache-max-file-size: 1024MB
performance.cache-size: 10GB
auth.allow: 192.168.201.*,192.168.69.120,192.168.3.*,192.168.69.*,192.168.202.*,192.168.255.66,192.168.3.*,gfs3,gfs8,gfs9,gfs4,gfs11,gfs12,gfs6,gfs5,gfs10,gfs7,gfs13,gfs14,gfs15,gfs2
performance.parallel-readdir: on
client.event-threads: 12
server.event-threads: 12
transport.address-family: inet
nfs.disable: on
cluster.min-free-disk: 11%
[root@gfs2 ~]#
[root@gfs2 ~]#
[root@gfs2 ~]#
[root@gfs2 ~]#
[root@gfs2 ~]# gluster vol status
Status of volume: dvol
Gluster process TCP Port RDMA Port Online Pid
------------------------------------------------------------------------------
Brick gfs12:/1/brick_a 49152 0 Y 2084
Brick gfs8:/a1/brick 49152 0 Y 3053
Brick gfs2:/1/brick 49152 0 Y 2064
Brick gfs12:/2/brick_b 49153 0 Y 2094
Brick gfs8:/a2/brick 49153 0 Y 3062
Brick gfs2:/2/brick 49153 0 Y 2066
Brick gfs12:/3/brick_c 49154 0 Y 2103
Brick gfs8:/a3/brick 49154 0 Y 3071
Brick gfs2:/3/brick 49154 0 Y 2082
Brick gfs12:/4/brick_d 49155 0 Y 2110
Brick gfs8:/a4/brick 49155 0 Y 3080
Brick gfs2:/4/brick 49155 0 Y 2093
Brick gfs12:/5/brick_e 49156 0 Y 2119
Brick gfs8:/a5/brick 49156 0 Y 3092
Brick gfs2:/5/brick 49156 0 Y 2101
Brick gfs12:/6/brick_f 49157 0 Y 2128
Brick gfs8:/a6/brick 49157 0 Y 3102
Brick gfs2:/6/brick 49157 0 Y 2110
Brick gfs12:/7/brick_g 49158 0 Y 2138
Brick gfs8:/a7/brick 49158 0 Y 3112
Brick gfs2:/7/brick_b 49158 0 Y 2126
Brick gfs12:/8/brick_h 49159 0 Y 2148
Brick gfs8:/a8/brick 49159 0 Y 3120
Brick gfs2:/8/brick 49159 0 Y 2131
Brick gfs12:/9/brick_i 49160 0 Y 2160
Brick gfs8:/a9/brick 49160 0 Y 3135
Brick gfs2:/9/brick 49160 0 Y 2147
Brick gfs12:/10/brick_j 49161 0 Y 2176
Brick gfs8:/a10/brick 49161 0 Y 3144
Brick gfs2:/10/brick 49161 0 Y 2166
Brick gfs12:/11/brick_k 49162 0 Y 2185
Brick gfs8:/a11/brick 49162 0 Y 3153
Brick gfs2:/11/brick 49162 0 Y 2168
Brick gfs12:/12/brick_l 49163 0 Y 2192
Brick gfs8:/a12/brick 49163 0 Y 3162
Brick gfs2:/12/brick_b 49163 0 Y 2184
Brick gfs3:/1/brick 49152 0 Y 2697
Brick gfs4:/1/brick 49152 0 Y 2411
Brick gfs11:/1/brick 49152 0 Y 2378
Brick gfs3:/2/brick 49153 0 Y 2706
Brick gfs4:/2/brick 49153 0 Y 2421
Brick gfs11:/2/brick 49153 0 Y 2387
Brick gfs3:/3/brick 49154 0 Y 2715
Brick gfs4:/3/brick 49154 0 Y 2432
Brick gfs11:/3/brick 49154 0 Y 2397
Brick gfs3:/4/brick 49155 0 Y 2724
Brick gfs4:/4/brick 49155 0 Y 2442
Brick gfs11:/4/brick 49155 0 Y 2405
Brick gfs3:/5/brick 49156 0 Y 2733
Brick gfs4:/5/brick 49156 0 Y 2453
Brick gfs11:/5/brick 49156 0 Y 2414
Brick gfs3:/6/brick 49157 0 Y 2742
Brick gfs4:/6/brick 49157 0 Y 2466
Brick gfs11:/6/brick 49157 0 Y 2423
Brick gfs3:/7/brick 49158 0 Y 2753
Brick gfs4:/7/brick 49158 0 Y 2478
Brick gfs11:/7/brick 49158 0 Y 2436
Brick gfs3:/8/brick 49159 0 Y 2764
Brick gfs4:/8/brick 49159 0 Y 2491
Brick gfs11:/8/brick 49159 0 Y 2447
Brick gfs3:/9/brick 49160 0 Y 2773
Brick gfs4:/9/brick 49160 0 Y 2503
Brick gfs11:/9/brick 49160 0 Y 2460
Brick gfs3:/10/brick 49161 0 Y 2788
Brick gfs4:/10/brick 49161 0 Y 2516
Brick gfs11:/10/brick 49161 0 Y 2467
Brick gfs3:/11/brick 49162 0 Y 2797
Brick gfs4:/11/brick 49162 0 Y 2527
Brick gfs11:/11/brick 49162 0 Y 2480
Brick gfs3:/12/brick 49163 0 Y 2808
Brick gfs4:/12/brick 49163 0 Y 2538
Brick gfs11:/12/brick 49163 0 Y 2489
Brick gfs5:/1/brick_a 49152 0 Y 2072
Brick gfs5:/2/brick_b 49153 0 Y 14369
Brick gfs10:/1/brick_a 49152 0 Y 1931
Brick gfs10:/2/brick_b 49153 0 Y 1940
Brick gfs6:/1/brick_a 49152 0 Y 2112
Brick gfs6:/2/brick_b 49153 0 Y 2121
Brick gfs5:/3/brick_c 49154 0 Y 14376
Brick gfs5:/4/brick_d 49155 0 Y 2079
Brick gfs10:/3/brick_c 49154 0 Y 1949
Brick gfs10:/4/brick_d 49155 0 Y 1959
Brick gfs6:/3/brick_c 49154 0 Y 2130
Brick gfs6:/4/brick_d 49155 0 Y 2139
Brick gfs5:/5/brick_e 49156 0 Y 2142
Brick gfs5:/6/brick_f 49157 0 Y 2088
Brick gfs10:/5/brick_e 49156 0 Y 1967
Brick gfs10:/6/brick_f 49157 0 Y 1976
Brick gfs6:/5/brick_e 49156 0 Y 2148
Brick gfs6:/6/brick_f 49157 0 Y 2157
Brick gfs5:/7/brick_g 49158 0 Y 2124
Brick gfs5:/8/brick_h 49159 0 Y 2115
Brick gfs10:/7/brick_g 49158 0 Y 1989
Brick gfs10:/8/brick_h 49159 0 Y 2000
Brick gfs6:/7/brick_g 49158 0 Y 2168
Brick gfs6:/8/brick_h 49159 0 Y 2180
Brick gfs5:/9/brick_i 49160 0 Y 2097
Brick gfs5:/10/brick_j 49161 0 Y 2106
Brick gfs10:/9/brick_i 49160 0 Y 2009
Brick gfs10:/10/brick_j 49161 0 Y 2022
Brick gfs6:/9/brick_i 49160 0 Y 2192
Brick gfs6:/10/brick_j 49161 0 Y 2203
Brick gfs5:/11/brick_k 49162 0 Y 2133
Brick gfs5:/12/brick_l 49163 0 Y 2153
Brick gfs10:/11/brick_k 49162 0 Y 2033
Brick gfs10:/12/brick_l 49163 0 Y 2044
Brick gfs6:/11/brick_k 49162 0 Y 2214
Brick gfs6:/12/brick_l 49163 0 Y 2223
Brick gfs1:/1/brick N/A N/A N N/A
Brick gfs7:/1/brick 49152 0 Y 1892
Brick gfs9:/1/brick 49152 0 Y 2595
Brick gfs1:/2/brick 49153 0 Y 2497
Brick gfs7:/2/brick 49153 0 Y 1891
Brick gfs9:/2/brick 49153 0 Y 2604
Brick gfs1:/3/brick 49154 0 Y 2537
Brick gfs7:/3/brick 49154 0 Y 1893
Brick gfs9:/3/brick 49154 0 Y 2613
Brick gfs1:/4/brick 49155 0 Y 2630
Brick gfs7:/4/brick 49155 0 Y 1916
Brick gfs9:/4/brick 49155 0 Y 2622
Brick gfs1:/5/brick 49156 0 Y 2519
Brick gfs7:/5/brick 49156 0 Y 1920
Brick gfs9:/5/brick 49156 0 Y 2631
Brick gfs1:/6/brick 49157 0 Y 2645
Brick gfs7:/6/brick 49157 0 Y 1942
Brick gfs9:/6/brick 49157 0 Y 2642
Brick gfs1:/7/brick 49158 0 Y 2551
Brick gfs7:/7/brick 49158 0 Y 1934
Brick gfs9:/7/brick 49158 0 Y 2651
Brick gfs1:/8/brick 49159 0 Y 2583
Brick gfs7:/8/brick 49159 0 Y 1958
Brick gfs9:/8/brick 49159 0 Y 2664
Brick gfs1:/9/brick 49160 0 Y 2566
Brick gfs7:/9/brick 49160 0 Y 1965
Brick gfs9:/9/brick 49160 0 Y 2675
Brick gfs1:/10/brick 49161 0 Y 2614
Brick gfs7:/10/brick 49161 0 Y 1978
Brick gfs9:/10/brick 49161 0 Y 2684
Brick gfs1:/11/brick 49162 0 Y 2598
Brick gfs7:/11/brick 49162 0 Y 1989
Brick gfs9:/11/brick 49162 0 Y 2697
Brick gfs1:/12/brick 49163 0 Y 2662
Brick gfs7:/12/brick 49163 0 Y 1996
Brick gfs9:/12/brick 49163 0 Y 2706
Brick gfs13:/1/brick 49152 0 Y 2204
Brick gfs14:/1/brick 49152 0 Y 1995
Brick gfs15:/1/brick 49152 0 Y 1985
Brick gfs13:/2/brick 49153 0 Y 2211
Brick gfs14:/2/brick 49153 0 Y 2008
Brick gfs15:/2/brick 49153 0 Y 1992
Brick gfs13:/3/brick 49154 0 Y 2220
Brick gfs14:/3/brick 49154 0 Y 2017
Brick gfs15:/3/brick 49154 0 Y 2002
Brick gfs13:/4/brick 49155 0 Y 2229
Brick gfs14:/4/brick 49155 0 Y 2018
Brick gfs15:/4/brick 49155 0 Y 2010
Brick gfs13:/5/brick 49156 0 Y 2238
Brick gfs14:/5/brick 49156 0 Y 2032
Brick gfs15:/5/brick 49156 0 Y 2019
Brick gfs13:/6/brick 49157 0 Y 2249
Brick gfs14:/6/brick 49157 0 Y 2045
Brick gfs15:/6/brick 49157 0 Y 2028
Brick gfs13:/7/brick 49158 0 Y 2260
Brick gfs14:/7/brick 49158 0 Y 2055
Brick gfs15:/7/brick 49158 0 Y 2041
Brick gfs13:/8/brick 49159 0 Y 2271
Brick gfs14:/8/brick 49159 0 Y 2059
Brick gfs15:/8/brick 49159 0 Y 2052
Brick gfs13:/9/brick 49160 0 Y 2282
Brick gfs14:/9/brick 49160 0 Y 2066
Brick gfs15:/9/brick 49160 0 Y 2065
Brick gfs13:/10/brick 49161 0 Y 2295
Brick gfs14:/10/brick 49161 0 Y 2082
Brick gfs15:/10/brick 49161 0 Y 2076
Brick gfs13:/11/brick 49162 0 Y 2304
Brick gfs14:/11/brick 49162 0 Y 2093
Brick gfs15:/11/brick_b 49162 0 Y 2087
Brick gfs13:/12/brick 49163 0 Y 2315
Brick gfs14:/12/brick 49163 0 Y 2110
Brick gfs15:/12/brick 49163 0 Y 2094
Self-heal Daemon on localhost N/A N/A Y 2942
Bitrot Daemon on localhost N/A N/A Y 15263
Scrubber Daemon on localhost N/A N/A Y 15296
Self-heal Daemon on gfs3 N/A N/A Y 3713
Bitrot Daemon on gfs3 N/A N/A Y 20369
Scrubber Daemon on gfs3 N/A N/A Y 20396
Self-heal Daemon on gfs4 N/A N/A Y 3478
Bitrot Daemon on gfs4 N/A N/A Y 2879
Scrubber Daemon on gfs4 N/A N/A Y 3212
Self-heal Daemon on gfs6 N/A N/A Y 2343
Bitrot Daemon on gfs6 N/A N/A Y 4921
Scrubber Daemon on gfs6 N/A N/A Y 4950
Self-heal Daemon on gfs1 N/A N/A Y 8211
Bitrot Daemon on gfs1 N/A N/A Y 7618
Scrubber Daemon on gfs1 N/A N/A Y 7931
Self-heal Daemon on gfs10 N/A N/A Y 2245
Bitrot Daemon on gfs10 N/A N/A Y 31966
Scrubber Daemon on gfs10 N/A N/A Y 31995
Self-heal Daemon on gfs7 N/A N/A Y 2217
Bitrot Daemon on gfs7 N/A N/A Y 18184
Scrubber Daemon on gfs7 N/A N/A Y 18211
Self-heal Daemon on gfs9 N/A N/A Y 2836
Bitrot Daemon on gfs9 N/A N/A Y 15120
Scrubber Daemon on gfs9 N/A N/A Y 15148
Self-heal Daemon on gfs11 N/A N/A Y 2724
Bitrot Daemon on gfs11 N/A N/A Y 28425
Scrubber Daemon on gfs11 N/A N/A Y 28452
Self-heal Daemon on gfs12 N/A N/A Y 2414
Bitrot Daemon on gfs12 N/A N/A Y 5307
Scrubber Daemon on gfs12 N/A N/A Y 5334
Self-heal Daemon on gfs15 N/A N/A Y 32732
Bitrot Daemon on gfs15 N/A N/A Y 1229
Scrubber Daemon on gfs15 N/A N/A Y 1248
Self-heal Daemon on gfs13 N/A N/A Y 2354
Bitrot Daemon on gfs13 N/A N/A Y 19655
Scrubber Daemon on gfs13 N/A N/A Y 19674
Self-heal Daemon on gfs8 N/A N/A Y 3217
Bitrot Daemon on gfs8 N/A N/A Y 3183
Scrubber Daemon on gfs8 N/A N/A Y 3199
Self-heal Daemon on gfs14 N/A N/A Y 2332
Bitrot Daemon on gfs14 N/A N/A Y 18309
Scrubber Daemon on gfs14 N/A N/A Y 18328
Self-heal Daemon on gfs5 N/A N/A Y 14494
Bitrot Daemon on gfs5 N/A N/A Y 4934
Scrubber Daemon on gfs5 N/A N/A Y 4961
Task Status of Volume dvol
------------------------------------------------------------------------------
Task : Rebalance
ID : a91079d3-54e6-4bd5-96d2-a9266b96bb27
Status : completed
