glusterfs
glusterfs copied to clipboard
Published
20 hours ago •
gluster
gluster
rebalance doing nothing on massively disbalanced distributed volume
Description of problem:
Gluster 9.1-2
A distributed volume has 2 bricks:
------------------------------------------------------------------------------
Brick : Brick 267a47b1-365f-4221-915f-5eadabf90a4c-StoreInt2:/storages/zfs/zfs-vmp2
TCP Port : 49152
RDMA Port : 0
Online : Y
Pid : 4132252
File System : zfs
Device : zfs-vmp2
Mount Options : rw,xattr,noacl
Inode Size : N/A
Disk Space Free : 863.7GB
Total Disk Space : 1.1TB
Inode Count : 1811347424
Free Inodes : 1811346832
------------------------------------------------------------------------------
Brick : Brick 7cd6a976-e2dc-4f56-9e03-a8d0cf409595-StoreInt1:/storages/zfs/zfs-server1
TCP Port : 49152
RDMA Port : 0
Online : Y
Pid : 4104490
File System : zfs
Device : zfs-server1
Mount Options : rw,xattr,noacl
Inode Size : N/A
Disk Space Free : 57.1GB
Total Disk Space : 1.6TB
Inode Count : 119786215
Free Inodes : 119785392
These bricks were created simultaiously. so no fix-layout is needed. (ps: I did fix-layout too but no effect) The vast majority of data is on the big brick. Maybe these were thin images so in the very beginning they were small and placing them on bigger brick was proper decision. But now its size has grown and part of data needs to be migrated to free brick.
Small brick:
-rw-r--r-- 2 node node 53918367744 Apr 9 20:03 0f9d4fe5-5ec3-4240-a81f-61a78cfd72e6
-rw-r--r-- 2 node node 298 May 20 00:48 13ebf333-fa32-4a9f-bf61-a7ed296ef4a2
-rw-r--r-- 2 node node 63504384 May 12 17:06 15e3bf80-6395-4318-8408-2c227c8af5fe
-rw-r--r-- 2 node node 55394172928 Apr 9 20:02 1ea204a4-c484-4ef6-8958-263aa9fe73b2
-rw-r--r-- 2 node node 302 May 7 17:15 21630fe0-6357-4faf-b70d-0b1f10f205e1
-rw-r--r-- 2 node node 8049000448 May 12 17:01 2967cd1b-1433-42cf-979f-a93c29cf0443
-rw-r--r-- 2 node node 1261371392 Sep 2 2021 2c0922d5-2954-4bfc-a640-f4527a8b380f
-rw-r--r-- 2 node node 416 Aug 29 2021 2e05a842-0334-4be4-bfde-63cf323ffa55
-rw-r--r-- 2 node node 8763998208 Jun 22 14:08 4190fa42-02f9-4145-9e48-d701831c0afe
-rw-r--r-- 2 node node 298 Dec 9 2021 4477bd5b-f424-4cbc-93e1-d16c61151203
-rw-r--r-- 2 node node 8012038144 May 26 19:09 5b16411e-8182-49f4-9957-d98607d7d67f
-rw-r--r-- 2 node node 302 Jun 3 01:02 5b31f42f-6082-4a11-b573-8ae83b4c9e87
-rw-r--r-- 2 node node 4657905664 Jun 16 16:12 60477a47-afae-4960-aaee-adec16f525f7
-rw-r--r-- 2 node node 21295661056 Jun 22 19:01 6896a7ab-c786-4b83-8134-dfd5a0751d0e
-rw-r--r-- 2 node node 304 Sep 28 2021 6f9f7e13-f3e8-4724-90ab-b2d3954db5cd
-rw-r--r-- 2 node node 1691877376 May 12 14:21 7028b671-4a83-403f-8b4d-c0c04d637912
-rw-r--r-- 2 node node 11286675456 Aug 29 2021 70685485-dfc9-4980-ae34-03b9436519eb
-rw-r--r-- 2 node node 302 Jan 30 19:30 863fd112-4670-4646-8ec8-9411e9123d16
-rw-r--r-- 2 node node 299 Apr 3 2021 8ac2d377-ee4e-4ff3-8d50-b762286a9fc7
-rw-r--r-- 2 node node 1718616064 May 11 18:56 8dd5ca5a-1a29-4247-9566-89dff873c54b
-rw-r--r-- 2 node node 197248 Apr 15 13:26 9d246ed3-c54a-4a21-8e55-402e20376d83
-rw-r--r-- 2 node node 7479361536 May 26 19:09 a419b0d8-c99f-4d6f-97d7-c215190c3a90
-rw-r--r-- 2 node node 8030257152 May 26 19:09 a59e7894-a47e-4153-b8c3-6eec802bd8ee
-rw-r--r-- 2 node node 298 May 20 00:48 a5cbac3a-0e10-426b-8d97-0a8be947370c
-rw-r--r-- 2 node node 1481900032 Sep 8 2021 abe6361d-016f-4ea3-b3cb-3b538f278504
-rw-r--r-- 2 node node 6736183296 Dec 14 2021 b2453701-2cec-4401-b4a3-7fe5a94e9323
-rw-r--r-- 2 node node 302 Jan 30 22:59 bc63d3f8-ecca-47a6-b076-34bef5fe7d3e
-rw-r--r-- 2 node node 1264189440 Dec 14 2021 bd2cb731-1faa-4841-9e01-d5c50acac763
-rw-r--r-- 2 node node 7162298368 May 12 17:01 be9e298a-0b9b-41c9-8f8a-b6f70b15f8a0
-rw-r--r-- 2 node node 0 Jun 22 19:01 cb5cd3f6-1ee5-4791-95ea-af8b853ecb85
-rw-r--r-- 2 node node 15331688448 Apr 9 20:01 d361dc3e-81bd-4785-bccf-885dff0c424b
-rw-r--r-- 2 node node 42084352 Mar 21 19:42 d9d16227-e19f-441b-a21d-450ccc30954d
-rw-r--r-- 2 node node 298 May 20 00:48 dc5246f3-ad77-4ebf-98c0-f5002939cfd7
-rw-r--r-- 2 node node 302 Jan 30 23:00 ed9daf86-117b-4e41-8501-1537f4717689
-rw-r--r-- 2 node node 0 Jun 22 19:01 ee2350e1-2b79-497d-add0-f5e6ec89f227
-rw-r--r-- 2 node node 302 Jun 3 01:02 fb5481a4-9ed8-4c07-abec-819347171984
Big brick:
[root@7cd6a976-e2dc-4f56-9e03-a8d0cf409595 ~]# ls -l /storages/zfs/zfs-server1/.glusterfs/*/*/ | grep node
-rw-r--r-- 2 node node 1935555072 Apr 3 2021 0239fef6-f64f-4def-a4fc-def71488a529
-rw-r--r-- 2 node node 307 Aug 29 2021 0903a9b5-60c2-479c-a755-097f69344ba6
-rwxr-xr-x 2 node node 21478375424 Aug 27 2021 11055a0c-79d4-4f84-8b83-f22a064b34e3
-rw-r--r-- 2 node node 13275561984 Jun 12 03:04 20b05dad-fbd9-4ef8-ba44-f9d63e65002a
-rw-r--r-- 2 node node 2585231360 Mar 21 21:34 289fca3d-7317-4a26-855e-e64a1f114f7a
-rw-r--r-- 2 node node 40930902016 Jun 15 21:21 2d54f016-eec5-4dab-829e-67e57dfe5f26
-rw-r--r-- 2 node node 1297874944 Dec 14 2021 322b1d73-12fd-4793-8b23-029c73018606
-rw-r--r-- 2 node node 298 Dec 9 2021 33b61d50-5926-482e-b44e-2d5129bb2459
-rw-r--r-- 2 node node 1580218880 Apr 3 2021 3568ad1c-8eca-4e32-809c-0a1829cb0d9f
-rw-r--r-- 2 node node 116273184768 Jun 15 21:21 35fca4fc-529a-4399-9006-166f2b2a1250
-rw-r--r-- 2 node node 2332753920 May 7 17:07 36f686e8-fee2-4961-8af5-8abdb03a46a2
-rw-r--r-- 2 node node 300 May 7 17:09 3d10b678-fbfe-4057-83a7-fd610b283c41
-rw-r--r-- 2 node node 4652531712 Apr 9 19:11 3f48e767-753c-45db-a8dc-f66d00c10b05
-rw-r--r-- 2 node node 6335692800 Dec 14 2021 4102222b-09a4-4788-b0e0-5cbe93a30021
-rw-r--r-- 2 node node 160013418496 May 26 19:10 411cdca9-a87d-4c02-9a75-b5f1f37d45bc
-rw-r--r-- 2 node node 197248 Apr 15 13:27 4145d126-2486-426b-b993-9b53ebb97e67
-rw-r--r-- 2 node node 7935557632 Apr 9 19:13 428d8b85-ec00-4ea2-82de-ca856784ad7c
-rw-r--r-- 2 node node 302 Jan 30 19:30 4495c527-527b-4f17-b928-faa3ce2fb632
-rw-r--r-- 2 node node 4752513024 Mar 14 18:29 45090653-60ef-4ce4-9d4e-f984338d9e42
-rw-r--r-- 2 node node 160013418496 May 26 19:10 47291db1-4d45-465f-9b40-85b63b41e285
-rw-r--r-- 2 node node 154134577152 May 26 19:09 55e42c44-c925-4232-9947-aa0045749549
-rw-r--r-- 2 node node 39849230336 Jun 15 21:21 57d37631-cfc7-43fb-b405-2dd735ad8042
-rw-r--r-- 2 node node 22980657152 Jun 2 13:00 595d0537-ff7a-4076-a1ff-87ac00cd6db3
-rw-r--r-- 2 node node 302 Jan 30 22:59 5f694b34-f91d-4ffd-bcc0-6826b5c56451
-rw-r--r-- 2 node node 12751863808 Apr 9 20:02 67bda769-e845-4f9e-99d1-ee4fb9f32faf
-rw-r--r-- 2 node node 6632570880 Dec 14 2021 69871208-7d7d-45de-8641-9a563d24e83e
-rw-r--r-- 2 node node 131268608 May 12 17:06 6afdfc98-f241-4d4a-8f75-aca289165365
-rw-r--r-- 2 node node 17373331456 Sep 29 2021 6cd55daf-3d4d-4451-8258-50b7dea5c564
-rw-r--r-- 2 node node 40021131264 Jun 15 21:21 711b5259-fb93-48c1-bd63-9c73dd45abc0
-rw-r--r-- 2 node node 6267207680 Dec 14 2021 72aa10da-48d4-49da-ba4b-99da25d36486
-rw-r--r-- 2 node node 36536123392 Mar 17 14:38 77ecb78e-dbdf-4bbf-aae2-3b752e8fe2a0
-rw-r--r-- 2 node node 54661480448 Apr 9 20:01 79823dd6-6637-4857-87ac-24561e689d16
-rw-r--r-- 2 node node 197248 May 11 13:03 807d344b-04aa-4185-9a43-cbecd6551dcb
-rw-r--r-- 2 node node 398 Dec 9 2021 822ae025-aa4f-48b9-a91d-be11ded41964
-rw-r--r-- 2 node node 1822564864 Apr 3 2021 82e9045f-42bb-44ea-85c0-e02f5612b73d
-rw-r--r-- 2 node node 197248 May 11 11:46 83aeaadb-5e0b-429c-9c7d-41c2b5864642
-rw-r--r-- 2 node node 160013484032 May 26 19:09 84259100-0bc2-423f-a759-97c0b30f5145
-rw-r--r-- 2 node node 3050024073 Apr 9 20:02 85f055d5-bd34-4e6d-a1ba-82c01cfa7ed7
-rw-r--r-- 2 node node 74107846656 May 27 18:59 8edf305b-cd87-43cf-b5ad-34f318b158da
-rw-r--r-- 2 node node 23237754880 May 26 19:10 90141f80-e356-4905-940a-2eb8d9ceaef3
-rw-r--r-- 2 node node 3554213888 May 12 17:01 9360280e-2ffd-44b1-890c-738e15bee246
-rw-r--r-- 2 node node 158052909056 May 27 13:05 93dce628-3e34-435a-8dc4-d02e2ee10ff7
-rw-r--r-- 2 node node 6651677696 Sep 14 2021 95e0f403-bfc1-475e-a270-b6632bd9f812
-rw-r--r-- 2 node node 8763080704 Apr 9 19:13 96f4dd38-24ac-472c-80b5-a5b1897788de
-rw-r--r-- 2 node node 3106406400 Oct 8 2021 9d7951b2-c23f-4022-9dde-71b7f70495b2
-rw-r--r-- 2 node node 498321408 Aug 29 2021 a8e098cc-29b8-404d-90f0-a15befaba5ea
-rw-r--r-- 2 node node 302 Jan 30 19:28 b067b05a-d726-4e3d-8599-092ccfedfe60
-rw-r--r-- 2 node node 302 Sep 2 2021 b1583040-4f5d-42a7-901a-d6ef3357d16c
-rw-r--r-- 2 node node 13366657024 Apr 9 20:03 b54c5144-7525-44f8-b57a-6305d33bc2cc
-rw-r--r-- 2 node node 302 May 7 17:15 b56c8350-c0ad-4739-b3c6-cc537af0f75b
-rw-r--r-- 2 node node 302 Jun 3 01:03 b726869e-361b-426b-84b0-0d46362c769f
-rw-r--r-- 2 node node 15017771008 Jun 15 21:21 b7a170bf-7990-4dc1-888d-3a228ab7214a
-rwxr-xr-x 2 node node 21478375424 Jun 22 19:05 b7bda031-8950-48a2-bfd0-73d4859740a8
-rw-r--r-- 2 node node 2944201832 Apr 9 20:03 bc6696d0-e753-4240-a462-a6760afd9253
-rw-r--r-- 2 node node 6550323200 Aug 29 2021 c4de665d-a304-40ec-81c5-595c69c3228f
-rw-r--r-- 2 node node 13495173120 Jun 12 03:04 c61cdad0-cc86-48d2-95c3-e8c87adc1c1d
-rw-r--r-- 2 node node 3102016272 Apr 9 20:01 c74a4c1c-d243-4a45-a18c-5149fdc14188
-rw-r--r-- 2 node node 4148953088 Dec 14 2021 d4267001-6a3e-4bb0-b3af-84c5d96f2147
-rw-r--r-- 2 node node 8107720704 Dec 14 2021 d4fa0a32-06a2-46fd-8f3d-abe5a5e456d5
-rw-r--r-- 2 node node 6550333440 May 30 19:48 e2072041-43b5-4daf-9dd8-ebc6f9b64bd2
-rw-r--r-- 2 node node 18350080000 May 26 19:10 e3884c97-852d-47f8-8ac8-a47105ad2e83
-rw-r--r-- 2 node node 300 May 7 17:57 ee7de43f-c569-46a4-a692-d014efd1b8f3
-rw-r--r-- 2 node node 1977503744 Apr 3 2021 f087101f-44d9-4b09-bf15-261a833c0a4d
-rw-r--r-- 2 node node 2593411072 Apr 3 2021 f256a6d7-c078-40c7-a2ff-7830db9e2030
-rw-r--r-- 2 node node 8961785856 May 12 17:01 f52f7c76-3fb9-49b9-8ede-d35d94fb3128
-rw-r--r-- 2 node node 295 Mar 15 18:47 fa42eeab-55cd-4883-8ba8-036dded66d7e
-rw-r--r-- 2 node node 160012566528 May 26 19:09 fe0898ad-ffc1-4d01-882b-baf2d6e419b3
-rw-r--r-- 2 node node 2100960256 Apr 3 2021 fe7f3baa-df81-40a2-8e73-cb026a74f26f
The exact command to reproduce the issue: gluster volume rebalance start or gluster volume rebalance start force
The full output of the command that failed:
We see a number of failures, what it is? Log file shows no errors (see below)
[root@7cd6a976-e2dc-4f56-9e03-a8d0cf409595 ~]# gluster volume rebalance DS status
Node Rebalanced-files size scanned failures skipped status run time in h:m:s
--------- ----------- ----------- ----------- ----------- ----------- ------------ --------------
267a47b1-365f-4221-915f-5eadabf90a4c-StoreInt2 0 0Bytes 37 0 0 completed 0:00:00
localhost 0 0Bytes 68 0 0 completed 0:00:00
volume rebalance: DS: success
Expected results:
files are distributed on bricks more evenly
Mandatory info:
- The output of the gluster volume info command:
[root@7cd6a976-e2dc-4f56-9e03-a8d0cf409595 ~]# gluster volume info
Volume Name: DS Type: Distribute Volume ID: 4ca74cda-0416-4251-895c-68aa9a227f45 Status: Started Snapshot Count: 0 Number of Bricks: 2 Transport-type: tcp Bricks: Brick1: 267a47b1-365f-4221-915f-5eadabf90a4c-StoreInt2:/storages/zfs/zfs-vmp2 Brick2: 7cd6a976-e2dc-4f56-9e03-a8d0cf409595-StoreInt1:/storages/zfs/zfs-server1 Options Reconfigured: cluster.min-free-disk: 10% cluster.lookup-optimize: off server.keepalive-count: 5 server.keepalive-interval: 2 server.keepalive-time: 10 server.tcp-user-timeout: 20 network.ping-timeout: 20 performance.client-io-threads: on server.event-threads: 4 client.event-threads: 4 cluster.choose-local: off user.cifs: off cluster.locking-scheme: granular cluster.data-self-heal-algorithm: full cluster.server-quorum-type: none cluster.quorum-type: none cluster.eager-lock: enable performance.strict-o-direct: on network.remote-dio: disable performance.low-prio-threads: 32 performance.io-cache: off performance.read-ahead: off performance.quick-read: off features.shard: off storage.fips-mode-rchecksum: on transport.address-family: inet nfs.disable: on
- The output of the gluster volume status command:
Status of volume: DS
Gluster process TCP Port RDMA Port Online Pid
------------------------------------------------------------------------------
Brick 267a47b1-365f-4221-915f-5eadabf90a4c-
StoreInt2:/storages/zfs/zfs-vmp2 49152 0 Y 4132252
Brick 7cd6a976-e2dc-4f56-9e03-a8d0cf409595-
StoreInt1:/storages/zfs/zfs-server1 49152 0 Y 4104490
Task Status of Volume DS
------------------------------------------------------------------------------
Task : Rebalance
ID : 0b679142-1c74-4e1f-a853-c2e5e9ddb582
Status : completed
- The output of the gluster volume heal command:
no redundancy so no heal
**- Provide logs present on following locations of client and server nodes - /var/log/glusterfs/
Here is rebalance log.
[2022-06-22 15:53:05.536262 +0000] I [MSGID: 100030] [glusterfsd.c:2685:main] 0-/usr/sbin/glusterfs: Started running version [{arg=/usr/sbin/glusterfs}, {version=9.2}, {cmdlinestr=/usr/sbin/glusterfs -s localhost --volfile-id rebalance/DS --xlator-option *dht.use-readdirp=yes --xlator-option *dht.lookup-unhashed=yes --xlator-option *dht.assert-no-child-down=yes --xlator-option *dht.readdir-optimize=on --process-name rebalance --xlator-option *dht.rebalance-cmd=1 --xlator-option *dht.node-uuid=3b6fae85-0b22-48aa-91e0-4645856254c0 --xlator-option *dht.commit-hash=362403554 --socket-file /var/run/gluster/36ba4d64e05b0648.socket --pid-file /var/lib/glusterd/vols/DS/rebalance/3b6fae85-0b22-48aa-91e0-4645856254c0.pid -l /var/log/glusterfs/DS-rebalance.log}]
[2022-06-22 15:53:05.537132 +0000] I [glusterfsd.c:2421:daemonize] 0-glusterfs: Pid of current running process is 559172
[2022-06-22 15:53:05.542719 +0000] I [socket.c:928:__socket_server_bind] 0-socket.glusterfsd: closing (AF_UNIX) reuse check socket 9
[2022-06-22 15:53:05.544637 +0000] I [MSGID: 101190] [event-epoll.c:669:event_dispatch_epoll_worker] 0-epoll: Started thread with index [{index=0}]
[2022-06-22 15:53:05.544760 +0000] I [MSGID: 101190] [event-epoll.c:669:event_dispatch_epoll_worker] 0-epoll: Started thread with index [{index=1}]
[2022-06-22 15:53:10.542013 +0000] I [glusterfsd-mgmt.c:2170:mgmt_getspec_cbk] 0-glusterfs: Received list of available volfile servers: 267a47b1-365f-4221-915f-5eadabf90a4c-StoreInt2:24007
[2022-06-22 15:53:10.544253 +0000] I [io-stats.c:3706:ios_sample_buf_size_configure] 0-DS: Configure ios_sample_buf size is 1024 because ios_sample_interval is 0
[2022-06-22 15:53:10.544385 +0000] I [MSGID: 109102] [dht-shared.c:646:dht_init] 0-DS-dht: dht_init using commit hash 362403554
[2022-06-22 15:53:10.544823 +0000] I [MSGID: 101190] [event-epoll.c:669:event_dispatch_epoll_worker] 0-epoll: Started thread with index [{index=2}]
[2022-06-22 15:53:10.544922 +0000] I [MSGID: 101190] [event-epoll.c:669:event_dispatch_epoll_worker] 0-epoll: Started thread with index [{index=3}]
[2022-06-22 15:53:10.545637 +0000] I [MSGID: 114020] [client.c:2319:notify] 0-DS-client-0: parent translators are ready, attempting connect on transport []
[2022-06-22 15:53:10.546029 +0000] I [MSGID: 114020] [client.c:2319:notify] 0-DS-client-1: parent translators are ready, attempting connect on transport []
Final graph:
+------------------------------------------------------------------------------+
1: volume DS-client-0
2: type protocol/client
3: option ping-timeout 20
4: option remote-host 267a47b1-365f-4221-915f-5eadabf90a4c-StoreInt2
5: option remote-subvolume /storages/zfs/zfs-vmp2
6: option transport-type socket
7: option transport.address-family inet
8: option username 573bceab-02c5-4a15-8839-d37f5501cdd5
9: option password f5058879-24cc-4c05-8aa0-ff1ae9529c5c
10: option transport.socket.ssl-enabled off
11: option filter-O_DIRECT disable
12: option event-threads 4
13: option transport.tcp-user-timeout 0
14: option transport.socket.keepalive-time 20
15: option transport.socket.keepalive-interval 2
16: option transport.socket.keepalive-count 9
17: option strict-locks off
18: end-volume
19:
20: volume DS-client-1
21: type protocol/client
22: option ping-timeout 20
23: option remote-host 7cd6a976-e2dc-4f56-9e03-a8d0cf409595-StoreInt1
24: option remote-subvolume /storages/zfs/zfs-server1
25: option transport-type socket
26: option transport.address-family inet
27: option username 573bceab-02c5-4a15-8839-d37f5501cdd5
28: option password f5058879-24cc-4c05-8aa0-ff1ae9529c5c
29: option transport.socket.ssl-enabled off
30: option filter-O_DIRECT disable
31: option event-threads 4
32: option transport.tcp-user-timeout 0
33: option transport.socket.keepalive-time 20
34: option transport.socket.keepalive-interval 2
35: option transport.socket.keepalive-count 9
36: option strict-locks off
37: end-volume
38:
39: volume DS-dht
40: type cluster/distribute
41: option use-readdirp yes
42: option lookup-unhashed yes
43: option assert-no-child-down yes
44: option readdir-optimize on
45: option rebalance-cmd 1
46: option node-uuid 3b6fae85-0b22-48aa-91e0-4645856254c0
47: option commit-hash 362403554
48: option lookup-optimize off
49: option min-free-disk 1%
50: option lock-migration off
51: option force-migration off
52: subvolumes DS-client-0 DS-client-1
53: end-volume
54:
55: volume DS
56: type debug/io-stats
57: option log-level INFO
58: option threads 16
59: option latency-measurement off
60: option count-fop-hits off
61: option global-threading off
62: subvolumes DS-dht
63: end-volume
[2022-06-22 15:53:10.546680 +0000] I [rpc-clnt.c:1967:rpc_clnt_reconfig] 0-DS-client-1: changing port to 49152 (from 0)
[2022-06-22 15:53:10.546709 +0000] I [rpc-clnt.c:1967:rpc_clnt_reconfig] 0-DS-client-0: changing port to 49152 (from 0)
64:
[2022-06-22 15:53:10.546763 +0000] I [socket.c:848:__socket_shutdown] 0-DS-client-0: intentional socket shutdown(12)
+------------------------------------------------------------------------------+
[2022-06-22 15:53:10.546745 +0000] I [socket.c:848:__socket_shutdown] 0-DS-client-1: intentional socket shutdown(13)
[2022-06-22 15:53:10.547449 +0000] I [MSGID: 114057] [client-handshake.c:1126:select_server_supported_programs] 0-DS-client-0: Using Program [{Program-name=GlusterFS 4.x v1}, {Num=1298437}, {Version=400}]
[2022-06-22 15:53:10.547496 +0000] I [MSGID: 114057] [client-handshake.c:1126:select_server_supported_programs] 0-DS-client-1: Using Program [{Program-name=GlusterFS 4.x v1}, {Num=1298437}, {Version=400}]
[2022-06-22 15:53:10.548104 +0000] I [MSGID: 114046] [client-handshake.c:855:client_setvolume_cbk] 0-DS-client-0: Connected, attached to remote volume [{conn-name=DS-client-0}, {remote_subvol=/storages/zfs/zfs-vmp2}]
[2022-06-22 15:53:10.548373 +0000] I [MSGID: 114046] [client-handshake.c:855:client_setvolume_cbk] 0-DS-client-1: Connected, attached to remote volume [{conn-name=DS-client-1}, {remote_subvol=/storages/zfs/zfs-server1}]
[2022-06-22 15:53:10.551882 +0000] I [dht-rebalance.c:4327:gf_defrag_start_crawl] 0-DS-dht: gf_defrag_start_crawl using commit hash 362403554
[2022-06-22 15:53:10.552504 +0000] I [MSGID: 109079] [dht-common.c:5929:dht_setxattr] 0-DS-dht: fixing the layout of /
[2022-06-22 15:53:10.554766 +0000] I [MSGID: 0] [dht-rebalance.c:3962:dht_init_local_subvols_and_nodeuuids] 0-DS-dht: local subvol: DS-client-1
[2022-06-22 15:53:10.554802 +0000] I [MSGID: 0] [dht-rebalance.c:3969:dht_init_local_subvols_and_nodeuuids] 0-DS-dht: node uuid : 3b6fae85-0b22-48aa-91e0-4645856254c0
[2022-06-22 15:53:10.555853 +0000] I [dht-rebalance.c:4182:gf_defrag_parallel_migration_init] 0-DHT: Thread[0] creation successful
[2022-06-22 15:53:10.555933 +0000] I [dht-rebalance.c:4182:gf_defrag_parallel_migration_init] 0-DHT: Thread[1] creation successful
[2022-06-22 15:53:10.556001 +0000] I [dht-rebalance.c:4182:gf_defrag_parallel_migration_init] 0-DHT: Thread[2] creation successful
[2022-06-22 15:53:10.556064 +0000] I [dht-rebalance.c:4182:gf_defrag_parallel_migration_init] 0-DHT: Thread[3] creation successful
[2022-06-22 15:53:10.556131 +0000] I [dht-rebalance.c:4182:gf_defrag_parallel_migration_init] 0-DHT: Thread[4] creation successful
[2022-06-22 15:53:10.556198 +0000] I [dht-rebalance.c:4182:gf_defrag_parallel_migration_init] 0-DHT: Thread[5] creation successful
[2022-06-22 15:53:10.556268 +0000] I [dht-rebalance.c:4182:gf_defrag_parallel_migration_init] 0-DHT: Thread[6] creation successful
[2022-06-22 15:53:10.556347 +0000] I [dht-rebalance.c:4182:gf_defrag_parallel_migration_init] 0-DHT: Thread[7] creation successful
[2022-06-22 15:53:10.556416 +0000] I [dht-rebalance.c:4182:gf_defrag_parallel_migration_init] 0-DHT: Thread[8] creation successful
[2022-06-22 15:53:10.556483 +0000] I [dht-rebalance.c:4182:gf_defrag_parallel_migration_init] 0-DHT: Thread[9] creation successful
[2022-06-22 15:53:10.556551 +0000] I [dht-rebalance.c:4182:gf_defrag_parallel_migration_init] 0-DHT: Thread[10] creation successful
[2022-06-22 15:53:10.556613 +0000] I [dht-rebalance.c:4182:gf_defrag_parallel_migration_init] 0-DHT: Thread[11] creation successful
[2022-06-22 15:53:10.556680 +0000] I [dht-rebalance.c:4182:gf_defrag_parallel_migration_init] 0-DHT: Thread[12] creation successful
[2022-06-22 15:53:10.556747 +0000] I [dht-rebalance.c:4182:gf_defrag_parallel_migration_init] 0-DHT: Thread[13] creation successful
[2022-06-22 15:53:10.556814 +0000] I [dht-rebalance.c:4182:gf_defrag_parallel_migration_init] 0-DHT: Thread[14] creation successful
[2022-06-22 15:53:10.556903 +0000] I [dht-rebalance.c:4182:gf_defrag_parallel_migration_init] 0-DHT: Thread[15] creation successful
[2022-06-22 15:53:10.556975 +0000] I [dht-rebalance.c:4182:gf_defrag_parallel_migration_init] 0-DHT: Thread[16] creation successful
[2022-06-22 15:53:10.557035 +0000] I [dht-rebalance.c:4182:gf_defrag_parallel_migration_init] 0-DHT: Thread[17] creation successful
[2022-06-22 15:53:10.557098 +0000] I [dht-rebalance.c:4182:gf_defrag_parallel_migration_init] 0-DHT: Thread[18] creation successful
[2022-06-22 15:53:10.557158 +0000] I [dht-rebalance.c:4182:gf_defrag_parallel_migration_init] 0-DHT: Thread[19] creation successful
[2022-06-22 15:53:10.557218 +0000] I [dht-rebalance.c:4182:gf_defrag_parallel_migration_init] 0-DHT: Thread[20] creation successful
[2022-06-22 15:53:10.557278 +0000] I [dht-rebalance.c:4182:gf_defrag_parallel_migration_init] 0-DHT: Thread[21] creation successful
[2022-06-22 15:53:10.557358 +0000] I [dht-rebalance.c:4182:gf_defrag_parallel_migration_init] 0-DHT: Thread[22] creation successful
[2022-06-22 15:53:10.557419 +0000] I [dht-rebalance.c:4182:gf_defrag_parallel_migration_init] 0-DHT: Thread[23] creation successful
[2022-06-22 15:53:10.557478 +0000] I [dht-rebalance.c:4182:gf_defrag_parallel_migration_init] 0-DHT: Thread[24] creation successful
[2022-06-22 15:53:10.557542 +0000] I [dht-rebalance.c:4182:gf_defrag_parallel_migration_init] 0-DHT: Thread[25] creation successful
[2022-06-22 15:53:10.557606 +0000] I [dht-rebalance.c:4182:gf_defrag_parallel_migration_init] 0-DHT: Thread[26] creation successful
[2022-06-22 15:53:10.557671 +0000] I [dht-rebalance.c:4182:gf_defrag_parallel_migration_init] 0-DHT: Thread[27] creation successful
[2022-06-22 15:53:10.557746 +0000] I [dht-rebalance.c:4182:gf_defrag_parallel_migration_init] 0-DHT: Thread[28] creation successful
[2022-06-22 15:53:10.557818 +0000] I [dht-rebalance.c:4182:gf_defrag_parallel_migration_init] 0-DHT: Thread[29] creation successful
[2022-06-22 15:53:10.557885 +0000] I [dht-rebalance.c:4182:gf_defrag_parallel_migration_init] 0-DHT: Thread[30] creation successful
[2022-06-22 15:53:10.557958 +0000] I [dht-rebalance.c:4182:gf_defrag_parallel_migration_init] 0-DHT: Thread[31] creation successful
[2022-06-22 15:53:10.558029 +0000] I [dht-rebalance.c:4182:gf_defrag_parallel_migration_init] 0-DHT: Thread[32] creation successful
[2022-06-22 15:53:10.558100 +0000] I [dht-rebalance.c:4182:gf_defrag_parallel_migration_init] 0-DHT: Thread[33] creation successful
[2022-06-22 15:53:10.558175 +0000] I [dht-rebalance.c:4182:gf_defrag_parallel_migration_init] 0-DHT: Thread[34] creation successful
[2022-06-22 15:53:10.558245 +0000] I [dht-rebalance.c:4182:gf_defrag_parallel_migration_init] 0-DHT: Thread[35] creation successful
[2022-06-22 15:53:10.558314 +0000] I [dht-rebalance.c:4182:gf_defrag_parallel_migration_init] 0-DHT: Thread[36] creation successful
[2022-06-22 15:53:10.558378 +0000] I [dht-rebalance.c:4182:gf_defrag_parallel_migration_init] 0-DHT: Thread[37] creation successful
[2022-06-22 15:53:10.558450 +0000] I [dht-rebalance.c:4182:gf_defrag_parallel_migration_init] 0-DHT: Thread[38] creation successful
[2022-06-22 15:53:10.558513 +0000] I [dht-rebalance.c:4182:gf_defrag_parallel_migration_init] 0-DHT: Thread[39] creation successful
[2022-06-22 15:53:10.558574 +0000] I [dht-rebalance.c:4182:gf_defrag_parallel_migration_init] 0-DHT: Thread[40] creation successful
[2022-06-22 15:53:10.558653 +0000] I [dht-rebalance.c:4182:gf_defrag_parallel_migration_init] 0-DHT: Thread[41] creation successful
[2022-06-22 15:53:10.558720 +0000] I [dht-rebalance.c:4182:gf_defrag_parallel_migration_init] 0-DHT: Thread[42] creation successful
[2022-06-22 15:53:10.558795 +0000] I [dht-rebalance.c:4182:gf_defrag_parallel_migration_init] 0-DHT: Thread[43] creation successful
[2022-06-22 15:53:10.558858 +0000] I [dht-rebalance.c:4182:gf_defrag_parallel_migration_init] 0-DHT: Thread[44] creation successful
[2022-06-22 15:53:10.558922 +0000] I [dht-rebalance.c:4182:gf_defrag_parallel_migration_init] 0-DHT: Thread[45] creation successful
[2022-06-22 15:53:10.558983 +0000] I [dht-rebalance.c:4182:gf_defrag_parallel_migration_init] 0-DHT: Thread[46] creation successful
[2022-06-22 15:53:10.559044 +0000] I [dht-rebalance.c:4182:gf_defrag_parallel_migration_init] 0-DHT: Thread[47] creation successful
[2022-06-22 15:53:10.560216 +0000] I [MSGID: 0] [dht-rebalance.c:4012:gf_defrag_total_file_size] 0-DS-dht: local subvol: DS-client-1,cnt = 1686822453248
[2022-06-22 15:53:10.560255 +0000] I [MSGID: 0] [dht-rebalance.c:4018:gf_defrag_total_file_size] 0-DS-dht: Total size files = 1686822453248
[2022-06-22 15:53:10.574856 +0000] I [MSGID: 109079] [dht-common.c:5929:dht_setxattr] 0-DS-dht: fixing the layout of /PathPoolDS/_HEARTBEAT
[2022-06-22 15:53:10.576138 +0000] I [dht-rebalance.c:3280:gf_defrag_process_dir] 0-DS-dht: migrate data called on /PathPoolDS/_HEARTBEAT
[2022-06-22 15:53:10.577856 +0000] I [dht-rebalance.c:3503:gf_defrag_process_dir] 0-DS-dht: Migration operation on dir /PathPoolDS/_HEARTBEAT took 0.00 secs
[2022-06-22 15:53:10.583072 +0000] I [MSGID: 109079] [dht-common.c:5929:dht_setxattr] 0-DS-dht: fixing the layout of /PathPoolDS/_LIBRARY
[2022-06-22 15:53:10.584111 +0000] I [dht-rebalance.c:3280:gf_defrag_process_dir] 0-DS-dht: migrate data called on /PathPoolDS/_LIBRARY
[2022-06-22 15:53:10.586562 +0000] I [dht-rebalance.c:3503:gf_defrag_process_dir] 0-DS-dht: Migration operation on dir /PathPoolDS/_LIBRARY took 0.00 secs
[2022-06-22 15:53:10.591271 +0000] I [MSGID: 109079] [dht-common.c:5929:dht_setxattr] 0-DS-dht: fixing the layout of /PathPoolDS/_ISO
[2022-06-22 15:53:10.592295 +0000] I [dht-rebalance.c:3280:gf_defrag_process_dir] 0-DS-dht: migrate data called on /PathPoolDS/_ISO
[2022-06-22 15:53:10.594204 +0000] I [dht-rebalance.c:3503:gf_defrag_process_dir] 0-DS-dht: Migration operation on dir /PathPoolDS/_ISO took 0.00 secs
[2022-06-22 15:53:10.599917 +0000] I [MSGID: 109079] [dht-common.c:5929:dht_setxattr] 0-DS-dht: fixing the layout of /PathPoolDS
[2022-06-22 15:53:10.601102 +0000] I [dht-rebalance.c:3280:gf_defrag_process_dir] 0-DS-dht: migrate data called on /PathPoolDS
[2022-06-22 15:53:10.606938 +0000] I [dht-rebalance.c:3503:gf_defrag_process_dir] 0-DS-dht: Migration operation on dir /PathPoolDS took 0.01 secs
[2022-06-22 15:53:10.608175 +0000] I [MSGID: 109079] [dht-common.c:5929:dht_setxattr] 0-DS-dht: fixing the layout of /
[2022-06-22 15:53:10.609553 +0000] I [dht-rebalance.c:3280:gf_defrag_process_dir] 0-DS-dht: migrate data called on /
[2022-06-22 15:53:10.611244 +0000] I [dht-rebalance.c:3503:gf_defrag_process_dir] 0-DS-dht: Migration operation on dir / took 0.00 secs
[2022-06-22 15:53:10.611419 +0000] I [dht-rebalance.c:4429:gf_defrag_start_crawl] 0-DHT: crawling file-system completed
[2022-06-22 15:53:10.619980 +0000] I [MSGID: 109028] [dht-rebalance.c:4687:gf_defrag_status_get] 0-DHT: Rebalance is completed. Time taken is 0.00 secs Files migrated: 0, size: 0, lookups: 68, failures: 0, skipped: 0
[2022-06-22 15:53:10.620719 +0000] W [glusterfsd.c:1432:cleanup_and_exit] (-->/lib/x86_64-linux-gnu/libpthread.so.0(+0x8ea7) [0x7fcf3b8faea7] -->/usr/sbin/glusterfs(glusterfs_sigwaiter+0xc5) [0x556254bb1175] -->/usr/sbin/glusterfs(cleanup_and_exit+0x57) [0x556254ba94f7] ) 0-: received signum (15), shutting down
**- Is there any crash ? Provide the backtrace and coredump None
Additional info:
- The operating system / glusterfs version: Debian 11 / glusterfs 9.2-1
Note: Please hide any confidential data which you don't want to share in public like IP address, file name, hostname or any other configuration
Hi @olegkrutov ,
- I dont see errors from rebalance output and as no bricks were added or removed rebalance had no layout to fix or file(s) to migrate.
We see a number of failures, what it is? Log file shows no errors (see below)
[root@7cd6a976-e2dc-4f56-9e03-a8d0cf409595 ~]# gluster volume rebalance DS status Node Rebalanced-files size scanned failures skipped status run time in h:m:s --------- ----------- ----------- ----------- ----------- ----------- ------------ -------------- 267a47b1-365f-4221-915f-5eadabf90a4c-StoreInt2 0 0Bytes 37 0 0 completed 0:00:00 localhost 0 0Bytes 68 0 0 completed 0:00:00 volume rebalance: DS: success
- Your comment explain how the cluster got to this state of : Brick1: 76.6% free-space, Brick2: 3.4% free-space
The vast majority of data is on the big brick. Maybe these were thin images so in the very beginning they were small and placing them on bigger brick was proper decision. But now its size has grown and part of data needs to be migrated to free brick.
-
glusterfs has the cluster.min-free-disk option, but it is being checked only during creation of new file. So it mean that new files will not be created on Brick2 but writes to files already on disk will continue.
-
I guess your volume doesn't configured with sharding which provided the ability to stripe data across bricks. @xhernandez - is there an option to add the
shard xlatorto the volume? will it be influencing only new files?
Thanks, Tamar
Hi @tshacked. Thank you for your reply. Yes, we do not use sharding there. So isn't there any feature to migrate some files from full brick to one having more space? Maybe manually setting some xattrs or so?
P.S. the reason not to use sharding for us is that https://github.com/gluster/glusterfs/issues/2892 is in wontfix state but data integrity is important for us.
So isn't there any feature to migrate some files from full brick to one having more space? Maybe manually setting some xattrs or so?
There's no feature that migrate file that already exist on the correct brick.
One hack that can work is doing a manual copy:
1 disable Lookup-optimize (glusterfs will search file on all bricks and not only on the hashed brick).
2 copy a FILE from Brick2 (3.4% free-space) -> Brick1 (76.6% free-space)
3 from the mount-point: ls FILE
4 check that a linkto file named FILE created on Brick2 and trusted.dht.linkto xattr points to Brick1.
--> You may repeat 2-4 for several files.
5 enable Lookup-optimize
Please notice that rebalance may migrate the files back to the correct bricks. (but I think it can be prevented with the cluster.min-free-disk option)
Hi @tshacked, thank you for your reply. But did you mean "move" instead of "copy"? And do I need to set some xattrs of migrated FILE on destination? Should I copy this file to all replicas manually, or it will be distributed by SHD automatically? Or, maybe, there is some way to change "proper brick" for a file? But fix-layout does not the trick when new brick is added. Thank you.
Hi @tshacked, thank you for your reply. But did you mean "move" instead of "copy"?
Yes, I meant move
And do I need to set some xattrs of migrated FILE on destination?
No, as this is a manual migration. On a 2nd thought there's also need to mv file's hard-link from /brick_root/.glusterfs, the full path is according to file gfid xattr. For example:
File with gfid ------> c2210e03-eaa8-47d6-8754-a59c76b92077
located at: .glusterfs/c2/21/c2210e03-eaa8-47d6-8754-a59c76b92077
Should I copy this file to all replicas manually I thought this is a pure distributed volume with 2 bricks.
Before the move backup the file and also umount any client and stop gluster service. please notice that it is a trial and if it doesn't work you can restore the file / hardlink to the original place.
Thank you for your contributions. Noticed that this issue is not having any activity in last ~6 months! We are marking this issue as stale because it has not had recent activity. It will be closed in 2 weeks if no one responds with a comment here.
![stale[bot] avatar](https://avatars.githubusercontent.com/in/1724?v=4 "Published by a pub.dev verified publisher"){kind=small}
Closing this issue as there was no update since my last update on issue. If this is an issue which is still valid, feel free to open it.