Rebalance causing duplicated files

[smremde]

Description of problem:

During a rebalance, errors occured for some files resulting in duplicate files on the volume.

[2022-03-14 16:24:47.957556 +0000] I [dht-rebalance.c:1536:dht_migrate_file] 0-dc4-dht: /data/queues/label-full-defect-qa-20191107.json: attempting to move from dc4-readdir-ahead-28 to dc4-readdir-ahead-16
[2022-03-14 16:24:48.053314 +0000] W [MSGID: 0] [dht-rebalance.c:922:__dht_check_free_space] 0-dc4-dht: Write will cross min-free-disk for file - /data/queues/label-full-defect-qa-20191107.json on subvol - dc4-readdir-ahead-16. Looking for new subvol
[2022-03-14 16:24:48.053455 +0000] I [MSGID: 0] [dht-rebalance.c:979:__dht_check_free_space] 0-dc4-dht: new target found - dc4-readdir-ahead-47 for file - /data/queues/label-full-defect-qa-20191107.json
[2022-03-14 16:24:48.192489 +0000] I [MSGID: 0] [dht-rebalance.c:1741:dht_migrate_file] 0-dc4-dht: destination for file - /data/queues/label-full-defect-qa-20191107.json is changed to - dc4-readdir-ahead-47
[2022-03-14 16:24:48.385145 +0000] E [MSGID: 109023] [dht-rebalance.c:2014:dht_migrate_file] 0-dc4-dht: failed to set xattr on /data/queues/label-full-defect-qa-20191107.json in dc4-readdir-ahead-16 [Operation not supported]
[2022-03-14 16:24:48.388013 +0000] E [MSGID: 109023] [dht-rebalance.c:2859:gf_defrag_migrate_single_file] 0-dc4-dht: migrate-data failed for /data/queues/label-full-defect-qa-20191107.json [Operation not supported]

With the corresponding messages on the bricks logs:

/var/log/glusterfs/bricks/bricks-1c9a05b3-a21b-4afb-abe2-620e6093863a-brick.log.4.gz:[2022-03-14 16:24:48.385321 +0000] E [MSGID: 113001] [posix-helpers.c:1273:posix_handle_pair] 0-dc4-posix: /bricks/1c9a05b3-a21b-4afb-abe2-620e6093863a/brick/.glusterfs/68/bc/68bc19ae-4a72-4e3d-9a60-13d21b2ff0c6: key:glusterfs.inodelk-dom-countflags: 0 length:16 [Operation not supported]
/var/log/glusterfs/bricks/bricks-30efd892-2652-4dfc-9393-4bc82191c1cf-brick.log.4.gz:[2022-03-14 16:24:48.384871 +0000] E [MSGID: 113001] [posix-helpers.c:1273:posix_handle_pair] 0-dc4-posix: /bricks/30efd892-2652-4dfc-9393-4bc82191c1cf/brick/.glusterfs/68/bc/68bc19ae-4a72-4e3d-9a60-13d21b2ff0c6: key:glusterfs.inodelk-dom-countflags: 0 length:16 [Operation not supported]
/var/log/glusterfs/bricks/bricks-7c73f6a5-17f2-4c68-a8a1-b16d5b00eb63-brick.log.4.gz:[2022-03-14 16:24:48.385173 +0000] E [MSGID: 113001] [posix-helpers.c:1273:posix_handle_pair] 0-dc4-posix: /bricks/7c73f6a5-17f2-4c68-a8a1-b16d5b00eb63/brick/.glusterfs/68/bc/68bc19ae-4a72-4e3d-9a60-13d21b2ff0c6: key:glusterfs.inodelk-dom-countflags: 0 length:16 [Operation not supported]
/var/log/glusterfs/bricks/bricks-4de9a7c4-e907-4a3d-b826-01d3aee7e45d-brick.log.4.gz:[2022-03-14 16:24:48.384902 +0000] E [MSGID: 113001] [posix-helpers.c:1273:posix_handle_pair] 0-dc4-posix: /bricks/4de9a7c4-e907-4a3d-b826-01d3aee7e45d/brick/.glusterfs/68/bc/68bc19ae-4a72-4e3d-9a60-13d21b2ff0c6: key:glusterfs.inodelk-dom-countflags: 0 length:16 [Operation not supported]
/var/log/glusterfs/bricks/bricks-01b4239c-072c-446e-bece-aae004cf023b-brick.log.4.gz:[2022-03-14 16:24:48.384330 +0000] E [MSGID: 113001] [posix-helpers.c:1273:posix_handle_pair] 0-dc4-posix: /bricks/01b4239c-072c-446e-bece-aae004cf023b/brick/.glusterfs/68/bc/68bc19ae-4a72-4e3d-9a60-13d21b2ff0c6: key:glusterfs.inodelk-dom-countflags: 0 length:16 [Operation not supported]
/var/log/glusterfs/bricks/bricks-34f0327b-baf0-46d8-b090-377302191cba-brick.log.4.gz:[2022-03-14 16:24:48.385132 +0000] E [MSGID: 113001] [posix-helpers.c:1273:posix_handle_pair] 0-dc4-posix: /bricks/34f0327b-baf0-46d8-b090-377302191cba/brick/.glusterfs/68/bc/68bc19ae-4a72-4e3d-9a60-13d21b2ff0c6: key:glusterfs.inodelk-dom-countflags: 0 length:16 [Operation not supported]

My understanding is that this xattr (glusterfs.inodelk-dom-count) could not be written (because it doesnt start with trusted, user, system, or security) and that actually these should not be attempted as this is filtered: https://github.com/gluster/glusterfs/blob/v9.3/xlators/storage/posix/src/posix-helpers.c#L1853

I'm not sure how else the execution could have got to that point.

We have noticed some issues with the volume previously. Some directories had inconsistent gfids (they were consistent across the subvolumes, but about 15 of the subvolumes differed for some directories)

Any help appreciated

Mandatory info: - The output of the gluster volume info command:

Volume Name: dc4
Type: Distributed-Disperse
Volume ID: 67697f6a-0005-4cf6-9dd1-c46a32379f96
Status: Started
Snapshot Count: 0
Number of Bricks: 58 x (4 + 2) = 348
Transport-type: tcp
Bricks:
Brick1: dc4-data-01:/bricks/98e5d8f2-6f41-4db7-87ef-be1f5c408f89/brick
Brick2: dc4-data-02:/bricks/38abf253-f718-4ff4-bb9c-570715ccbe56/brick
Brick3: dc4-data-03:/bricks/6d08b403-922e-4bd4-9b64-78c2d91f3811/brick
Brick4: dc4-data-04:/bricks/e3d27f27-abbb-4c77-b07a-ad54eb1971ec/brick
Brick5: dc4-data-05:/bricks/87bec547-c1ac-420b-ad3f-fc71c3a3b205/brick
Brick6: dc4-data-06:/bricks/1353ab19-a878-4774-9071-a4fbe17942c8/brick
Brick7: dc4-data-01:/bricks/a769c62c-d0c5-4b8c-a9e1-858a053c81d7/brick
Brick8: dc4-data-02:/bricks/cf5872c4-5b37-4f44-bc16-cd6386735904/brick
Brick9: dc4-data-03:/bricks/4119a466-7cee-4595-ae1f-3ebafcc18774/brick
Brick10: dc4-data-04:/bricks/bf3e7ef5-98d3-4bea-8d3f-11291bc47829/brick
Brick11: dc4-data-07:/bricks/4aa6521f-a82a-4252-98a8-6c1a61847600/brick
Brick12: dc4-data-08:/bricks/58a56dfd-5e57-4b2d-b9a7-08758bff03fd/brick
Brick13: dc4-data-01:/bricks/c4ec3568-1d44-48f9-b707-e358df496a52/brick
Brick14: dc4-data-02:/bricks/9b25fb78-eeca-4db1-9489-24fefa5a0723/brick
Brick15: dc4-data-05:/bricks/b0605bfe-06d4-4778-b55c-093b20a3b953/brick
Brick16: dc4-data-06:/bricks/17af6f36-ac4f-441f-bfbc-2fcc28c5f48a/brick
Brick17: dc4-data-07:/bricks/fa16fbb5-c693-4c85-aa2b-777e39ba8fb2/brick
Brick18: dc4-data-08:/bricks/61140136-786f-4a24-a8ea-0036f6802028/brick
Brick19: dc4-data-03:/bricks/7e8be266-ff22-401a-84bf-9fd984a4268a/brick
Brick20: dc4-data-04:/bricks/68f613c4-ec34-460f-b310-c6ae9ef6ed83/brick
Brick21: dc4-data-05:/bricks/34a6e8ee-2abb-411b-b065-e55f6a2cfc37/brick
Brick22: dc4-data-06:/bricks/149ceb0d-9297-486d-9f90-33f57074c71d/brick
Brick23: dc4-data-07:/bricks/8e53d861-5069-43d6-a08b-7b4f97e50de1/brick
Brick24: dc4-data-08:/bricks/8394de65-3023-49d7-a368-5061d48d38f2/brick
Brick25: dc4-data-01:/bricks/9efa9b15-186e-4fa2-8e54-bf820f66ba4b/brick
Brick26: dc4-data-02:/bricks/28835bb4-d79a-4ab9-a800-679787d2ab1a/brick
Brick27: dc4-data-03:/bricks/e283dbff-093c-4729-bf74-5612ec7e3790/brick
Brick28: dc4-data-04:/bricks/bf719933-1cfd-43c7-a68e-525f43ea657c/brick
Brick29: dc4-data-05:/bricks/5adc4ada-5c42-424d-ad75-ff05a4f4bdfe/brick
Brick30: dc4-data-07:/bricks/6e7b858b-8792-4c06-85ee-6e8e610b214c/brick
Brick31: dc4-data-01:/bricks/355ee153-b16f-4877-90df-2752f100fee0/brick
Brick32: dc4-data-02:/bricks/949c6ca2-9fb5-4028-9ccd-e84aaecdee22/brick
Brick33: dc4-data-03:/bricks/7a2b840f-02f4-4088-b722-d3eec2017454/brick
Brick34: dc4-data-04:/bricks/9699ca48-8cf6-42d1-8a87-94ecdd6ec3ba/brick
Brick35: dc4-data-06:/bricks/47f1faea-3e23-47f3-8b31-e41e59968843/brick
Brick36: dc4-data-08:/bricks/096a6ffe-2389-4162-9deb-d977959f6cf7/brick
Brick37: dc4-data-01:/bricks/68c4c38f-ff27-4a8b-96a9-6a1a841d7a9c/brick
Brick38: dc4-data-03:/bricks/d01695b8-aaa7-4846-8f96-000e17f25cf8/brick
Brick39: dc4-data-05:/bricks/727c994e-69f2-4ebe-b927-636dcc5de8bd/brick
Brick40: dc4-data-06:/bricks/43bc40ce-7e3c-4342-8fc7-debc939ccab4/brick
Brick41: dc4-data-07:/bricks/ce047004-d6ff-4658-b673-4deaadd7072f/brick
Brick42: dc4-data-08:/bricks/7dac0e8f-4c2f-4a99-91c4-327efb85af90/brick
Brick43: dc4-data-02:/bricks/016d9174-2f14-417e-b398-ea10e2200590/brick
Brick44: dc4-data-04:/bricks/9152f266-5c4a-4c89-a716-4aea6eac2edc/brick
Brick45: dc4-data-05:/bricks/61398509-55cd-4a31-882a-0d487eb2d985/brick
Brick46: dc4-data-06:/bricks/53cf6d0c-d188-4035-bd53-eb583a42d5a9/brick
Brick47: dc4-data-07:/bricks/628a0acd-2d16-49c1-bf8c-82105f412708/brick
Brick48: dc4-data-08:/bricks/f0a8391e-0924-4eeb-8616-c066d8e1e566/brick
Brick49: dc4-data-01:/bricks/bafe4a7d-3a04-49a7-92d5-a8d6e075d14e/brick
Brick50: dc4-data-02:/bricks/97ecd06a-aa03-46b4-8bfe-82a1cee67dac/brick
Brick51: dc4-data-03:/bricks/0c88efec-8c5b-4b0d-8ef8-f3d31e90a9e6/brick
Brick52: dc4-data-04:/bricks/1dcd597c-aacf-4f4a-938c-8f1ebe0feb67/brick
Brick53: dc4-data-05:/bricks/76cb2fa9-8752-47c6-b2b8-54343acc00d5/brick
Brick54: dc4-data-08:/bricks/f864ba95-d746-4d6b-bca0-c1f3b3c3f40d/brick
Brick55: dc4-data-01:/bricks/98fc4f41-70f2-401f-9618-f4b7e5cccef7/brick
Brick56: dc4-data-02:/bricks/6fd876d7-2abc-4b7c-9942-7cf0081bda38/brick
Brick57: dc4-data-03:/bricks/78c23b9e-973c-46cc-8673-2544c54c47dd/brick
Brick58: dc4-data-04:/bricks/56611698-68b0-4e9c-ae35-73fcb2e074f8/brick
Brick59: dc4-data-06:/bricks/bc62ab78-abeb-4ffb-aeed-513a7dc4abbe/brick
Brick60: dc4-data-07:/bricks/93857422-d62a-4f2f-a37e-124d2c6d6085/brick
Brick61: dc4-data-01:/bricks/14c2df57-2751-4567-8849-5ae9786eb264/brick
Brick62: dc4-data-04:/bricks/96870e54-4a0b-4ada-9339-fcde096c346a/brick
Brick63: dc4-data-05:/bricks/3ba68667-bd43-45e0-8e8e-b2538dcfeb96/brick
Brick64: dc4-data-06:/bricks/0e213dec-2b72-4dd1-94aa-85d94e5a3639/brick
Brick65: dc4-data-07:/bricks/8e953fb6-c0d2-40e0-831d-f67965186915/brick
Brick66: dc4-data-08:/bricks/f84410a1-636b-48e5-b1c2-ff1ff87504a0/brick
Brick67: dc4-data-02:/bricks/01aa1067-bb85-4d08-b397-98239f80ae69/brick
Brick68: dc4-data-03:/bricks/26138ce2-8d93-43bd-ae10-3b72d260b358/brick
Brick69: dc4-data-05:/bricks/329efe4a-68e1-43bf-a876-99aa70546e07/brick
Brick70: dc4-data-06:/bricks/618e269d-4fdf-4104-81bd-768ca0e1cb06/brick
Brick71: dc4-data-07:/bricks/704d5417-317c-4552-816f-6d06074306ac/brick
Brick72: dc4-data-08:/bricks/bffeab1f-67d4-48ec-bb6e-ace1aa350c27/brick
Brick73: dc4-data-01:/bricks/41b71fc9-2b71-4f04-83d3-f83b939bb0fd/brick
Brick74: dc4-data-02:/bricks/c89adf4f-e1d7-4db1-be4e-b9291ef34ebe/brick
Brick75: dc4-data-03:/bricks/f424939d-9d44-4ae6-9f85-5961e4470975/brick
Brick76: dc4-data-05:/bricks/a756f495-2f07-4246-8c5c-8ac9dbf0d98b/brick
Brick77: dc4-data-06:/bricks/9a7c2d45-e634-44c6-bf74-16f523a4b0b3/brick
Brick78: dc4-data-07:/bricks/b468766d-2115-4f95-a845-d938a89fe5b8/brick
Brick79: dc4-data-01:/bricks/7bae4e51-d1ab-4dbe-a714-4a4c19d5fd2d/brick
Brick80: dc4-data-02:/bricks/92eab016-56e4-4054-a121-ee4124acf896/brick
Brick81: dc4-data-04:/bricks/40bded9f-3064-4aac-a1cb-0f3d4b89c782/brick
Brick82: dc4-data-05:/bricks/e767a950-e8f3-49f9-846c-6d7e7b0a1e45/brick
Brick83: dc4-data-06:/bricks/a65115c8-e1e4-4fab-9aba-d0fda994cb21/brick
Brick84: dc4-data-08:/bricks/b0c22965-22e7-4799-9cd0-47ee53b0b82c/brick
Brick85: dc4-data-01:/bricks/7ea55c4c-dd0b-4489-ac65-af6fd184ebb2/brick
Brick86: dc4-data-03:/bricks/564c8eb9-2cb8-4a44-a8c1-a279f6923c71/brick
Brick87: dc4-data-04:/bricks/4e147bbb-e88e-4391-b46c-06bcaeb9146e/brick
Brick88: dc4-data-05:/bricks/5651757c-1c29-49ad-9305-ba025ae35874/brick
Brick89: dc4-data-07:/bricks/a326f387-5a01-4761-a9bc-13753403ba27/brick
Brick90: dc4-data-08:/bricks/14c8c856-d68c-40b8-97e0-3a8d7b315087/brick
Brick91: dc4-data-02:/bricks/be4d0ca2-5742-4287-bc69-3645991e37ad/brick
Brick92: dc4-data-03:/bricks/019d80f6-1fe1-4fb5-baf1-364c17e7cafa/brick
Brick93: dc4-data-04:/bricks/0034e5bc-e673-461f-bd53-ea42113cbbe4/brick
Brick94: dc4-data-06:/bricks/8c36a53e-4b38-4e1c-938b-e505201e2133/brick
Brick95: dc4-data-07:/bricks/0432abd5-9339-4e5a-9efe-3fddbb11935c/brick
Brick96: dc4-data-08:/bricks/3aad4a4f-818d-4879-ae38-f607497a54e8/brick
Brick97: dc4-data-01:/bricks/1c9a05b3-a21b-4afb-abe2-620e6093863a/brick
Brick98: dc4-data-02:/bricks/30efd892-2652-4dfc-9393-4bc82191c1cf/brick
Brick99: dc4-data-03:/bricks/7c73f6a5-17f2-4c68-a8a1-b16d5b00eb63/brick
Brick100: dc4-data-05:/bricks/4de9a7c4-e907-4a3d-b826-01d3aee7e45d/brick
Brick101: dc4-data-06:/bricks/01b4239c-072c-446e-bece-aae004cf023b/brick
Brick102: dc4-data-08:/bricks/34f0327b-baf0-46d8-b090-377302191cba/brick
Brick103: dc4-data-01:/bricks/0c2332b7-52e3-44fe-b573-9f7b90b7d174/brick
Brick104: dc4-data-02:/bricks/180e6373-6f0b-442d-bcf5-2ebb64273e6e/brick
Brick105: dc4-data-04:/bricks/7506ec08-fbe6-4955-bc5b-f2a84fbc4872/brick
Brick106: dc4-data-05:/bricks/ed9ed283-5903-4782-bcde-909f40325f45/brick
Brick107: dc4-data-06:/bricks/c33ff241-4942-4fe8-a348-6e3c5b67d63f/brick
Brick108: dc4-data-07:/bricks/a9392c96-ff68-48fc-b130-6bc44c1c160c/brick
Brick109: dc4-data-01:/bricks/32713096-d920-42e4-bbc7-b916cbb174d7/brick
Brick110: dc4-data-03:/bricks/2c34d3bc-4dbc-4133-be47-823487c77184/brick
Brick111: dc4-data-04:/bricks/2fe52ebc-7038-42da-8a24-69c2ffa90acf/brick
Brick112: dc4-data-06:/bricks/4816360c-6cd1-4d42-a43d-1aab4efc9343/brick
Brick113: dc4-data-07:/bricks/72e449eb-92e3-4cc7-aa62-b464f2113d7d/brick
Brick114: dc4-data-08:/bricks/0dd1b242-12f4-420f-a84e-b8197b1372cc/brick
Brick115: dc4-data-02:/bricks/74d2a853-0ef5-4656-8c6d-447570b1e43c/brick
Brick116: dc4-data-03:/bricks/895a5355-ccce-47df-8186-b2deb2c3ecc6/brick
Brick117: dc4-data-04:/bricks/2424bc49-a08d-4aca-b503-f565d9f29b53/brick
Brick118: dc4-data-05:/bricks/fb87835f-1dfb-47d2-b7c5-929577833608/brick
Brick119: dc4-data-07:/bricks/7b9f6881-ecae-43dc-9eef-380f0b2ac752/brick
Brick120: dc4-data-08:/bricks/25622c46-24ef-4072-8122-d10fd9643057/brick
Brick121: dc4-data-01:/bricks/269a00dd-11c9-4558-8abd-c0f5060ddb56/brick
Brick122: dc4-data-02:/bricks/698c400f-164e-4822-923b-15aab9899b03/brick
Brick123: dc4-data-03:/bricks/8af36ce6-568d-4e24-ad3b-fa1d1648d0f5/brick
Brick124: dc4-data-04:/bricks/d2ef4d1c-7055-4c35-98d4-a0ba4ef619d3/brick
Brick125: dc4-data-05:/bricks/4623f208-d020-47ef-8ca8-24a740630e92/brick
Brick126: dc4-data-06:/bricks/870f5b60-dfc8-4e11-a766-b5d153ac2d94/brick
Brick127: dc4-data-01:/bricks/fcb0bcef-91c6-4c97-a949-064cb9ec2c8c/brick
Brick128: dc4-data-02:/bricks/bfb73523-9206-4116-9497-fde7591f7442/brick
Brick129: dc4-data-03:/bricks/8f48fa1d-5975-4c8b-8e08-5a86ead36857/brick
Brick130: dc4-data-04:/bricks/b6a7959e-8c24-427a-8c3a-a9c97a30c0eb/brick
Brick131: dc4-data-07:/bricks/3b13b8ef-161d-43c6-906d-acbc9004f48f/brick
Brick132: dc4-data-08:/bricks/8501dd69-f19b-4faa-bc18-d8fcaad072c0/brick
Brick133: dc4-data-01:/bricks/43f4591a-bf13-4da3-9453-3cbdad4b2896/brick
Brick134: dc4-data-02:/bricks/698385a0-7adb-4556-813d-b58e2bec9ed5/brick
Brick135: dc4-data-05:/bricks/f8377f0f-b7f7-45a0-83cf-298a09f44ee9/brick
Brick136: dc4-data-06:/bricks/c41f0298-9f29-413d-8a24-465db272df31/brick
Brick137: dc4-data-07:/bricks/de076b77-6777-45d4-9387-52d87aceb823/brick
Brick138: dc4-data-08:/bricks/7645ac8b-3745-4843-bad2-47ed51e075bb/brick
Brick139: dc4-data-03:/bricks/39684cf4-7a71-43c0-b490-f7b2c7f7f5c3/brick
Brick140: dc4-data-04:/bricks/a6092f37-5387-48e8-b09e-bbf61bb2c5ee/brick
Brick141: dc4-data-05:/bricks/b68ac6f4-3c69-4a81-a133-d42ceabc5d4a/brick
Brick142: dc4-data-06:/bricks/f1d4c315-a93c-4eb2-9dc1-75f306416969/brick
Brick143: dc4-data-07:/bricks/839da62b-be1b-43c2-9945-af8e40ad629c/brick
Brick144: dc4-data-08:/bricks/e3d9d0a8-ad24-4a43-b005-9c14503748cb/brick
Brick145: dc4-data-01:/bricks/c940ff26-1be0-4b4f-81b0-5833ee96e481/brick
Brick146: dc4-data-02:/bricks/fda4e5f3-3956-4914-8f31-5abb5ebac2ad/brick
Brick147: dc4-data-03:/bricks/84b05270-8df8-4e1c-8b8c-52e2ba0c929c/brick
Brick148: dc4-data-04:/bricks/f597a072-eefe-4423-a476-d75198bbbebe/brick
Brick149: dc4-data-05:/bricks/e19ac7ad-194b-48b0-a244-a992ef865614/brick
Brick150: dc4-data-07:/bricks/45ce6a2b-9b45-427f-8a14-c397f901eb6e/brick
Brick151: dc4-data-01:/bricks/5ec1ecc9-eec1-43fc-b011-42946938735a/brick
Brick152: dc4-data-02:/bricks/be42b015-5453-414b-abba-2a5e05f3fa47/brick
Brick153: dc4-data-03:/bricks/5819677f-fb7d-42e7-8220-4df559dd62b4/brick
Brick154: dc4-data-04:/bricks/a8b29014-f17d-4f44-968e-d7ed4c3ec148/brick
Brick155: dc4-data-06:/bricks/0ddf7d14-77c6-41da-83eb-13e90a05b914/brick
Brick156: dc4-data-08:/bricks/51ad681a-84a6-443e-b4d5-2aa3f2844450/brick
Brick157: dc4-data-01:/bricks/d71c2b31-8e8c-4857-8ae6-c92ea55db2ff/brick
Brick158: dc4-data-03:/bricks/64419ba1-02a6-4d9d-8148-2db78d3d809d/brick
Brick159: dc4-data-05:/bricks/dfbfa963-f68f-4ee0-9680-4ff1a49aa2e0/brick
Brick160: dc4-data-06:/bricks/eadd3e05-d1e8-4345-8095-03007fa7e158/brick
Brick161: dc4-data-07:/bricks/8817a65e-0a36-4a69-ad89-d748c7433e85/brick
Brick162: dc4-data-08:/bricks/d14bd3fb-47cd-48db-926c-4d49bde68851/brick
Brick163: dc4-data-02:/bricks/b299aad5-05e2-4c9e-8280-b78ea08be458/brick
Brick164: dc4-data-04:/bricks/a3b090bc-afc7-45aa-b7d3-6a293b9574f6/brick
Brick165: dc4-data-05:/bricks/9c304795-44c0-485d-a85e-71c4a902c022/brick
Brick166: dc4-data-06:/bricks/1880df0f-c6e8-48d3-97e4-2dff36e97087/brick
Brick167: dc4-data-07:/bricks/7350581b-0a3b-4e2e-aa43-22e968aab95b/brick
Brick168: dc4-data-08:/bricks/204cf751-56c9-41a3-8ec6-9bd818cac02a/brick
Brick169: dc4-data-01:/bricks/062a0def-2478-452b-9544-788cd8c7bc96/brick
Brick170: dc4-data-02:/bricks/a2b188f5-f239-4ccd-8778-5963734fad8f/brick
Brick171: dc4-data-03:/bricks/1842d244-3581-43e1-ae65-cc0247fd2d0a/brick
Brick172: dc4-data-04:/bricks/b38f81b9-bd97-4551-9eb2-6303c15c3724/brick
Brick173: dc4-data-06:/bricks/c7b54765-0867-4349-8976-2fd0a8bde7a7/brick
Brick174: dc4-data-07:/bricks/8ecf3f7e-a5c5-48fd-a63e-0dca2d4c2957/brick
Brick175: dc4-data-01:/bricks/1fc6680a-35a2-4a12-8f48-4d1c227df2b1/brick
Brick176: dc4-data-04:/bricks/a75005f2-d1fe-4455-848c-7340ee6bce5c/brick
Brick177: dc4-data-05:/bricks/48320abb-3fcc-4eab-ba5b-4449141e7f13/brick
Brick178: dc4-data-06:/bricks/5ed5a40c-62b7-4d0f-a163-3d94d405a07b/brick
Brick179: dc4-data-07:/bricks/8f228a0d-a575-43a3-b4db-26d76b597293/brick
Brick180: dc4-data-08:/bricks/90dc9cee-2f55-40d0-b51b-d40134d994f7/brick
Brick181: dc4-data-02:/bricks/c7f0e44a-3065-4e07-8ebd-c9d35e08efdb/brick
Brick182: dc4-data-03:/bricks/9b0a963d-4a47-415d-9486-efc72b76ec7a/brick
Brick183: dc4-data-05:/bricks/628cbaa3-5351-4f14-adfd-28182bb4539b/brick
Brick184: dc4-data-06:/bricks/b4cd57c9-6ff0-4071-b48f-80b320f2c5ea/brick
Brick185: dc4-data-07:/bricks/b1dacc57-97e8-4bfd-86c3-f6ad46927ae2/brick
Brick186: dc4-data-08:/bricks/7af73b5a-cece-422d-acde-fafe1c43f17a/brick
Brick187: dc4-data-01:/bricks/92f8afa8-5169-450d-8896-d148ca303c3f/brick
Brick188: dc4-data-02:/bricks/f50fc5a6-519f-4f08-9353-869e12a6e91a/brick
Brick189: dc4-data-03:/bricks/08fb01a9-bb8a-4cdb-9269-27244873aa67/brick
Brick190: dc4-data-05:/bricks/7411968d-93f4-42c8-99a9-ef632ee47f07/brick
Brick191: dc4-data-06:/bricks/a2b2738d-84dd-4e58-94c6-d2cbed0fba99/brick
Brick192: dc4-data-07:/bricks/cd2619c1-f95f-430e-abbe-ce4a4430d682/brick
Brick193: dc4-data-01:/bricks/506f2be8-eca4-4581-8799-03fdefa23701/brick
Brick194: dc4-data-02:/bricks/7fbb4e2d-723a-467f-a682-04081e731d95/brick
Brick195: dc4-data-04:/bricks/1e2c3a8f-8162-4838-870f-7d31fd9683db/brick
Brick196: dc4-data-05:/bricks/9462cd79-c633-4e52-b743-2384146b99c4/brick
Brick197: dc4-data-06:/bricks/8df43fbb-469d-4250-b11c-9c00f279e28f/brick
Brick198: dc4-data-08:/bricks/9adb1dc2-8b29-44ee-9b1f-018162a486b9/brick
Brick199: dc4-data-01:/bricks/f2dd1b02-b5c0-40cd-8c86-72e45e87fd60/brick
Brick200: dc4-data-03:/bricks/f3fe9ab5-3f99-45ec-89d7-41fa93884bec/brick
Brick201: dc4-data-04:/bricks/609c01fa-846f-4ab6-ad10-1f0236a6057b/brick
Brick202: dc4-data-05:/bricks/33820955-d41e-4ffb-94b0-37b4a88b8c63/brick
Brick203: dc4-data-07:/bricks/bca0f7ec-0efe-4815-a9f8-f78b1fdfeca9/brick
Brick204: dc4-data-08:/bricks/99e66491-96c3-4983-9d35-b2db266c6e9a/brick
Brick205: dc4-data-02:/bricks/8f9615ad-7d3f-434c-a13f-41cf69f919c3/brick
Brick206: dc4-data-03:/bricks/02f0d768-8fed-4ff0-a713-3dd5432c830e/brick
Brick207: dc4-data-04:/bricks/cceab27e-3b89-4d78-90f1-6cb5144af3b8/brick
Brick208: dc4-data-06:/bricks/d91e81bc-1616-4ede-bc6f-5fc2e8e44d3b/brick
Brick209: dc4-data-07:/bricks/da33268a-47ed-4d4f-9361-d330b841497a/brick
Brick210: dc4-data-08:/bricks/2ee6d931-5b30-4147-b2c3-7ea005810678/brick
Brick211: dc4-data-01:/bricks/cf7d491e-ccf4-4d0c-bdef-a91a50b9693e/brick
Brick212: dc4-data-02:/bricks/6050bb3f-1e9e-4097-9b00-49cc04cb6c2a/brick
Brick213: dc4-data-03:/bricks/bee82bcc-3f37-4e46-870d-d99508216d8f/brick
Brick214: dc4-data-05:/bricks/4644b844-37e0-42f4-b010-6b79ebf5569f/brick
Brick215: dc4-data-06:/bricks/f7378998-13ab-4662-8119-a91b94a8827c/brick
Brick216: dc4-data-08:/bricks/a4003298-3792-4097-bb0b-a514e6078e59/brick
Brick217: dc4-data-01:/bricks/18364e38-ce5b-4737-9832-3ceb7c35f3ef/brick
Brick218: dc4-data-02:/bricks/db6a19ac-32e7-4b28-8c75-efedaa1fae2d/brick
Brick219: dc4-data-04:/bricks/4b2a333e-a132-4f69-bf16-29c9947f3303/brick
Brick220: dc4-data-05:/bricks/f68e50b8-c154-44a0-a70e-2b5dac0dde9b/brick
Brick221: dc4-data-06:/bricks/97239520-592f-4253-bd28-a62d418c1d1b/brick
Brick222: dc4-data-07:/bricks/bb36ec62-b169-41de-abcd-752a7ee3ed1c/brick
Brick223: dc4-data-01:/bricks/35b5ded7-973c-4dde-8379-26c14e669340/brick
Brick224: dc4-data-03:/bricks/f1ff9593-a5af-43fc-b357-39667e1e4366/brick
Brick225: dc4-data-04:/bricks/de34d04b-ddb2-4c8f-aa2a-fa23eeb35fcf/brick
Brick226: dc4-data-06:/bricks/14b30e8f-8ae3-4f02-9d3c-6ad77433fc81/brick
Brick227: dc4-data-07:/bricks/fabd6f77-f7e0-416d-ada6-3e57d64621bd/brick
Brick228: dc4-data-08:/bricks/047c3334-5260-4e24-a04f-267354696dbd/brick
Brick229: dc4-data-02:/bricks/44483dec-0459-4fe6-aec1-067eb63938c8/brick
Brick230: dc4-data-03:/bricks/e7753e5d-86ec-492b-bf46-cd4acabcc4fa/brick
Brick231: dc4-data-04:/bricks/06b7692e-db1e-4562-8aac-a1ac6260ee8c/brick
Brick232: dc4-data-05:/bricks/e616a50f-dca6-441a-bc95-b01ce5b3e6d9/brick
Brick233: dc4-data-07:/bricks/cf3ea474-0858-495a-83b8-4680e6807bc6/brick
Brick234: dc4-data-08:/bricks/64045d29-d832-4000-aedb-f99956ffd1df/brick
Brick235: dc4-data-01:/bricks/42d39157-f3e0-455b-8222-e457fa951a92/brick
Brick236: dc4-data-02:/bricks/2c134ab6-ffb9-440b-8029-636fad8df8ef/brick
Brick237: dc4-data-03:/bricks/9b09ecc5-915b-4692-b31b-325cc05a54f2/brick
Brick238: dc4-data-05:/bricks/61cd0519-8479-4208-a7a4-e118a406079b/brick
Brick239: dc4-data-07:/bricks/742cd7fe-0e7d-44f0-80ac-add033aa0b49/brick
Brick240: dc4-data-08:/bricks/ccac7912-94e9-49cc-9a07-fa3b3ad08d8b/brick
Brick241: dc4-data-01:/bricks/efa6d60d-0976-4e4d-9ea7-3f8a88593639/brick
Brick242: dc4-data-02:/bricks/14ab052c-665d-4fd8-9369-1f5813de95e1/brick
Brick243: dc4-data-04:/bricks/dc4d269e-a8b4-4515-b0ec-c86998a9e5b9/brick
Brick244: dc4-data-06:/bricks/1b69cb1c-5ef7-47f3-8d9f-abb97c4870a5/brick
Brick245: dc4-data-07:/bricks/13a42d55-e55d-448e-8076-83e3ed1ce857/brick
Brick246: dc4-data-08:/bricks/5b59501d-a21f-4749-9fbf-db8f58400e72/brick
Brick247: dc4-data-01:/bricks/de7394af-754d-4b27-b058-39cb59df209e/brick
Brick248: dc4-data-03:/bricks/4053e884-ba17-44fd-a695-b5bb509b284a/brick
Brick249: dc4-data-04:/bricks/2360572a-6651-4c7d-be46-09655c72345e/brick
Brick250: dc4-data-05:/bricks/d50f4080-3563-490c-920a-5643c01c5d44/brick
Brick251: dc4-data-06:/bricks/bfa8b49c-57c3-4451-95f0-1865e72c84b8/brick
Brick252: dc4-data-07:/bricks/9bdc7239-41cf-445f-baf0-7e2f219bce20/brick
Brick253: dc4-data-02:/bricks/ed282b52-7929-4494-b3d4-0a75b7d48c25/brick
Brick254: dc4-data-03:/bricks/db221763-9e0e-4254-a882-04a7ca2b7cd7/brick
Brick255: dc4-data-04:/bricks/048f0649-d99d-4bef-a57b-de6df115505c/brick
Brick256: dc4-data-05:/bricks/055a0168-fcd4-44f2-963e-7d2b924c1268/brick
Brick257: dc4-data-06:/bricks/9e79de75-763c-4598-8594-95f1b76bd00c/brick
Brick258: dc4-data-08:/bricks/9bf9aa32-806f-468f-a89a-0f6a43886ba8/brick
Brick259: dc4-data-01:/bricks/87b8d70f-0e5c-4111-bbe5-f2d86bd94392/brick
Brick260: dc4-data-02:/bricks/f9e6a80f-a2c5-432d-81d6-33580b3b6b79/brick
Brick261: dc4-data-03:/bricks/8429b01e-70e9-4713-b4fe-bac434bb8732/brick
Brick262: dc4-data-06:/bricks/0555ee69-c71f-4b95-ae36-9229bc2e5128/brick
Brick263: dc4-data-07:/bricks/0f0a57e4-b751-47a8-8303-34c741393933/brick
Brick264: dc4-data-08:/bricks/2caa84ab-9a5c-45b2-aadd-1c86ae587144/brick
Brick265: dc4-data-01:/bricks/a9fc5c13-8361-4ec8-8a13-04a3733e4675/brick
Brick266: dc4-data-02:/bricks/3f91a1d4-4dbd-4e04-93e4-8afcdab80d1e/brick
Brick267: dc4-data-04:/bricks/5e6b01ec-9f21-4f92-828f-bd092817d951/brick
Brick268: dc4-data-05:/bricks/5c3fb591-dca5-42a0-8b44-bb637a2c4083/brick
Brick269: dc4-data-07:/bricks/a2ca3d2f-2582-4867-a0bf-4746aeb97dba/brick
Brick270: dc4-data-08:/bricks/66795508-201a-471c-8417-d88fce4b88af/brick
Brick271: dc4-data-01:/bricks/7ccf35e0-5a91-4de0-8aef-c0fe5960a240/brick
Brick272: dc4-data-03:/bricks/4caf6806-1951-441d-9702-cf10fcddd61f/brick
Brick273: dc4-data-04:/bricks/4f1c7ded-1ef8-4e83-aa33-47abe0a7fa23/brick
Brick274: dc4-data-05:/bricks/1d4071e4-3c28-4dc1-9f68-1fffe4ceac24/brick
Brick275: dc4-data-06:/bricks/5e520c12-7628-4dd1-9601-62548bdbf7dd/brick
Brick276: dc4-data-08:/bricks/df4c8f2f-81f1-40b2-9a50-04f673cac61c/brick
Brick277: dc4-data-02:/bricks/cc9db49c-0f52-4bb8-a526-59cae80a2914/brick
Brick278: dc4-data-03:/bricks/2d5280c8-d8b6-4435-98e9-1e7d1b1c20a9/brick
Brick279: dc4-data-04:/bricks/5ccc726e-6e75-4e50-98fa-a06d57bd1741/brick
Brick280: dc4-data-05:/bricks/0a39fc16-3562-48b3-a1e4-2ef4a977bbd2/brick
Brick281: dc4-data-06:/bricks/96f4c0ad-26c5-46fc-87ae-a0e1441bdd2d/brick
Brick282: dc4-data-07:/bricks/956b1909-0537-40be-9cd2-9d92f21db3f3/brick
Brick283: dc4-data-01:/bricks/9e893d26-cd87-490e-9d6e-83c6fecebf54/brick
Brick284: dc4-data-02:/bricks/fce7e9b3-4ff4-4e01-ae02-8d27d175485c/brick
Brick285: dc4-data-03:/bricks/143427df-c5b1-4b4b-b38a-9f337418c187/brick
Brick286: dc4-data-04:/bricks/ff78d3c6-0ce2-4c7a-98ed-8b7cc1d588be/brick
Brick287: dc4-data-05:/bricks/43b04ce5-0f35-4eab-8ca6-aaaf3c0a91cc/brick
Brick288: dc4-data-06:/bricks/9a3d92a4-49b5-449a-97ea-e49f66f9b672/brick
Brick289: dc4-data-01:/bricks/fab7c509-5ef2-4e05-8dcc-4cdbf8d527eb/brick
Brick290: dc4-data-02:/bricks/25398e71-bfb9-4461-9971-ccce45d57560/brick
Brick291: dc4-data-03:/bricks/24a66144-965a-48cd-807d-27f6767c88a0/brick
Brick292: dc4-data-04:/bricks/131c905c-8364-4c79-9d88-686e5acb7d00/brick
Brick293: dc4-data-07:/bricks/3a06f214-c884-4bbc-b7d9-eec61794ee25/brick
Brick294: dc4-data-08:/bricks/cb9e06e8-a8ef-48b1-bb1d-7a5d11df6647/brick
Brick295: dc4-data-01:/bricks/ef6e8346-4203-4d5b-a16e-7175dc777848/brick
Brick296: dc4-data-02:/bricks/8c3b387c-e0ee-40dc-9f0e-5379c75422cd/brick
Brick297: dc4-data-05:/bricks/594feaca-bc5f-45cb-9706-3058768e8cdb/brick
Brick298: dc4-data-06:/bricks/c1424f31-a696-4619-89bf-92ab3de42338/brick
Brick299: dc4-data-07:/bricks/82ac6458-d01c-44d0-8df0-dd1fcd4e110f/brick
Brick300: dc4-data-08:/bricks/a5da60c2-32b0-4ee7-b7af-36cce55a4158/brick
Brick301: dc4-data-03:/bricks/0fa7d10f-f4c7-43c5-84a5-623534462cae/brick
Brick302: dc4-data-04:/bricks/842ea5a9-bb11-470f-9add-79c2ff504789/brick
Brick303: dc4-data-05:/bricks/1b31e7c5-063b-4ace-9152-20462d1a95e5/brick
Brick304: dc4-data-06:/bricks/0b74127b-e23f-45c2-a50c-227f99b5f5bf/brick
Brick305: dc4-data-07:/bricks/f033c41a-58e2-4eba-854e-6c9f90e29f8c/brick
Brick306: dc4-data-08:/bricks/39b9defd-9391-41b6-a94c-d42a3139368c/brick
Brick307: dc4-data-01:/bricks/de4b7692-3739-4d99-888e-0f28bc61aea2/brick
Brick308: dc4-data-02:/bricks/6329c8f5-a512-4d9d-9f0a-c7aedf69d47a/brick
Brick309: dc4-data-03:/bricks/aee04550-1b88-4ea0-a671-2f32f76fd71b/brick
Brick310: dc4-data-04:/bricks/f54b800a-c86e-41a6-95d7-808c9199eb2f/brick
Brick311: dc4-data-05:/bricks/ab1b5a5c-3df6-41f8-9dc1-17941486871b/brick
Brick312: dc4-data-07:/bricks/07e29047-8a2b-491f-a7a9-85253d173982/brick
Brick313: dc4-data-01:/bricks/7f4488cb-0bdb-4603-bf50-e392e5b49360/brick
Brick314: dc4-data-02:/bricks/4cf41147-72cf-48a9-83aa-7f0c23e65c6e/brick
Brick315: dc4-data-03:/bricks/a734dd2a-fb5b-4b4d-b29d-7dc737b373f8/brick
Brick316: dc4-data-04:/bricks/0d223cf2-d87f-45f3-b2d5-b263bae74780/brick
Brick317: dc4-data-06:/bricks/d109b20b-fd99-4189-aa2d-b219f2afd379/brick
Brick318: dc4-data-08:/bricks/d39b5497-6674-4571-8cdb-4bcb508bc834/brick
Brick319: dc4-data-01:/bricks/de539508-3f2c-47ab-89c4-aab082ed1383/brick
Brick320: dc4-data-03:/bricks/fbeca7c5-282a-42a9-9496-7eb16927f511/brick
Brick321: dc4-data-05:/bricks/bb68cf06-0d3d-4317-876d-5f5d5e986c0e/brick
Brick322: dc4-data-06:/bricks/be5fd104-f624-48c0-a101-fc3010b74a32/brick
Brick323: dc4-data-07:/bricks/94352edf-d036-4dcb-a508-62b34091d75a/brick
Brick324: dc4-data-08:/bricks/5354752c-a188-48aa-8aa8-692de9372a6b/brick
Brick325: dc4-data-02:/bricks/914ee0af-2c4b-4811-80da-129c1ce19b10/brick
Brick326: dc4-data-04:/bricks/91b943b2-796f-458a-9940-01cfc01980fd/brick
Brick327: dc4-data-05:/bricks/c532fab2-062e-4f55-a931-fbabe7b5019f/brick
Brick328: dc4-data-06:/bricks/524bd9f3-03b1-4749-aefc-f4278f29c437/brick
Brick329: dc4-data-07:/bricks/0a782811-817d-4a10-88a9-9e9cec0d61c1/brick
Brick330: dc4-data-08:/bricks/56cb020a-8716-4e83-8fc9-e3c2f7181ebc/brick
Brick331: dc4-data-01:/bricks/66a633d7-8e95-463f-bd2c-51e4e3693b38/brick
Brick332: dc4-data-02:/bricks/6be8ceca-ab67-4c84-b598-ea1b298804fe/brick
Brick333: dc4-data-03:/bricks/23b8d715-65a7-410e-876c-63b533de967a/brick
Brick334: dc4-data-04:/bricks/88cf78fe-dad5-4605-bd5d-06fbf435b089/brick
Brick335: dc4-data-05:/bricks/28e1b6c9-1fc4-4cc0-86a6-89e41281eb91/brick
Brick336: dc4-data-08:/bricks/f59883fc-6af0-4a03-a611-e9f894464cfd/brick
Brick337: dc4-data-01:/bricks/8de99f3e-f0b1-462b-adb4-e94e7822347d/brick
Brick338: dc4-data-02:/bricks/ab2234fd-a400-4933-9af1-9c213444df77/brick
Brick339: dc4-data-03:/bricks/70479b15-2011-4163-8466-836615d7fa2d/brick
Brick340: dc4-data-04:/bricks/0bc1d587-c761-4134-a878-e2e98039bc95/brick
Brick341: dc4-data-06:/bricks/871c4394-e0ab-42e1-b454-896c1908cec6/brick
Brick342: dc4-data-07:/bricks/3c6fa101-030b-41e1-a432-b4e6776b0a6b/brick
Brick343: dc4-data-02:/bricks/090e1c55-1e9c-466d-bd4b-8e2e36a22470/brick
Brick344: dc4-data-03:/bricks/2759e55b-2596-4a0c-a421-de0f09dc5732/brick
Brick345: dc4-data-05:/bricks/2f924940-6f3c-4755-be48-fb1fda7ba135/brick
Brick346: dc4-data-06:/bricks/a4c58999-cfb5-4770-8a8e-9d53e25542ec/brick
Brick347: dc4-data-07:/bricks/12f337f6-17e5-448d-878d-42cd45c599de/brick
Brick348: dc4-data-08:/bricks/dc5f9678-8836-4d0e-bfa8-9b5d48245733/brick
Options Reconfigured:
diagnostics.brick-log-level: INFO
cluster.disperse-self-heal-daemon: enable
diagnostics.client-log-level: INFO
disperse.shd-max-threads: 2
performance.client-io-threads: on
cluster.rebal-throttle: normal
cluster.min-free-disk: 1%
performance.readdir-ahead: on
performance.parallel-readdir: on
storage.fips-mode-rchecksum: on
transport.address-family: inet
nfs.disable: on

- The output of the gluster volume status command: Normal

- The output of the gluster volume heal command: Normal

Additional info:

- The operating system / glusterfs version: Ubuntu 20.04 Glusterfs 9.4

[smremde]

This particular files has 2 copies + 2 linkto

ls /mnt/volume/data/queues
...
-rw-rw-r--  1 user users   3087081 Nov  7  2019 label-full-defect-qa-20191107.json
-rw-rw-r--  1 user users         0 Nov  7  2019 label-full-defect-qa-20191107.json
...

files (14th of Feb and 14th March correlate with rebalance operations)

dc4-data-01
-rw-rw-r-- 2 gaist users 772096 Nov  7  2019 /bricks/062a0def-2478-452b-9544-788cd8c7bc96/brick/data/queues/label-full-defect-qa-20191107.json
---------T 2 gaist users      0 Mar 14 16:44 /bricks/1c9a05b3-a21b-4afb-abe2-620e6093863a/brick/data/queues/label-full-defect-qa-20191107.json
-rw-rw-r-- 2 gaist users      0 Mar 14 16:24 /bricks/9e893d26-cd87-490e-9d6e-83c6fecebf54/brick/data/queues/label-full-defect-qa-20191107.json
dc4-data-02
---------T 2 gaist users      0 Feb 14 20:56 /bricks/016d9174-2f14-417e-b398-ea10e2200590/brick/data/queues/label-full-defect-qa-20191107.json
---------T 2 gaist users      0 Mar 14 16:44 /bricks/30efd892-2652-4dfc-9393-4bc82191c1cf/brick/data/queues/label-full-defect-qa-20191107.json
-rw-rw-r-- 2 gaist users 772096 Nov  7  2019 /bricks/a2b188f5-f239-4ccd-8778-5963734fad8f/brick/data/queues/label-full-defect-qa-20191107.json
-rw-rw-r-- 2 gaist users      0 Mar 14 16:24 /bricks/fce7e9b3-4ff4-4e01-ae02-8d27d175485c/brick/data/queues/label-full-defect-qa-20191107.json
dc4-data-03
-rw-rw-r-- 2 gaist users      0 Mar 14 16:24 /bricks/143427df-c5b1-4b4b-b38a-9f337418c187/brick/data/queues/label-full-defect-qa-20191107.json
-rw-rw-r-- 2 gaist users 772096 Nov  7  2019 /bricks/1842d244-3581-43e1-ae65-cc0247fd2d0a/brick/data/queues/label-full-defect-qa-20191107.json
---------T 2 gaist users      0 Mar 14 16:44 /bricks/7c73f6a5-17f2-4c68-a8a1-b16d5b00eb63/brick/data/queues/label-full-defect-qa-20191107.json
dc4-data-04
---------T 2 gaist users      0 Feb 14 20:56 /bricks/9152f266-5c4a-4c89-a716-4aea6eac2edc/brick/data/queues/label-full-defect-qa-20191107.json
-rw-rw-r-- 2 gaist users 772096 Nov  7  2019 /bricks/b38f81b9-bd97-4551-9eb2-6303c15c3724/brick/data/queues/label-full-defect-qa-20191107.json
-rw-rw-r-- 2 gaist users      0 Mar 14 16:24 /bricks/ff78d3c6-0ce2-4c7a-98ed-8b7cc1d588be/brick/data/queues/label-full-defect-qa-20191107.json
dc4-data-05
-rw-rw-r-- 2 gaist users 0 Mar 14 16:24 /bricks/43b04ce5-0f35-4eab-8ca6-aaaf3c0a91cc/brick/data/queues/label-full-defect-qa-20191107.json
---------T 2 gaist users 0 Mar 14 16:44 /bricks/4de9a7c4-e907-4a3d-b826-01d3aee7e45d/brick/data/queues/label-full-defect-qa-20191107.json
---------T 2 gaist users 0 Feb 14 20:56 /bricks/61398509-55cd-4a31-882a-0d487eb2d985/brick/data/queues/label-full-defect-qa-20191107.json
dc4-data-06
---------T 2 gaist users      0 Mar 14 16:44 /bricks/01b4239c-072c-446e-bece-aae004cf023b/brick/data/queues/label-full-defect-qa-20191107.json
---------T 2 gaist users      0 Feb 14 20:56 /bricks/53cf6d0c-d188-4035-bd53-eb583a42d5a9/brick/data/queues/label-full-defect-qa-20191107.json
-rw-rw-r-- 2 gaist users      0 Mar 14 16:24 /bricks/9a3d92a4-49b5-449a-97ea-e49f66f9b672/brick/data/queues/label-full-defect-qa-20191107.json
-rw-rw-r-- 2 gaist users 772096 Nov  7  2019 /bricks/c7b54765-0867-4349-8976-2fd0a8bde7a7/brick/data/queues/label-full-defect-qa-20191107.json
dc4-data-07
---------T 2 gaist users      0 Feb 14 20:56 /bricks/628a0acd-2d16-49c1-bf8c-82105f412708/brick/data/queues/label-full-defect-qa-20191107.json
-rw-rw-r-- 2 gaist users 772096 Nov  7  2019 /bricks/8ecf3f7e-a5c5-48fd-a63e-0dca2d4c2957/brick/data/queues/label-full-defect-qa-20191107.json
dc4-data-08
---------T 2 gaist users 0 Mar 14 16:44 /bricks/34f0327b-baf0-46d8-b090-377302191cba/brick/data/queues/label-full-defect-qa-20191107.json
---------T 2 gaist users 0 Feb 14 20:56 /bricks/f0a8391e-0924-4eeb-8616-c066d8e1e566/brick/data/queues/label-full-defect-qa-20191107.json

xatts for files on bricks:

dc4-data-01:

# file: bricks/062a0def-2478-452b-9544-788cd8c7bc96/brick/data/queues/label-full-defect-qa-20191107.json
trusted.ec.config=0sAAAIBgIAAgA=
trusted.ec.dirty=0sAAAAAAAAAAAAAAAAAAAAAA==
trusted.ec.size=0sAAAAAAAvGuk=
trusted.ec.version=0sAAAAAAAAABkAAAAAAAAAGw==
trusted.gfid=0saLwZrkpyTj2aYBPSGy/wxg==
trusted.gfid2path.051d3b8ac52e2061="bfa08f7f-7237-4ae4-9ed1-f6963530efb6/label-full-defect-qa-20191107.json"
trusted.glusterfs.mdata=0sAQAAAAAAAAAAAAAAAGIKwfUAAAAAFq/iiwAAAABdxBaVAAAAACWH+KAAAAAAXcQWlQAAAAAKWbgg

# file: bricks/1c9a05b3-a21b-4afb-abe2-620e6093863a/brick/data/queues/label-full-defect-qa-20191107.json
trusted.ec.config=0sAAAIBgIAAgA=
trusted.ec.dirty=0sAAAAAAAAAAAAAAAAAAAAAA==
trusted.ec.size=0sAAAAAAAAAAA=
trusted.ec.version=0sAAAAAAAAAAIAAAAAAAAAAQ==
trusted.gfid=0saLwZrkpyTj2aYBPSGy/wxg==
trusted.gfid2path.051d3b8ac52e2061="bfa08f7f-7237-4ae4-9ed1-f6963530efb6/label-full-defect-qa-20191107.json"
trusted.glusterfs.dht.linkto="dc4-readdir-ahead-47"

# file: bricks/9e893d26-cd87-490e-9d6e-83c6fecebf54/brick/data/queues/label-full-defect-qa-20191107.json
trusted.ec.config=0sAAAIBgIAAgA=
trusted.ec.dirty=0sAAAAAAAAAAAAAAAAAAAAAA==
trusted.ec.size=0sAAAAAAAAAAA=
trusted.ec.version=0sAAAAAAAAAAQAAAAAAAAABg==
trusted.gfid=0saLwZrkpyTj2aYBPSGy/wxg==
trusted.gfid2path.051d3b8ac52e2061="bfa08f7f-7237-4ae4-9ed1-f6963530efb6/label-full-defect-qa-20191107.json"
trusted.glusterfs.dht.linkto="dc4-readdir-ahead-28"
trusted.glusterfs.mdata=0sAQAAAAAAAAAAAAAAAGIKwfUAAAAAFq/iiwAAAABdxBaVAAAAACWH+KAAAAAAXcQWlQAAAAAKWbgg

dc4-data-02:

# file: bricks/016d9174-2f14-417e-b398-ea10e2200590/brick/data/queues/label-full-defect-qa-20191107.json
trusted.ec.config=0sAAAIBgIAAgA=
trusted.ec.size=0sAAAAAAAAAAA=
trusted.ec.version=0sAAAAAAAAAAAAAAAAAAAAAQ==
trusted.gfid=0saLwZrkpyTj2aYBPSGy/wxg==
trusted.gfid2path.051d3b8ac52e2061="bfa08f7f-7237-4ae4-9ed1-f6963530efb6/label-full-defect-qa-20191107.json"
trusted.glusterfs.dht.linkto="dc4-readdir-ahead-28"
trusted.glusterfs.mdata=0sAQAAAAAAAAAAAAAAAGIKwfQAAAAAHJYtWQAAAABiCsH0AAAAAByWLVkAAAAAYgrB9AAAAAAcli1Z

# file: bricks/30efd892-2652-4dfc-9393-4bc82191c1cf/brick/data/queues/label-full-defect-qa-20191107.json
trusted.ec.config=0sAAAIBgIAAgA=
trusted.ec.dirty=0sAAAAAAAAAAAAAAAAAAAAAA==
trusted.ec.size=0sAAAAAAAAAAA=
trusted.ec.version=0sAAAAAAAAAAIAAAAAAAAAAQ==
trusted.gfid=0saLwZrkpyTj2aYBPSGy/wxg==
trusted.gfid2path.051d3b8ac52e2061="bfa08f7f-7237-4ae4-9ed1-f6963530efb6/label-full-defect-qa-20191107.json"
trusted.glusterfs.dht.linkto="dc4-readdir-ahead-47"

# file: bricks/a2b188f5-f239-4ccd-8778-5963734fad8f/brick/data/queues/label-full-defect-qa-20191107.json
trusted.ec.config=0sAAAIBgIAAgA=
trusted.ec.dirty=0sAAAAAAAAAAAAAAAAAAAAAA==
trusted.ec.size=0sAAAAAAAvGuk=
trusted.ec.version=0sAAAAAAAAABkAAAAAAAAAGw==
trusted.gfid=0saLwZrkpyTj2aYBPSGy/wxg==
trusted.gfid2path.051d3b8ac52e2061="bfa08f7f-7237-4ae4-9ed1-f6963530efb6/label-full-defect-qa-20191107.json"
trusted.glusterfs.mdata=0sAQAAAAAAAAAAAAAAAGIKwfUAAAAAFq/iiwAAAABdxBaVAAAAACWH+KAAAAAAXcQWlQAAAAAKWbgg

# file: bricks/fce7e9b3-4ff4-4e01-ae02-8d27d175485c/brick/data/queues/label-full-defect-qa-20191107.json
trusted.ec.config=0sAAAIBgIAAgA=
trusted.ec.dirty=0sAAAAAAAAAAAAAAAAAAAAAA==
trusted.ec.size=0sAAAAAAAAAAA=
trusted.ec.version=0sAAAAAAAAAAQAAAAAAAAABg==
trusted.gfid=0saLwZrkpyTj2aYBPSGy/wxg==
trusted.gfid2path.051d3b8ac52e2061="bfa08f7f-7237-4ae4-9ed1-f6963530efb6/label-full-defect-qa-20191107.json"
trusted.glusterfs.dht.linkto="dc4-readdir-ahead-28"
trusted.glusterfs.mdata=0sAQAAAAAAAAAAAAAAAGIKwfUAAAAAFq/iiwAAAABdxBaVAAAAACWH+KAAAAAAXcQWlQAAAAAKWbgg

dc4-data-03:

# file: bricks/143427df-c5b1-4b4b-b38a-9f337418c187/brick/data/queues/label-full-defect-qa-20191107.json
trusted.ec.config=0sAAAIBgIAAgA=
trusted.ec.dirty=0sAAAAAAAAAAAAAAAAAAAAAA==
trusted.ec.size=0sAAAAAAAAAAA=
trusted.ec.version=0sAAAAAAAAAAQAAAAAAAAABg==
trusted.gfid=0saLwZrkpyTj2aYBPSGy/wxg==
trusted.gfid2path.051d3b8ac52e2061="bfa08f7f-7237-4ae4-9ed1-f6963530efb6/label-full-defect-qa-20191107.json"
trusted.glusterfs.dht.linkto="dc4-readdir-ahead-28"
trusted.glusterfs.mdata=0sAQAAAAAAAAAAAAAAAGIKwfUAAAAAFq/iiwAAAABdxBaVAAAAACWH+KAAAAAAXcQWlQAAAAAKWbgg

# file: bricks/1842d244-3581-43e1-ae65-cc0247fd2d0a/brick/data/queues/label-full-defect-qa-20191107.json
trusted.ec.config=0sAAAIBgIAAgA=
trusted.ec.dirty=0sAAAAAAAAAAAAAAAAAAAAAA==
trusted.ec.size=0sAAAAAAAvGuk=
trusted.ec.version=0sAAAAAAAAABkAAAAAAAAAGw==
trusted.gfid=0saLwZrkpyTj2aYBPSGy/wxg==
trusted.gfid2path.051d3b8ac52e2061="bfa08f7f-7237-4ae4-9ed1-f6963530efb6/label-full-defect-qa-20191107.json"
trusted.glusterfs.mdata=0sAQAAAAAAAAAAAAAAAGIKwfUAAAAAFq/iiwAAAABdxBaVAAAAACWH+KAAAAAAXcQWlQAAAAAKWbgg

# file: bricks/7c73f6a5-17f2-4c68-a8a1-b16d5b00eb63/brick/data/queues/label-full-defect-qa-20191107.json
trusted.ec.config=0sAAAIBgIAAgA=
trusted.ec.dirty=0sAAAAAAAAAAAAAAAAAAAAAA==
trusted.ec.size=0sAAAAAAAAAAA=
trusted.ec.version=0sAAAAAAAAAAIAAAAAAAAAAQ==
trusted.gfid=0saLwZrkpyTj2aYBPSGy/wxg==
trusted.gfid2path.051d3b8ac52e2061="bfa08f7f-7237-4ae4-9ed1-f6963530efb6/label-full-defect-qa-20191107.json"
trusted.glusterfs.dht.linkto="dc4-readdir-ahead-47"

dc4-data-04:

# file: bricks/9152f266-5c4a-4c89-a716-4aea6eac2edc/brick/data/queues/label-full-defect-qa-20191107.json
trusted.ec.config=0sAAAIBgIAAgA=
trusted.ec.size=0sAAAAAAAAAAA=
trusted.ec.version=0sAAAAAAAAAAAAAAAAAAAAAQ==
trusted.gfid=0saLwZrkpyTj2aYBPSGy/wxg==
trusted.gfid2path.051d3b8ac52e2061="bfa08f7f-7237-4ae4-9ed1-f6963530efb6/label-full-defect-qa-20191107.json"
trusted.glusterfs.dht.linkto="dc4-readdir-ahead-28"
trusted.glusterfs.mdata=0sAQAAAAAAAAAAAAAAAGIKwfQAAAAAHJYtWQAAAABiCsH0AAAAAByWLVkAAAAAYgrB9AAAAAAcli1Z

# file: bricks/b38f81b9-bd97-4551-9eb2-6303c15c3724/brick/data/queues/label-full-defect-qa-20191107.json
trusted.ec.config=0sAAAIBgIAAgA=
trusted.ec.dirty=0sAAAAAAAAAAAAAAAAAAAAAA==
trusted.ec.size=0sAAAAAAAvGuk=
trusted.ec.version=0sAAAAAAAAABkAAAAAAAAAGw==
trusted.gfid=0saLwZrkpyTj2aYBPSGy/wxg==
trusted.gfid2path.051d3b8ac52e2061="bfa08f7f-7237-4ae4-9ed1-f6963530efb6/label-full-defect-qa-20191107.json"
trusted.glusterfs.mdata=0sAQAAAAAAAAAAAAAAAGIKwfUAAAAAFq/iiwAAAABdxBaVAAAAACWH+KAAAAAAXcQWlQAAAAAKWbgg

# file: bricks/ff78d3c6-0ce2-4c7a-98ed-8b7cc1d588be/brick/data/queues/label-full-defect-qa-20191107.json
trusted.ec.config=0sAAAIBgIAAgA=
trusted.ec.dirty=0sAAAAAAAAAAAAAAAAAAAAAA==
trusted.ec.size=0sAAAAAAAAAAA=
trusted.ec.version=0sAAAAAAAAAAQAAAAAAAAABg==
trusted.gfid=0saLwZrkpyTj2aYBPSGy/wxg==
trusted.gfid2path.051d3b8ac52e2061="bfa08f7f-7237-4ae4-9ed1-f6963530efb6/label-full-defect-qa-20191107.json"
trusted.glusterfs.dht.linkto="dc4-readdir-ahead-28"
trusted.glusterfs.mdata=0sAQAAAAAAAAAAAAAAAGIKwfUAAAAAFq/iiwAAAABdxBaVAAAAACWH+KAAAAAAXcQWlQAAAAAKWbgg

dc4-data-05:

# file: bricks/43b04ce5-0f35-4eab-8ca6-aaaf3c0a91cc/brick/data/queues/label-full-defect-qa-20191107.json
trusted.ec.config=0sAAAIBgIAAgA=
trusted.ec.dirty=0sAAAAAAAAAAAAAAAAAAAAAA==
trusted.ec.size=0sAAAAAAAAAAA=
trusted.ec.version=0sAAAAAAAAAAQAAAAAAAAABg==
trusted.gfid=0saLwZrkpyTj2aYBPSGy/wxg==
trusted.gfid2path.051d3b8ac52e2061="bfa08f7f-7237-4ae4-9ed1-f6963530efb6/label-full-defect-qa-20191107.json"
trusted.glusterfs.dht.linkto="dc4-readdir-ahead-28"
trusted.glusterfs.mdata=0sAQAAAAAAAAAAAAAAAGIKwfUAAAAAFq/iiwAAAABdxBaVAAAAACWH+KAAAAAAXcQWlQAAAAAKWbgg

# file: bricks/4de9a7c4-e907-4a3d-b826-01d3aee7e45d/brick/data/queues/label-full-defect-qa-20191107.json
trusted.ec.config=0sAAAIBgIAAgA=
trusted.ec.dirty=0sAAAAAAAAAAAAAAAAAAAAAA==
trusted.ec.size=0sAAAAAAAAAAA=
trusted.ec.version=0sAAAAAAAAAAIAAAAAAAAAAQ==
trusted.gfid=0saLwZrkpyTj2aYBPSGy/wxg==
trusted.gfid2path.051d3b8ac52e2061="bfa08f7f-7237-4ae4-9ed1-f6963530efb6/label-full-defect-qa-20191107.json"
trusted.glusterfs.dht.linkto="dc4-readdir-ahead-47"

# file: bricks/61398509-55cd-4a31-882a-0d487eb2d985/brick/data/queues/label-full-defect-qa-20191107.json
trusted.ec.config=0sAAAIBgIAAgA=
trusted.ec.size=0sAAAAAAAAAAA=
trusted.ec.version=0sAAAAAAAAAAAAAAAAAAAAAQ==
trusted.gfid=0saLwZrkpyTj2aYBPSGy/wxg==
trusted.gfid2path.051d3b8ac52e2061="bfa08f7f-7237-4ae4-9ed1-f6963530efb6/label-full-defect-qa-20191107.json"
trusted.glusterfs.dht.linkto="dc4-readdir-ahead-28"
trusted.glusterfs.mdata=0sAQAAAAAAAAAAAAAAAGIKwfQAAAAAHJYtWQAAAABiCsH0AAAAAByWLVkAAAAAYgrB9AAAAAAcli1Z

dc4-data-06:

# file: bricks/01b4239c-072c-446e-bece-aae004cf023b/brick/data/queues/label-full-defect-qa-20191107.json
trusted.ec.config=0sAAAIBgIAAgA=
trusted.ec.dirty=0sAAAAAAAAAAAAAAAAAAAAAA==
trusted.ec.size=0sAAAAAAAAAAA=
trusted.ec.version=0sAAAAAAAAAAIAAAAAAAAAAQ==
trusted.gfid=0saLwZrkpyTj2aYBPSGy/wxg==
trusted.gfid2path.051d3b8ac52e2061="bfa08f7f-7237-4ae4-9ed1-f6963530efb6/label-full-defect-qa-20191107.json"
trusted.glusterfs.dht.linkto="dc4-readdir-ahead-47"

# file: bricks/53cf6d0c-d188-4035-bd53-eb583a42d5a9/brick/data/queues/label-full-defect-qa-20191107.json
trusted.ec.config=0sAAAIBgIAAgA=
trusted.ec.size=0sAAAAAAAAAAA=
trusted.ec.version=0sAAAAAAAAAAAAAAAAAAAAAQ==
trusted.gfid=0saLwZrkpyTj2aYBPSGy/wxg==
trusted.gfid2path.051d3b8ac52e2061="bfa08f7f-7237-4ae4-9ed1-f6963530efb6/label-full-defect-qa-20191107.json"
trusted.glusterfs.dht.linkto="dc4-readdir-ahead-28"
trusted.glusterfs.mdata=0sAQAAAAAAAAAAAAAAAGIKwfQAAAAAHJYtWQAAAABiCsH0AAAAAByWLVkAAAAAYgrB9AAAAAAcli1Z

# file: bricks/9a3d92a4-49b5-449a-97ea-e49f66f9b672/brick/data/queues/label-full-defect-qa-20191107.json
trusted.ec.config=0sAAAIBgIAAgA=
trusted.ec.dirty=0sAAAAAAAAAAAAAAAAAAAAAA==
trusted.ec.size=0sAAAAAAAAAAA=
trusted.ec.version=0sAAAAAAAAAAQAAAAAAAAABg==
trusted.gfid=0saLwZrkpyTj2aYBPSGy/wxg==
trusted.gfid2path.051d3b8ac52e2061="bfa08f7f-7237-4ae4-9ed1-f6963530efb6/label-full-defect-qa-20191107.json"
trusted.glusterfs.dht.linkto="dc4-readdir-ahead-28"
trusted.glusterfs.mdata=0sAQAAAAAAAAAAAAAAAGIKwfUAAAAAFq/iiwAAAABdxBaVAAAAACWH+KAAAAAAXcQWlQAAAAAKWbgg

# file: bricks/c7b54765-0867-4349-8976-2fd0a8bde7a7/brick/data/queues/label-full-defect-qa-20191107.json
trusted.ec.config=0sAAAIBgIAAgA=
trusted.ec.dirty=0sAAAAAAAAAAAAAAAAAAAAAA==
trusted.ec.size=0sAAAAAAAvGuk=
trusted.ec.version=0sAAAAAAAAABkAAAAAAAAAGw==
trusted.gfid=0saLwZrkpyTj2aYBPSGy/wxg==
trusted.gfid2path.051d3b8ac52e2061="bfa08f7f-7237-4ae4-9ed1-f6963530efb6/label-full-defect-qa-20191107.json"
trusted.glusterfs.mdata=0sAQAAAAAAAAAAAAAAAGIKwfUAAAAAFq/iiwAAAABdxBaVAAAAACWH+KAAAAAAXcQWlQAAAAAKWbgg

dc4-data-07:

# file: bricks/628a0acd-2d16-49c1-bf8c-82105f412708/brick/data/queues/label-full-defect-qa-20191107.json
trusted.ec.config=0sAAAIBgIAAgA=
trusted.ec.size=0sAAAAAAAAAAA=
trusted.ec.version=0sAAAAAAAAAAAAAAAAAAAAAQ==
trusted.gfid=0saLwZrkpyTj2aYBPSGy/wxg==
trusted.gfid2path.051d3b8ac52e2061="bfa08f7f-7237-4ae4-9ed1-f6963530efb6/label-full-defect-qa-20191107.json"
trusted.glusterfs.dht.linkto="dc4-readdir-ahead-28"
trusted.glusterfs.mdata=0sAQAAAAAAAAAAAAAAAGIKwfQAAAAAHJYtWQAAAABiCsH0AAAAAByWLVkAAAAAYgrB9AAAAAAcli1Z

# file: bricks/8ecf3f7e-a5c5-48fd-a63e-0dca2d4c2957/brick/data/queues/label-full-defect-qa-20191107.json
trusted.ec.config=0sAAAIBgIAAgA=
trusted.ec.dirty=0sAAAAAAAAAAAAAAAAAAAAAA==
trusted.ec.size=0sAAAAAAAvGuk=
trusted.ec.version=0sAAAAAAAAABkAAAAAAAAAGw==
trusted.gfid=0saLwZrkpyTj2aYBPSGy/wxg==
trusted.gfid2path.051d3b8ac52e2061="bfa08f7f-7237-4ae4-9ed1-f6963530efb6/label-full-defect-qa-20191107.json"
trusted.glusterfs.mdata=0sAQAAAAAAAAAAAAAAAGIKwfUAAAAAFq/iiwAAAABdxBaVAAAAACWH+KAAAAAAXcQWlQAAAAAKWbgg

dc4-data-08:

# file: bricks/34f0327b-baf0-46d8-b090-377302191cba/brick/data/queues/label-full-defect-qa-20191107.json
trusted.ec.config=0sAAAIBgIAAgA=
trusted.ec.dirty=0sAAAAAAAAAAAAAAAAAAAAAA==
trusted.ec.size=0sAAAAAAAAAAA=
trusted.ec.version=0sAAAAAAAAAAIAAAAAAAAAAQ==
trusted.gfid=0saLwZrkpyTj2aYBPSGy/wxg==
trusted.gfid2path.051d3b8ac52e2061="bfa08f7f-7237-4ae4-9ed1-f6963530efb6/label-full-defect-qa-20191107.json"
trusted.glusterfs.dht.linkto="dc4-readdir-ahead-47"

# file: bricks/f0a8391e-0924-4eeb-8616-c066d8e1e566/brick/data/queues/label-full-defect-qa-20191107.json
trusted.ec.config=0sAAAIBgIAAgA=
trusted.ec.size=0sAAAAAAAAAAA=
trusted.ec.version=0sAAAAAAAAAAAAAAAAAAAAAQ==
trusted.gfid=0saLwZrkpyTj2aYBPSGy/wxg==
trusted.gfid2path.051d3b8ac52e2061="bfa08f7f-7237-4ae4-9ed1-f6963530efb6/label-full-defect-qa-20191107.json"
trusted.glusterfs.dht.linkto="dc4-readdir-ahead-28"
trusted.glusterfs.mdata=0sAQAAAAAAAAAAAAAAAGIKwfQAAAAAHJYtWQAAAABiCsH0AAAAAByWLVkAAAAAYgrB9AAAAAAcli1Z

[mohit84]

@tshacked Can you please check this

[tshacked]

/var/log/glusterfs/bricks/bricks-8e953fb6-c0d2-40e0-831d-f67965186915-brick.log.1:[2022-03-14 16:24:47.687322 +0000] E [MSGID: 113001] [posix-helpers.c:1273:posix_handle_pair] 0-dc4-posix: /bricks/8e953fb6-c0d2-40e0-831d-f67965186915/brick/.glusterfs/4a/d4/4ad4d646-6f36-4849-bd17-0d12fd67152d: key:glusterfs.inodelk-dom-countflags: 0 length:16 [Operation not supported] /var/log/glusterfs/bricks/bricks-8e953fb6-c0d2-40e0-831d-f67965186915-brick.log.1:[2022-03-14 16:24:48.567089 +0000] E [MSGID: 113001] [posix-helpers.c:1273:posix_handle_pair] 0-dc4-posix: /bricks/8e953fb6-c0d2-40e0-831d-f67965186915/brick/.glusterfs/4a/d4/4ad4d646-6f36-4849-bd17-0d12fd67152d: key:glusterfs.inodelk-dom-countflags: 0 length:16 [Operation not supported]

My understanding is that this xattr (glusterfs.inodelk-dom-count) could not be written (because it doesnt start with trusted, user, system, or security) and that actually these should not be attempted as this is filtered: https://github.com/gluster/glusterfs/blob/v9.3/xlators/storage/posix/src/posix-helpers.c#L1853

Hey @smremde

Above xattr glusterfs.inodelk-dom-count is related to inodelk count in a given domain and being used by the EC/AFR layers. Looks it was triggered from posix_setxattr path and that's why it was not filtered https://github.com/gluster/glusterfs/blob/cf0620a1faf75224e0ef77ce2e7472cff60b2125/xlators/storage/posix/src/posix-inode-fd-ops.c#L2801

As the above volume includes 8 nodes: dc4-data-0{1..8}, it means that rebalance process ran on every node. It will help to check if extra logging exists for label-full-defect-qa-20191107.json (or its gfid) on the other rebalance logs and also what was logged before the error. Please add rebalance logs from all nodes or the relevant logging.

Regards, Tamar

[smremde]

Hi @tshacked - thanks for responding.

The file appears in 3 logs (unfiltered, 5 lines before and after):

dc4-data-06

[2022-03-14 16:44:21.397060 +0000] W [MSGID: 109023] [dht-rebalance.c:891:__dht_check_free_space] 0-dc4-dht: data movement of file {blocks:34224 name:(/data/queues/label-full-defect-ml.20200910.jsonl.gz)} would result in dst node (dc4-readdir-ahead-9:624388416) having lower disk space than the source node (dc4-readdir-ahead-52:17601594912).Skipping file.
[2022-03-14 16:44:21.404676 +0000] W [MSGID: 109023] [dht-rebalance.c:891:__dht_check_free_space] 0-dc4-dht: data movement of file {blocks:22072 name:(/data/queues/label-major-full-ml-20191126.json)} would result in dst node (dc4-readdir-ahead-25:936788864) having lower disk space than the source node (dc4-readdir-ahead-49:18028680160).Skipping file.
[2022-03-14 16:44:21.404817 +0000] I [MSGID: 109121] [dht-rebalance.c:2836:gf_defrag_migrate_single_file] 0-dc4-dht: File migration skipped for /data/queues/label-full-defect-ml.20200910.jsonl.gz.
[2022-03-14 16:44:21.409582 +0000] I [dht-rebalance.c:1536:dht_migrate_file] 0-dc4-dht: /data/queues/label-full-defect-ml.20200910.jsonl.gz: attempting to move from dc4-readdir-ahead-52 to dc4-readdir-ahead-9
[2022-03-14 16:44:21.414537 +0000] I [MSGID: 109121] [dht-rebalance.c:2836:gf_defrag_migrate_single_file] 0-dc4-dht: File migration skipped for /data/queues/label-major-full-ml-20191126.json.
[2022-03-14 16:44:21.437990 +0000] I [dht-rebalance.c:1536:dht_migrate_file] 0-dc4-dht: /data/queues/label-full-defect-qa-20191107.json: attempting to move from dc4-readdir-ahead-47 to dc4-readdir-ahead-16
[2022-03-14 16:44:21.477208 +0000] W [MSGID: 109023] [dht-rebalance.c:891:__dht_check_free_space] 0-dc4-dht: data movement of file {blocks:34224 name:(/data/queues/label-full-defect-ml.20200910.jsonl.gz)} would result in dst node (dc4-readdir-ahead-9:624388416) having lower disk space than the source node (dc4-readdir-ahead-52:17601594912).Skipping file.
[2022-03-14 16:44:21.482210 +0000] I [MSGID: 109121] [dht-rebalance.c:2836:gf_defrag_migrate_single_file] 0-dc4-dht: File migration skipped for /data/queues/label-full-defect-ml.20200910.jsonl.gz.
[2022-03-14 16:44:21.501858 +0000] W [MSGID: 109023] [dht-rebalance.c:891:__dht_check_free_space] 0-dc4-dht: data movement of file {blocks:0 name:(/data/queues/label-full-defect-qa-20191107.json)} would result in dst node (dc4-readdir-ahead-16:619448224) having lower disk space than the source node (dc4-readdir-ahead-47:18142017952).Skipping file.
[2022-03-14 16:44:21.575604 +0000] I [dht-rebalance.c:1536:dht_migrate_file] 0-dc4-dht: /data/queues/checkoutput-complete.20191018.jsonl.gz: attempting to move from dc4-readdir-ahead-47 to dc4-readdir-ahead-0
[2022-03-14 16:44:21.576804 +0000] I [MSGID: 109121] [dht-rebalance.c:2836:gf_defrag_migrate_single_file] 0-dc4-dht: File migration skipped for /data/queues/label-full-defect-qa-20191107.json.
[2022-03-14 16:44:21.611490 +0000] W [MSGID: 109023] [dht-rebalance.c:891:__dht_check_free_space] 0-dc4-dht: data movement of file {blocks:0 name:(/data/queues/checkoutput-complete.20191018.jsonl.gz)} would result in dst node (dc4-readdir-ahead-0:624256640) having lower disk space than the source node (dc4-readdir-ahead-47:18142017952).Skipping file.
[2022-03-14 16:44:21.624506 +0000] I [MSGID: 109121] [dht-rebalance.c:2836:gf_defrag_migrate_single_file] 0-dc4-dht: File migration skipped for /data/queues/checkoutput-complete.20191018.jsonl.gz.
[2022-03-14 16:44:21.857401 +0000] I [MSGID: 109079] [dht-common.c:5931:dht_setxattr] 0-dc4-dht: fixing the layout of /data/rinex/2019/294
[2022-03-14 16:44:21.923396 +0000] I [dht-rebalance.c:3280:gf_defrag_process_dir] 0-dc4-dht: migrate data called on /data/rinex/2019/294
[2022-03-14 16:44:22.027339 +0000] I [dht-rebalance.c:1536:dht_migrate_file] 0-dc4-dht: /data/rinex/2019/294/edin294h.19o: attempting to move from dc4-readdir-ahead-47 to dc4-readdir-ahead-8

dc4-data-07   

[2022-03-14 16:24:47.731418 +0000] I [dht-rebalance.c:3280:gf_defrag_process_dir] 0-dc4-dht: migrate data called on /data/queues
[2022-03-14 16:24:47.741666 +0000] I [MSGID: 0] [dht-rebalance.c:1741:dht_migrate_file] 0-dc4-dht: destination for file - /data/queues/20211214/label-cadent-360-verify.jsonl.gz is changed to - dc4-readdir-ahead-47
[2022-03-14 16:24:47.784682 +0000] E [MSGID: 109023] [dht-rebalance.c:2014:dht_migrate_file] 0-dc4-dht: failed to set xattr on /data/queues/20211214/label-cadent-360-verify.jsonl.gz in dc4-readdir-ahead-28 [Operation not supported]
[2022-03-14 16:24:47.788360 +0000] E [MSGID: 109023] [dht-rebalance.c:2859:gf_defrag_migrate_single_file] 0-dc4-dht: migrate-data failed for /data/queues/20211214/label-cadent-360-verify.jsonl.gz [Operation not supported]
[2022-03-14 16:24:47.796198 +0000] I [dht-rebalance.c:3503:gf_defrag_process_dir] 0-dc4-dht: Migration operation on dir /data/queues took 0.06 secs
[2022-03-14 16:24:47.957556 +0000] I [dht-rebalance.c:1536:dht_migrate_file] 0-dc4-dht: /data/queues/label-full-defect-qa-20191107.json: attempting to move from dc4-readdir-ahead-28 to dc4-readdir-ahead-16
[2022-03-14 16:24:47.982471 +0000] I [dht-rebalance.c:1536:dht_migrate_file] 0-dc4-dht: /data/queues/label-full-defect-ml-20191211.json: attempting to move from dc4-readdir-ahead-52 to dc4-readdir-ahead-29
[2022-03-14 16:24:48.053314 +0000] W [MSGID: 0] [dht-rebalance.c:922:__dht_check_free_space] 0-dc4-dht: Write will cross min-free-disk for file - /data/queues/label-full-defect-qa-20191107.json on subvol - dc4-readdir-ahead-16. Looking for new subvol
[2022-03-14 16:24:48.053455 +0000] I [MSGID: 0] [dht-rebalance.c:979:__dht_check_free_space] 0-dc4-dht: new target found - dc4-readdir-ahead-47 for file - /data/queues/label-full-defect-qa-20191107.json
[2022-03-14 16:24:48.064491 +0000] W [MSGID: 109023] [dht-rebalance.c:891:__dht_check_free_space] 0-dc4-dht: data movement of file {blocks:101484 name:(/data/queues/label-full-defect-ml-20191211.json)} would result in dst node (dc4-readdir-ahead-29:932901568) having lower disk space than the source node (dc4-readdir-ahead-52:17603706752).Skipping file.
[2022-03-14 16:24:48.065909 +0000] I [MSGID: 109121] [dht-rebalance.c:2836:gf_defrag_migrate_single_file] 0-dc4-dht: File migration skipped for /data/queues/label-full-defect-ml-20191211.json.
[2022-03-14 16:24:48.192489 +0000] I [MSGID: 0] [dht-rebalance.c:1741:dht_migrate_file] 0-dc4-dht: destination for file - /data/queues/label-full-defect-qa-20191107.json is changed to - dc4-readdir-ahead-47
[2022-03-14 16:24:48.385145 +0000] E [MSGID: 109023] [dht-rebalance.c:2014:dht_migrate_file] 0-dc4-dht: failed to set xattr on /data/queues/label-full-defect-qa-20191107.json in dc4-readdir-ahead-16 [Operation not supported]
[2022-03-14 16:24:48.388013 +0000] E [MSGID: 109023] [dht-rebalance.c:2859:gf_defrag_migrate_single_file] 0-dc4-dht: migrate-data failed for /data/queues/label-full-defect-qa-20191107.json [Operation not supported]
[2022-03-14 16:24:48.974474 +0000] I [dht-rebalance.c:1536:dht_migrate_file] 0-dc4-dht: /data/queues/label-full-defect-ml-20200312.json: attempting to move from dc4-readdir-ahead-49 to dc4-readdir-ahead-27
[2022-03-14 16:24:49.026502 +0000] I [dht-rebalance.c:1536:dht_migrate_file] 0-dc4-dht: /data/queues/label-full-defect-ml-20200312.json: attempting to move from dc4-readdir-ahead-49 to dc4-readdir-ahead-27
[2022-03-14 16:24:49.031406 +0000] W [MSGID: 109023] [dht-rebalance.c:891:__dht_check_free_space] 0-dc4-dht: data movement of file {blocks:178012 name:(/data/queues/label-full-defect-ml-20200312.json)} would result in dst node (dc4-readdir-ahead-27:937113952) having lower disk space than the source node (dc4-readdir-ahead-49:18030758432).Skipping file.
[2022-03-14 16:24:49.032990 +0000] I [MSGID: 109121] [dht-rebalance.c:2836:gf_defrag_migrate_single_file] 0-dc4-dht: File migration skipped for /data/queues/label-full-defect-ml-20200312.json.
[2022-03-14 16:24:49.036280 +0000] W [MSGID: 109023] [dht-rebalance.c:891:__dht_check_free_space] 0-dc4-dht: data movement of file {blocks:178012 name:(/data/queues/label-full-defect-ml-20200312.json)} would result in dst node (dc4-readdir-ahead-27:937113952) having lower disk space than the source node (dc4-readdir-ahead-49:18030758432).Skipping file.

dc4-data-08

[2022-03-14 16:24:56.968584 +0000] I [dht-rebalance.c:3280:gf_defrag_process_dir] 0-dc4-dht: migrate data called on /data/queues/20220111
[2022-03-14 16:24:57.030479 +0000] I [dht-rebalance.c:3503:gf_defrag_process_dir] 0-dc4-dht: Migration operation on dir /data/queues/20220111 took 0.06 secs
[2022-03-14 16:24:57.160619 +0000] I [MSGID: 109079] [dht-common.c:5931:dht_setxattr] 0-dc4-dht: fixing the layout of /data/queues
[2022-03-14 16:24:57.302636 +0000] I [dht-rebalance.c:3280:gf_defrag_process_dir] 0-dc4-dht: migrate data called on /data/queues
[2022-03-14 16:24:57.360812 +0000] I [dht-rebalance.c:3503:gf_defrag_process_dir] 0-dc4-dht: Migration operation on dir /data/queues took 0.06 secs
[2022-03-14 16:24:57.398743 +0000] I [dht-rebalance.c:1536:dht_migrate_file] 0-dc4-dht: /data/queues/label-full-defect-qa-20191107.json: attempting to move from dc4-readdir-ahead-47 to dc4-readdir-ahead-16
[2022-03-14 16:24:57.419638 +0000] I [dht-rebalance.c:1536:dht_migrate_file] 0-dc4-dht: /data/queues/label-full-defect-qa-20191107.json: attempting to move from dc4-readdir-ahead-47 to dc4-readdir-ahead-16
[2022-03-14 16:24:57.422705 +0000] W [MSGID: 109023] [dht-rebalance.c:891:__dht_check_free_space] 0-dc4-dht: data movement of file {blocks:0 name:(/data/queues/label-full-defect-qa-20191107.json)} would result in dst node (dc4-readdir-ahead-16:619447680) having lower disk space than the source node (dc4-readdir-ahead-47:18183210208).Skipping file.
[2022-03-14 16:24:57.424220 +0000] I [MSGID: 109121] [dht-rebalance.c:2836:gf_defrag_migrate_single_file] 0-dc4-dht: File migration skipped for /data/queues/label-full-defect-qa-20191107.json.
[2022-03-14 16:24:57.427419 +0000] W [MSGID: 109023] [dht-rebalance.c:891:__dht_check_free_space] 0-dc4-dht: data movement of file {blocks:0 name:(/data/queues/label-full-defect-qa-20191107.json)} would result in dst node (dc4-readdir-ahead-16:619447680) having lower disk space than the source node (dc4-readdir-ahead-47:18183209728).Skipping file.
[2022-03-14 16:24:57.428619 +0000] I [MSGID: 109121] [dht-rebalance.c:2836:gf_defrag_migrate_single_file] 0-dc4-dht: File migration skipped for /data/queues/label-full-defect-qa-20191107.json.
[2022-03-14 16:24:57.494776 +0000] I [dht-rebalance.c:1536:dht_migrate_file] 0-dc4-dht: /data/queues/checkoutput-complete.20191018.jsonl.gz: attempting to move from dc4-readdir-ahead-37 to dc4-readdir-ahead-0
[2022-03-14 16:24:57.508318 +0000] W [MSGID: 0] [dht-rebalance.c:922:__dht_check_free_space] 0-dc4-dht: Write will cross min-free-disk for file - /data/queues/checkoutput-complete.20191018.jsonl.gz on subvol - dc4-readdir-ahead-0. Looking for new subvol
[2022-03-14 16:24:57.508364 +0000] I [MSGID: 0] [dht-rebalance.c:979:__dht_check_free_space] 0-dc4-dht: new target found - dc4-readdir-ahead-47 for file - /data/queues/checkoutput-complete.20191018.jsonl.gz
[2022-03-14 16:24:57.577952 +0000] I [MSGID: 0] [dht-rebalance.c:1741:dht_migrate_file] 0-dc4-dht: destination for file - /data/queues/checkoutput-complete.20191018.jsonl.gz is changed to - dc4-readdir-ahead-47
[2022-03-14 16:24:57.733762 +0000] E [MSGID: 109023] [dht-rebalance.c:2014:dht_migrate_file] 0-dc4-dht: failed to set xattr on /data/queues/checkoutput-complete.20191018.jsonl.gz in dc4-readdir-ahead-0 [Operation not supported]

filtered, in chrono order:

dc4-data-07:
[2022-03-14 16:24:47.957556 +0000] I [dht-rebalance.c:1536:dht_migrate_file] 0-dc4-dht: /data/queues/label-full-defect-qa-20191107.json: attempting to move from dc4-readdir-ahead-28 to dc4-readdir-ahead-16
[2022-03-14 16:24:48.053314 +0000] W [MSGID: 0] [dht-rebalance.c:922:__dht_check_free_space] 0-dc4-dht: Write will cross min-free-disk for file - /data/queues/label-full-defect-qa-20191107.json on subvol - dc4-readdir-ahead-16. Looking for new subvol
[2022-03-14 16:24:48.053455 +0000] I [MSGID: 0] [dht-rebalance.c:979:__dht_check_free_space] 0-dc4-dht: new target found - dc4-readdir-ahead-47 for file - /data/queues/label-full-defect-qa-20191107.json
[2022-03-14 16:24:48.192489 +0000] I [MSGID: 0] [dht-rebalance.c:1741:dht_migrate_file] 0-dc4-dht: destination for file - /data/queues/label-full-defect-qa-20191107.json is changed to - dc4-readdir-ahead-47
[2022-03-14 16:24:48.385145 +0000] E [MSGID: 109023] [dht-rebalance.c:2014:dht_migrate_file] 0-dc4-dht: failed to set xattr on /data/queues/label-full-defect-qa-20191107.json in dc4-readdir-ahead-16 [Operation not supported]
[2022-03-14 16:24:48.388013 +0000] E [MSGID: 109023] [dht-rebalance.c:2859:gf_defrag_migrate_single_file] 0-dc4-dht: migrate-data failed for /data/queues/label-full-defect-qa-20191107.json [Operation not supported]

dc4-data-08:
[2022-03-14 16:24:57.398743 +0000] I [dht-rebalance.c:1536:dht_migrate_file] 0-dc4-dht: /data/queues/label-full-defect-qa-20191107.json: attempting to move from dc4-readdir-ahead-47 to dc4-readdir-ahead-16
[2022-03-14 16:24:57.419638 +0000] I [dht-rebalance.c:1536:dht_migrate_file] 0-dc4-dht: /data/queues/label-full-defect-qa-20191107.json: attempting to move from dc4-readdir-ahead-47 to dc4-readdir-ahead-16
[2022-03-14 16:24:57.422705 +0000] W [MSGID: 109023] [dht-rebalance.c:891:__dht_check_free_space] 0-dc4-dht: data movement of file {blocks:0 name:(/data/queues/label-full-defect-qa-20191107.json)} would result in dst node (dc4-readdir-ahead-16:619447680) having lower disk space than the source node (dc4-readdir-ahead-47:18183210208).Skipping file.
[2022-03-14 16:24:57.424220 +0000] I [MSGID: 109121] [dht-rebalance.c:2836:gf_defrag_migrate_single_file] 0-dc4-dht: File migration skipped for /data/queues/label-full-defect-qa-20191107.json.
[2022-03-14 16:24:57.427419 +0000] W [MSGID: 109023] [dht-rebalance.c:891:__dht_check_free_space] 0-dc4-dht: data movement of file {blocks:0 name:(/data/queues/label-full-defect-qa-20191107.json)} would result in dst node (dc4-readdir-ahead-16:619447680) having lower disk space than the source node (dc4-readdir-ahead-47:18183209728).Skipping file.
[2022-03-14 16:24:57.428619 +0000] I [MSGID: 109121] [dht-rebalance.c:2836:gf_defrag_migrate_single_file] 0-dc4-dht: File migration skipped for /data/queues/label-full-defect-qa-20191107.json.

dc4-data-06:
[2022-03-14 16:44:21.437990 +0000] I [dht-rebalance.c:1536:dht_migrate_file] 0-dc4-dht: /data/queues/label-full-defect-qa-20191107.json: attempting to move from dc4-readdir-ahead-47 to dc4-readdir-ahead-16
[2022-03-14 16:44:21.501858 +0000] W [MSGID: 109023] [dht-rebalance.c:891:__dht_check_free_space] 0-dc4-dht: data movement of file {blocks:0 name:(/data/queues/label-full-defect-qa-20191107.json)} would result in dst node (dc4-readdir-ahead-16:619448224) having lower disk space than the source node (dc4-readdir-ahead-47:18142017952).Skipping file.
[2022-03-14 16:44:21.576804 +0000] I [MSGID: 109121] [dht-rebalance.c:2836:gf_defrag_migrate_single_file] 0-dc4-dht: File migration skipped for /data/queues/label-full-defect-qa-20191107.json.

There are no additional messages in the brick logs.

Hope this helps.

[smremde]

I noticed a mistake in the initial comment from the brick logs - this is the correct logs:

/var/log/glusterfs/bricks/bricks-1c9a05b3-a21b-4afb-abe2-620e6093863a-brick.log.4.gz:[2022-03-14 16:24:48.385321 +0000] E [MSGID: 113001] [posix-helpers.c:1273:posix_handle_pair] 0-dc4-posix: /bricks/1c9a05b3-a21b-4afb-abe2-620e6093863a/brick/.glusterfs/68/bc/68bc19ae-4a72-4e3d-9a60-13d21b2ff0c6: key:glusterfs.inodelk-dom-countflags: 0 length:16 [Operation not supported]
/var/log/glusterfs/bricks/bricks-30efd892-2652-4dfc-9393-4bc82191c1cf-brick.log.4.gz:[2022-03-14 16:24:48.384871 +0000] E [MSGID: 113001] [posix-helpers.c:1273:posix_handle_pair] 0-dc4-posix: /bricks/30efd892-2652-4dfc-9393-4bc82191c1cf/brick/.glusterfs/68/bc/68bc19ae-4a72-4e3d-9a60-13d21b2ff0c6: key:glusterfs.inodelk-dom-countflags: 0 length:16 [Operation not supported]
/var/log/glusterfs/bricks/bricks-7c73f6a5-17f2-4c68-a8a1-b16d5b00eb63-brick.log.4.gz:[2022-03-14 16:24:48.385173 +0000] E [MSGID: 113001] [posix-helpers.c:1273:posix_handle_pair] 0-dc4-posix: /bricks/7c73f6a5-17f2-4c68-a8a1-b16d5b00eb63/brick/.glusterfs/68/bc/68bc19ae-4a72-4e3d-9a60-13d21b2ff0c6: key:glusterfs.inodelk-dom-countflags: 0 length:16 [Operation not supported]
/var/log/glusterfs/bricks/bricks-4de9a7c4-e907-4a3d-b826-01d3aee7e45d-brick.log.4.gz:[2022-03-14 16:24:48.384902 +0000] E [MSGID: 113001] [posix-helpers.c:1273:posix_handle_pair] 0-dc4-posix: /bricks/4de9a7c4-e907-4a3d-b826-01d3aee7e45d/brick/.glusterfs/68/bc/68bc19ae-4a72-4e3d-9a60-13d21b2ff0c6: key:glusterfs.inodelk-dom-countflags: 0 length:16 [Operation not supported]
/var/log/glusterfs/bricks/bricks-01b4239c-072c-446e-bece-aae004cf023b-brick.log.4.gz:[2022-03-14 16:24:48.384330 +0000] E [MSGID: 113001] [posix-helpers.c:1273:posix_handle_pair] 0-dc4-posix: /bricks/01b4239c-072c-446e-bece-aae004cf023b/brick/.glusterfs/68/bc/68bc19ae-4a72-4e3d-9a60-13d21b2ff0c6: key:glusterfs.inodelk-dom-countflags: 0 length:16 [Operation not supported]
/var/log/glusterfs/bricks/bricks-34f0327b-baf0-46d8-b090-377302191cba-brick.log.4.gz:[2022-03-14 16:24:48.385132 +0000] E [MSGID: 113001] [posix-helpers.c:1273:posix_handle_pair] 0-dc4-posix: /bricks/34f0327b-baf0-46d8-b090-377302191cba/brick/.glusterfs/68/bc/68bc19ae-4a72-4e3d-9a60-13d21b2ff0c6: key:glusterfs.inodelk-dom-countflags: 0 length:16 [Operation not supported]

I have updated the original post.

By the way, this is happening on a good proportion of the files. Infact it seems that it happens every time it migrates a file (no data is migrated sucessfully) and the only reason some arent duplicated is because migration doesn't happen due to disk space constraints

[tshacked]

This particular files has 2 copies + 2 linkto

ls /mnt/volume/data/queues
...
-rw-rw-r--  1 user users   3087081 Nov  7  2019 label-full-defect-qa-20191107.json
-rw-rw-r--  1 user users         0 Nov  7  2019 label-full-defect-qa-20191107.json
...

Hey @smremde,

File data/queues/label-full-defect-qa-20191107.json has: 1 data-file, 1 linkto-file, 1 undefined linkto-file, 1 stale linkto-file (see details below). Since there's an "undefined linkto-file" it is also being listed on the MP (I could reproduce it).

In detail, the file is located on: (1) subvol-28: data-file from Nov 7 2019 (2) subvol-16: linkto-file pointing to (1) - according to rebalance logs here is the hashed location (3) subvol-47: undefined linkto-file: an empty file with a linkto xattr but it is missing the sticky bit 'T' (01000) and has the data-file permissions.

-rw-rw-r-- 2 gaist users      0 Mar 14 16:24 /bricks/9e893d26-cd87-490e-9d6e-83c6fecebf54/brick/data/queues/label-full-defect-qa-20191107.json  

Brick283: dc4-data-01:/bricks/9e893d26-cd87-490e-9d6e-83c6fecebf54/brick
# file: bricks/9e893d26-cd87-490e-9d6e-83c6fecebf54/brick/data/queues/label-full-defect-qa-20191107.json
trusted.glusterfs.dht.linkto="dc4-readdir-ahead-28"

(4) subvol-7: linkto-file points to data-file (1) (I think it is a stale linkto from Feb 14 20:56)

About the RC, I need to continue the investigation: it looks like an unclean exit from migration error or may be related to a race as another rebalance was trying to handle the file.
Rebalance dc4-data-07: created it as a new target for migration (due to space limitation), but after it was copied there was an error to set the glusterfs.inodelk-dom-count (still need to check if setting this xattr is needed from dht layer)

[2022-03-14 16:24:48.053314 +0000] W [MSGID: 0] [dht-rebalance.c:922:__dht_check_free_space] 0-dc4-dht: Write will cross min-free-disk for file - /data/queues/label-full-defect-qa-20191107.json on subvol - dc4-readdir-ahead-16. Looking for new subvol
[2022-03-14 16:24:48.053455 +0000] I [MSGID: 0] [dht-rebalance.c:979:__dht_check_free_space] 0-dc4-dht: new target found - dc4-readdir-ahead-47 for file - /data/queues/label-full-defect-qa-20191107.json
[2022-03-14 16:24:48.385145 +0000] E [MSGID: 109023] [dht-rebalance.c:2014:dht_migrate_file] 0-dc4-dht: failed to set xattr on /data/queues/label-full-defect-qa-20191107.json in dc4-readdir-ahead-16 [Operation not supported]

Rebalance dc4-data-08: find 2 files as candidate for migration (1-data file, 1-undefined-linkto), but "skip" the migration.

[2022-03-14 16:24:57.398743 +0000] I [dht-rebalance.c:1536:dht_migrate_file] 0-dc4-dht: /data/queues/label-full-defect-qa-20191107.json: attempting to move from dc4-readdir-ahead-47 to dc4-readdir-ahead-16
[2022-03-14 16:24:57.419638 +0000] I [dht-rebalance.c:1536:dht_migrate_file] 0-dc4-dht: /data/queues/label-full-defect-qa-20191107.json: attempting to move from dc4-readdir-ahead-47 to dc4-readdir-ahead-16
[2022-03-14 16:24:57.422705 +0000] W [MSGID: 109023] [dht-rebalance.c:891:__dht_check_free_space] 0-dc4-dht: data movement of file {blocks:0 name:(/data/queues/label-full-defect-qa-20191107.json)} would result in dst node (dc4-readdir-ahead-16:619447680) having lower disk space than the source node (dc4-readdir-ahead-47:18183210208).Skipping file.
[2022-03-14 16:24:57.424220 +0000] I [MSGID: 109121] [dht-rebalance.c:2836:gf_defrag_migrate_single_file] 0-dc4-dht: File migration skipped for /data/queues/label-full-defect-qa-20191107.json.
[2022-03-14 16:24:57.427419 +0000] W [MSGID: 109023] [dht-rebalance.c:891:__dht_check_free_space] 0-dc4-dht: data movement of file {blocks:0 name:(/data/queues/label-full-defect-qa-20191107.json)} would result in dst node (dc4-readdir-ahead-16:619447680) having lower disk space than the source node (dc4-readdir-ahead-47:18183209728).Skipping file.
[2022-03-14 16:24:57.428619 +0000] I [MSGID: 109121] [dht-rebalance.c:2836:gf_defrag_migrate_single_file] 0-dc4-dht: File migration skipped for /data/queues/label-full-defect-qa-20191107.json.

Manual fixing: For now, I think you can fix it by setting file (3) to be a well-defined linkto-file. (It could also be deleted as the hash-location seems to be subvol-16, but I think this is safer and when dht finds stale linko-files it deletes them, at least when lookup-optimize is off)

To correct the file set the sticky bit on all subvol-47 bricks:

#chmod 01000 /data/queues/label-full-defect-qa-20191107.json

#subvol-47 bricks
Brick283: dc4-data-01:/bricks/9e893d26-cd87-490e-9d6e-83c6fecebf54/brick
Brick284: dc4-data-02:/bricks/fce7e9b3-4ff4-4e01-ae02-8d27d175485c/brick
Brick285: dc4-data-03:/bricks/143427df-c5b1-4b4b-b38a-9f337418c187/brick
Brick286: dc4-data-04:/bricks/ff78d3c6-0ce2-4c7a-98ed-8b7cc1d588be/brick
Brick287: dc4-data-05:/bricks/43b04ce5-0f35-4eab-8ca6-aaaf3c0a91cc/brick
Brick288: dc4-data-06:/bricks/9a3d92a4-49b5-449a-97ea-e49f66f9b672/brick

Regards, Tamar

[smremde]

@tshacked

This seems to have fixed these files. We had around 57k files. Some were unrecoverable, We think this due to a previous attempt to fix some of them.

My concern is another rebalance, will cause more issues.

I'm happy to increase the log level and start another rebalance - my only concern is the logs will grow too quickly but we can monitor this.

Or do you have any more insight in to the cause or is there any more information we can provide?

[tshacked]

Or do you have any more insight in to the cause or is there any more information we can provide?

Many files were not rebalanced to the correct subvol or at all due to "lower disk space". Please check disk-usage per subvol using this cmd, expanding the volume may be needed before running rebalance: $ gluster volume status vol-name detail

[smremde]

All bricks have atleast 150gb with the majority have 300gb-2tb free

[smremde]

Hi @tshacked ,

We are seeing this issue once again since we just expanded another volume. (This is another deployment, with 6 servers 4+2, gluster 9.3 everything else equal)

We see files duplicate with the same "operation not supported" log message.

Any update on the issue or a work around? If this is a race condition, could we start the rebalance and kill the rebalance process on 5/6 servers?

Thanks

[tshacked]

Hi @smremde

Any update on the issue or a work around? If this is a race condition, could we start the rebalance and kill the rebalance process on 5/6 servers?

currently the workaround is doing a manual fix to the undefined linkto file as you've already done. Why do you want to kill the rebalance processes? you can send rebalance stop cmd but it's better to let the process complete the migration.

About the issue, I've not been able to reproduce it so it will help if u could load here: 1 - rebalance/brick logs from all nodes 2 - list the xattr of a problematic file I think that the RC related somehow to the EC layer, i"ll ask from the EC maintainer to check it too

[smremde]

@tshacked

1 - rebalance/brick logs from all nodes 2 - list the xattr of a problematic file

The problem is, this will make thousands of files randomly unavailable during the rebalance which will take several days to complete - we would have to implement a process to watch the logs and fix them on the fly.

Why do you want to kill the rebalance processes? you can send rebalance stop cmd but it's better to let the process complete the migration.

If the race condition is between two servers/instances of the rebalance process, I was suggesting we kill/term the rebalance process on all except 1 of the servers so there can be no race condition

[tshacked]

@tshacked

1 - rebalance/brick logs from all nodes 2 - list the xattr of a problematic file

The problem is, this will make thousands of files randomly unavailable during the rebalance which will take several days to complete - we would have to implement a process to watch the logs and fix them on the fly.

Why do you want to kill the rebalance processes? you can send rebalance stop cmd but it's better to let the process complete the migration.

If the race condition is between two servers/instances of the rebalance process, I was suggesting we kill/term the rebalance process on all except 1 of the servers so there can be no race condition

I don"t know the exact RC, but with above approach not all files will be migrated, as each rebalance process handle some of the files from his replication set. For example: rebalance from 2x2 -> 3x2

Replica-1:
- R1: if needed will migrate  f1...f10  //the decision if is according to hash calc
- R2: if needed will migrate f11..f20
Replica-2:
- R3: if needed will migrate f1...f10
- R4: if needed will migrate f11..f20
Replica-3: (New)
- R5: 
- R6: 

If you do think trying this approach, I suggest u to run fix-layout before the try

[smremde]

I think you are correct. My confusion was from the previous rebalance of a different volume in this ticket, the same file was migrated by two different servers. I now believe this was because of existing duplicates on the volume.

We have check the logs for this rebalance and the files are only attempted to be migrated on a single server. We stopped the rebalance when we noticed this happened, still in one hour with only a fraction of the rebalance completed there were over 150 duplicated files.

Additionally, we see the same pattern for each one in the logs (identical to the OP in this issue so copied here for reference):

[2022-03-14 16:24:47.957556 +0000] I [dht-rebalance.c:1536:dht_migrate_file] 0-dc4-dht: /data/queues/label-full-defect-qa-20191107.json: attempting to move from dc4-readdir-ahead-28 to dc4-readdir-ahead-16
[2022-03-14 16:24:48.053314 +0000] W [MSGID: 0] [dht-rebalance.c:922:__dht_check_free_space] 0-dc4-dht: Write will cross min-free-disk for file - /data/queues/label-full-defect-qa-20191107.json on subvol - dc4-readdir-ahead-16. Looking for new subvol
[2022-03-14 16:24:48.053455 +0000] I [MSGID: 0] [dht-rebalance.c:979:__dht_check_free_space] 0-dc4-dht: new target found - dc4-readdir-ahead-47 for file - /data/queues/label-full-defect-qa-20191107.json
[2022-03-14 16:24:48.192489 +0000] I [MSGID: 0] [dht-rebalance.c:1741:dht_migrate_file] 0-dc4-dht: destination for file - /data/queues/label-full-defect-qa-20191107.json is changed to - dc4-readdir-ahead-47
[2022-03-14 16:24:48.385145 +0000] E [MSGID: 109023] [dht-rebalance.c:2014:dht_migrate_file] 0-dc4-dht: failed to set xattr on /data/queues/label-full-defect-qa-20191107.json in dc4-readdir-ahead-16 [Operation not supported]
[2022-03-14 16:24:48.388013 +0000] E [MSGID: 109023] [dht-rebalance.c:2859:gf_defrag_migrate_single_file] 0-dc4-dht: migrate-data failed for /data/queues/label-full-defect-qa-20191107.json [Operation not supported]

and brick logs:

/var/log/glusterfs/bricks/bricks-1c9a05b3-a21b-4afb-abe2-620e6093863a-brick.log.4.gz:[2022-03-14 16:24:48.385321 +0000] E [MSGID: 113001] [posix-helpers.c:1273:posix_handle_pair] 0-dc4-posix: /bricks/1c9a05b3-a21b-4afb-abe2-620e6093863a/brick/.glusterfs/68/bc/68bc19ae-4a72-4e3d-9a60-13d21b2ff0c6: key:glusterfs.inodelk-dom-countflags: 0 length:16 [Operation not supported]
/var/log/glusterfs/bricks/bricks-30efd892-2652-4dfc-9393-4bc82191c1cf-brick.log.4.gz:[2022-03-14 16:24:48.384871 +0000] E [MSGID: 113001] [posix-helpers.c:1273:posix_handle_pair] 0-dc4-posix: /bricks/30efd892-2652-4dfc-9393-4bc82191c1cf/brick/.glusterfs/68/bc/68bc19ae-4a72-4e3d-9a60-13d21b2ff0c6: key:glusterfs.inodelk-dom-countflags: 0 length:16 [Operation not supported]
/var/log/glusterfs/bricks/bricks-7c73f6a5-17f2-4c68-a8a1-b16d5b00eb63-brick.log.4.gz:[2022-03-14 16:24:48.385173 +0000] E [MSGID: 113001] [posix-helpers.c:1273:posix_handle_pair] 0-dc4-posix: /bricks/7c73f6a5-17f2-4c68-a8a1-b16d5b00eb63/brick/.glusterfs/68/bc/68bc19ae-4a72-4e3d-9a60-13d21b2ff0c6: key:glusterfs.inodelk-dom-countflags: 0 length:16 [Operation not supported]
/var/log/glusterfs/bricks/bricks-4de9a7c4-e907-4a3d-b826-01d3aee7e45d-brick.log.4.gz:[2022-03-14 16:24:48.384902 +0000] E [MSGID: 113001] [posix-helpers.c:1273:posix_handle_pair] 0-dc4-posix: /bricks/4de9a7c4-e907-4a3d-b826-01d3aee7e45d/brick/.glusterfs/68/bc/68bc19ae-4a72-4e3d-9a60-13d21b2ff0c6: key:glusterfs.inodelk-dom-countflags: 0 length:16 [Operation not supported]
/var/log/glusterfs/bricks/bricks-01b4239c-072c-446e-bece-aae004cf023b-brick.log.4.gz:[2022-03-14 16:24:48.384330 +0000] E [MSGID: 113001] [posix-helpers.c:1273:posix_handle_pair] 0-dc4-posix: /bricks/01b4239c-072c-446e-bece-aae004cf023b/brick/.glusterfs/68/bc/68bc19ae-4a72-4e3d-9a60-13d21b2ff0c6: key:glusterfs.inodelk-dom-countflags: 0 length:16 [Operation not supported]
/var/log/glusterfs/bricks/bricks-34f0327b-baf0-46d8-b090-377302191cba-brick.log.4.gz:[2022-03-14 16:24:48.385132 +0000] E [MSGID: 113001] [posix-helpers.c:1273:posix_handle_pair] 0-dc4-posix: /bricks/34f0327b-baf0-46d8-b090-377302191cba/brick/.glusterfs/68/bc/68bc19ae-4a72-4e3d-9a60-13d21b2ff0c6: key:glusterfs.inodelk-dom-countflags: 0 length:16 [Operation not supported]

All files that migrated successfully did not report "Write will cross min-free-disk for file", some did not migrate due to having lower free space.

The min-free-disk is set to 1% which leaves over 300gb free on the bricks. We had considered reducing this to 0.5% and starting a rebalance. If we don't see "Operation not supported" within the hour, this may help track down the issue?

[smremde]

min-free-disk does not seem to be functioning correctly, or I have misintpreted it.

We have it set to 0.25% to test, however we are still seeing the Write will cross min-free-disk for file messages for the following subvol:

------------------------------------------------------------------------------
Brick                : Brick dc3-data-01:/bricks/0fb6816f-3ceb-461e-a275-b0a78fd49ad5/brick
TCP Port             : 49221
RDMA Port            : 0
Online               : Y
Pid                  : 3552794
File System          : xfs
Device               : /dev/sdas1
Mount Options        : rw,noatime,attr2,inode64,logbufs=8,logbsize=256k,noquota
Inode Size           : 512
Disk Space Free      : 297.5GB
Total Disk Space     : 14.5TB
Inode Count          : 626341760
Free Inodes          : 624119744
------------------------------------------------------------------------------
Brick                : Brick dc3-data-02:/bricks/c5d5e33f-4b4b-45c3-bf8b-a0218369ac2a/brick
TCP Port             : 49185
RDMA Port            : 0
Online               : Y
Pid                  : 3173
File System          : xfs
Device               : /dev/sdaj1
Mount Options        : rw,noatime,attr2,inode64,logbufs=8,logbsize=256k,noquota
Inode Size           : 512
Disk Space Free      : 297.5GB
Total Disk Space     : 14.5TB
Inode Count          : 626342520
Free Inodes          : 624120504
------------------------------------------------------------------------------
Brick                : Brick dc3-data-03:/bricks/0a8f6cd4-cc92-4e1f-bb7a-deded8a9cc9e/brick
TCP Port             : 49185
RDMA Port            : 0
Online               : Y
Pid                  : 3147
File System          : xfs
Device               : /dev/sdaj1
Mount Options        : rw,noatime,attr2,inode64,logbufs=8,logbsize=256k,noquota
Inode Size           : 512
Disk Space Free      : 297.5GB
Total Disk Space     : 14.5TB
Inode Count          : 626341840
Free Inodes          : 624119824
------------------------------------------------------------------------------
Brick                : Brick dc3-data-04:/bricks/bd80672d-e354-450a-893e-fcffa3138871/brick
TCP Port             : 49185
RDMA Port            : 0
Online               : Y
Pid                  : 4567
File System          : xfs
Device               : /dev/sdaj1
Mount Options        : rw,noatime,attr2,inode64,logbufs=8,logbsize=256k,noquota
Inode Size           : 512
Disk Space Free      : 297.5GB
Total Disk Space     : 14.5TB
Inode Count          : 626339712
Free Inodes          : 624117696
------------------------------------------------------------------------------
Brick                : Brick dc3-data-05:/bricks/28faa161-e339-4334-932e-f446fffef396/brick
TCP Port             : 49185
RDMA Port            : 0
Online               : Y
Pid                  : 4550
File System          : xfs
Device               : /dev/sdaj1
Mount Options        : rw,noatime,attr2,inode64,logbufs=8,logbsize=256k,noquota
Inode Size           : 512
Disk Space Free      : 297.5GB
Total Disk Space     : 14.5TB
Inode Count          : 626340448
Free Inodes          : 624118432
------------------------------------------------------------------------------
Brick                : Brick dc3-data-06:/bricks/4be45ac6-e47d-42b1-870e-4223a956a44f/brick
TCP Port             : 49185
RDMA Port            : 0
Online               : Y
Pid                  : 810720
File System          : xfs
Device               : /dev/sdak1
Mount Options        : rw,noatime,attr2,inode64,logbufs=8,logbsize=256k,noquota
Inode Size           : 512
Disk Space Free      : 297.5GB
Total Disk Space     : 14.5TB
Inode Count          : 626341200
Free Inodes          : 624119184
------------------------------------------------------------------------------

0.25% of 14.5TB is <40gb - well less than the free space we have so I can't understand why it is having issues.

Needless to say, we have temporarily removed some large files that resided on that sub volume and the issues has not shown up since.

The issue definately seems to be related a disk crossing a threshold.

In this circumstance, why isn't rebalance of the file just skipped anyway?

[stale[bot]]

Thank you for your contributions. Noticed that this issue is not having any activity in last ~6 months! We are marking this issue as stale because it has not had recent activity. It will be closed in 2 weeks if no one responds with a comment here.

[stale[bot]]

Closing this issue as there was no update since my last update on issue. If this is an issue which is still valid, feel free to open it.