Stephen Remde

This particular files has 2 copies + 2 linkto ``` ls /mnt/volume/data/queues ... -rw-rw-r-- 1 user users 3087081 Nov 7 2019 label-full-defect-qa-20191107.json -rw-rw-r-- 1 user users 0 Nov 7 2019...

Hi @tshacked - thanks for responding. The file appears in 3 logs (unfiltered, 5 lines before and after): ``` dc4-data-06 [2022-03-14 16:44:21.397060 +0000] W [MSGID: 109023] [dht-rebalance.c:891:__dht_check_free_space] 0-dc4-dht: data movement...

I noticed a mistake in the initial comment from the brick logs - this is the correct logs: ``` /var/log/glusterfs/bricks/bricks-1c9a05b3-a21b-4afb-abe2-620e6093863a-brick.log.4.gz:[2022-03-14 16:24:48.385321 +0000] E [MSGID: 113001] [posix-helpers.c:1273:posix_handle_pair] 0-dc4-posix: /bricks/1c9a05b3-a21b-4afb-abe2-620e6093863a/brick/.glusterfs/68/bc/68bc19ae-4a72-4e3d-9a60-13d21b2ff0c6: key:glusterfs.inodelk-dom-countflags: 0...

@tshacked This seems to have fixed these files. We had around 57k files. Some were unrecoverable, We think this due to a previous attempt to fix some of them. My...

All bricks have atleast 150gb with the majority have 300gb-2tb free

Hi @tshacked , We are seeing this issue once again since we just expanded another volume. (This is another deployment, with 6 servers 4+2, gluster 9.3 everything else equal) We...

@tshacked >1 - rebalance/brick logs from all nodes > 2 - list the xattr of a problematic file The problem is, this will make thousands of files randomly unavailable during...

I think you are correct. My confusion was from the previous rebalance of a different volume in this ticket, the same file was migrated by two different servers. I now...

`min-free-disk` does not seem to be functioning correctly, or I have misintpreted it. We have it set to 0.25% to test, however we are still seeing the `Write will cross...

Hi @mohit84 thanks for the quick response. You are correct, it appears we do have duplicate `brick-fsid` but they are most certainly different file systems: ``` [13:11] root@dc3-data-01:/# grep brick-fsid=2081...