gluster disconnections on the client side, resulting in locks being released by the client applications after restart of gluster nodes in a round robin manner.

[murali437]

Description of problem: we have a 3 node gluster cluster in replicated-arbiter configuration. we restart gluster servers one at a time. We wait till the brick is back online and the healing is complete before repeating the process for subsequent gluster cluster nodes. For the first two nodes, reboot doesn't impact the clients. when the third node is rebooted we see that the client loses the locks it held before and applications begins to fail. We have tried different order of stopping the gluster nodes (replicated nodes followed by arbiter or arbiter followed by replicated nodes), the end result remains the same (file system is not available for the clients). In the steps provided before, we perform glusterd stop and killall glusterfs glusterfsd process to simulate reboot of the gluster node.

We've attached the logs from each of the gluster server nodes (host05.tar.gz, host06.tar.gz, host07.tar.gz ) and the gluster client (CliHost.tar.gz) in addition to a sample application (ClientApp.tar.gz) which performs the locking and monitoring the lock acquired, get all parameters from the gluster server etc.. The ClientApp.tar.gz contains the sample application code and a README to explain what it does and how to use it.

The exact command to reproduce the issue: Step 1: On Client side, execute:    In First window:       ./lockfile
   In Second window:       ./checkLck See README for more information.

Step 2: On Gluster Server side: On node1: systemctl stop glusterd killall glusterfs glusterfsd systemctl start glusterd gluster volume status gluster volume heal info

On node2: systemctl stop glusterd killall glusterfs glusterfsd systemctl start glusterd gluster volume status gluster volume heal info

On node3: systemctl stop glusterd killall glusterfs glusterfsd systemctl start glusterd gluster volume status gluster volume heal info

The full output of the command that failed: none of the command failed, but we see that the client side start receiving errors. [2024-05-20 09:41:23.387247 +0000] I [MSGID: 114018] [client.c:2244:client_rpc_notify] 0-dr84dir_dirdata_gfs-client-2: disconnected from client, process will keep trying to connect glusterd until brick's port is available [{conn-name=dr84dir_dirdata_gfs-client-2}] The message "W [MSGID: 114061] [client-common.c:797:client_pre_lk_v2] 0-dr84dir_dirdata_gfs-client-2: remote_fd is -1. EBADFD [{gfid=2d80904a-ad62-42d8-8856-6f0653a61f54}, {errno=77}, {error=File descriptor in bad state}]" repeated 18 times between [2024-05-20 09:40:51.321731 +0000] and [2024-05-20 09:41:27.354988 +0000]

Sample application releasing the lock when we shutdown the 3rd gluster node as show below ./checkLck /opt/data/cloud_dir/host01.lock My PID is : 3530805 2024-05-20-15:03:56 IST: File /opt/data/cloud_dir/host01.lock is locked by 3530796 2024-05-20-15:10:51 IST: File /opt/data/cloud_dir/host01.lock is unlocked. Lock Type: 2 Whence: 0 Start: 0 len: 0 2024-05-20-15:10:53 IST: File /opt/data/cloud_dir/host01.lock is unlocked. Lock Type: 2 Whence: 0 Start: 0 len: 0 2024-05-20-15:10:55 IST: File /opt/data/cloud_dir/host01.lock is unlocked. Lock Type: 2 Whence: 0 Start: 0 len: 0 ....

Expected results: The client continues to function normally without disruption and the mounted file system is available for use by the applications and the locks held by the processes remains held until the process decides to release the locks.

Mandatory info: - The output of the gluster volume info command: Volume Name: dr84dir_dirdata_gfs Type: Replicate Volume ID: 0f91af99-291a-4a14-9010-a565b23a75d4 Status: Started Snapshot Count: 0 Number of Bricks: 1 x (2 + 1) = 3 Transport-type: tcp Bricks: Brick1: host05.domain:/Dirdata/dr84dir_dirdata_brk Brick2: host06.domain:/Dirdata/dr84dir_dirdata_brk Brick3: host07.domain:/Dirdata/dr84dir_dirdata_brk (arbiter) Options Reconfigured: network.frame-timeout: 300 server.tcp-user-timeout: 42 cluster.eager-lock: off cluster.server-quorum-type: server cluster.self-heal-daemon: enable network.tcp-window-size: 1048576 server.event-threads: 24 client.event-threads: 24 features.cache-invalidation: on performance.parallel-readdir: on performance.global-cache-invalidation: true performance.cache-invalidation: true performance.readdir-ahead: off performance.stat-prefetch: off performance.open-behind: off performance.quick-read: off performance.read-ahead: off performance.lazy-open: no performance.write-behind: off performance.strict-o-direct: on performance.flush-behind: off performance.io-cache: off cluster.consistent-metadata: yes cluster.quorum-type: auto cluster.data-self-heal-algorithm: full transport.address-family: inet performance.client-io-threads: off cluster.server-quorum-ratio: 51

- The output of the gluster volume status command: Status of volume: dr84dir_dirdata_gfs Gluster process TCP Port RDMA Port Online Pid

Brick host05.domain:/Dirdat a/dr84dir_dirdata_brk 49180 0 Y 251268 Brick host06.domain:/Dirdat a/dr84dir_dirdata_brk 49243 0 Y 482164 Brick host07.domain:/Dirdat a/dr84dir_dirdata_brk 49223 0 Y 206760 Self-heal Daemon on localhost N/A N/A Y 251306 Self-heal Daemon on host06 N/A N/A Y 482235 Self-heal Daemon on host07 N/A N/A Y 206790

Task Status of Volume dr84dir_dirdata_gfs

There are no active volume tasks

- The output of the gluster volume heal command: Launching heal operation to perform index self heal on volume dr84dir_dirdata_gfs has been successful Use heal info commands to check status.

gluster volume heal dr84dir_dirdata_gfs info

Brick host05.domain:/Dirdata/dr84dir_dirdata_brk Status: Connected Number of entries: 0

Brick host06.domain:/Dirdata/dr84dir_dirdata_brk Status: Connected Number of entries: 0

Brick host07.domain:/Dirdata/dr84dir_dirdata_brk Status: Connected Number of entries: 0

**- Provide logs present on following locations of client and server nodes - Logs from client and server nodes are attached. GlusterData.tar.gz

Sever Logs: GlusterData/host05.tar.gz GlusterData/host06.tar.gz GlusterData/host07.tar.gz Client Logs: GlusterData/CliHost.tar.gz

Client Application: GlusterData/Clientapp.tar.gz

**- Is there any crash ? Provide the backtrace and coredump No

Additional info:

- The operating system / glusterfs version:

Operating System: RHEL 8.8

glusterd --version glusterfs 11.1 Repository revision: git://git.gluster.org/glusterfs.git Copyright (c) 2006-2016 Red Hat, Inc. <https://www.gluster.org/> ...

glusterfs --version glusterfs 11.1 Repository revision: git://git.gluster.org/glusterfs.git Copyright (c) 2006-2016 Red Hat, Inc. <https://www.gluster.org/> ...

Note: Please hide any confidential data which you don't want to share in public like IP address, file name, hostname or any other configuration