Mount failing on machines with only IPv4 network configured

[johanneskastl]

Description of problem:

Servers and client are running SLES15 SP4, IPv6 is enabled but the machines only have the link-local addresses configured. Server names are resolvable via IPv4.

Management volume and the actual volume have the transport.address-family=inet settings. But the client must connect to the server once to get the information, and this is where it fails.

Mounting the volume fails, when using the hostname, log output see below.

Mounting the volume succeeds after we add xlator-option=transport.address-family=inet​ to the mount options in /etc/fstab or on the command line.

It looks like it tries to resolve the host via IPv6, which fails (as the host has no working IPv6 connection). It does not fall back to using IPv4 in case of errors, which I think is a bug.

If you need further details, please ask. I just did not want to clutter up this issue with lots of non-relevant logs...

The exact command to reproduce the issue:

mount -t glusterfs -o defaults,_netdev,ro myhostname:/myvolume  /opt/myvolume 

The log output of the command that failed:

This is from the /var/log/gluster/mnt-myvolume.log: (hostnames redacted and replaced by serverhostname, volume name redacted and replaced by myvolume)

[2022-07-14 12:40:01.651962 +0000] I [MSGID: 100030] [glusterfsd.c:2683:main] 0-/usr/sbin/glusterfs: Started running version [{arg=/usr/sbin/glusterfs}, {version=9.3}, {cmdlinestr=/usr/sbin/glusterfs --read-only
 --process-name fuse --volfile-server=serverhostname --volfile-id=/myvolume /opt/myvolume}]
[2022-07-14 12:40:01.653854 +0000] I [glusterfsd.c:2418:daemonize] 0-glusterfs: Pid of current running process is 16010
[2022-07-14 12:40:01.657407 +0000] I [socket.c:4284:ssl_setup_connection_params] 0-glusterfs: SSL support for MGMT is ENABLED IO path is ENABLED certificate depth is 2 for peer
[2022-07-14 12:40:01.658262 +0000] E [MSGID: 101075] [common-utils.c:520:gf_resolve_ip6] 0-resolver: error in getaddrinfo [{family=10}, {ret=Name or service not known}]
[2022-07-14 12:40:01.658289 +0000] E [name.c:265:af_inet_client_get_remote_sockaddr] 0-glusterfs: DNS resolution failed on host serverhostname
[2022-07-14 12:40:01.658463 +0000] I [glusterfsd-mgmt.c:2643:mgmt_rpc_notify] 0-glusterfsd-mgmt: disconnected from remote-host: serverhostname
[2022-07-14 12:40:01.658488 +0000] I [glusterfsd-mgmt.c:2663:mgmt_rpc_notify] 0-glusterfsd-mgmt: Exhausted all volfile servers
[2022-07-14 12:40:01.658567 +0000] I [MSGID: 101190] [event-epoll.c:670:event_dispatch_epoll_worker] 0-epoll: Started thread with index [{index=1}]
[2022-07-14 12:40:01.658644 +0000] I [MSGID: 101190] [event-epoll.c:670:event_dispatch_epoll_worker] 0-epoll: Started thread with index [{index=0}]
[2022-07-14 12:40:01.658705 +0000] W [glusterfsd.c:1429:cleanup_and_exit] (-->/usr/lib64/libgfrpc.so.0(+0xfbfb) [0x7f9ce67e0bfb] -->/usr/sbin/glusterfs(+0x140d7) [0x55904c2140d7] -->/usr/sbin/glusterfs(cleanup_a
nd_exit+0x54) [0x55904c208524] ) 0-: received signum (1), shutting down
[2022-07-14 12:40:01.658744 +0000] I [fuse-bridge.c:7063:fini] 0-fuse: Unmounting '/opt/myvolume'.
[2022-07-14 12:40:01.659215 +0000] I [fuse-bridge.c:7068:fini] 0-fuse: Closing fuse connection to '/opt/myvolume'.

- The operating system / glusterfs version:

SLES15 SP4, glusterfs 9.3

[chen1195585098]

set option transport.address-family in glusterd.vol according to your network type, or update your glusterfs to latest version may be helpful to avoid this.

Since your command works after add address-family, it may caused due to mismatch of address-family between the record in glusterd.vol and that you actually use.

[johanneskastl]

It was already set on the server side, but the client needs to reach the server initially for fetching all the information inside the vol-files. And this is what fails, unless I set the xlator-option parameter...

[chen1195585098]

Hi, is this issue solved? I cannot reproduce it, but it seems to be similar to that i have met ever. If this issue keeps on, can you provide a more detailed procedure to reproduce it?

[manfredw]

Have the same issue after OS Upgrade from SUSE 15.3 to 15.4 where glusterfs version changes from 3.12 to 9.3.

Mounting the glusterfs volume works with the above mentioned mount option, I've enabled IPv6 on OS level (with link-local addresses only). transport.address-family=inet is also set.

self-heal-daemon seems to be unable to start or communicate. glusterd logfile records this message every 3 seconds: W [socket.c:3434:socket_connect] 0-glustershd: Error disabling sockopt IPV6_V6ONLY: "Operation not supported"

When trying to start a heal operation it fails with following message: Self-heal daemon is not running. Check self-heal daemon log file.

glustershd.log shows this block repeatly (sensitive infos removed):

I [MSGID: 100030] [glusterfsd.c:2683:main] 0-/usr/sbin/glusterfs: Started running version [{arg=/usr/sbin/glusterfs}, {version=9.3}, {cmdlinestr=/usr/sbin/glusterfs -s localhost --volfile-id shd/XXX -p /var/run/gluster/shd/XXX/XXX-shd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/XXX.socket --xlator-option *replicate*.node-uuid=XXX --process-name glustershd --client-pid=-6}]
I [glusterfsd.c:2418:daemonize] 0-glusterfs: Pid of current running process is 22539
I [socket.c:929:__socket_server_bind] 0-socket.glusterfsd: closing (AF_UNIX) reuse check socket 9
I [MSGID: 101190] [event-epoll.c:670:event_dispatch_epoll_worker] 0-epoll: Started thread with index [{index=0}]
I [glusterfsd-mgmt.c:2643:mgmt_rpc_notify] 0-glusterfsd-mgmt: disconnected from remote-host: localhost
I [glusterfsd-mgmt.c:2663:mgmt_rpc_notify] 0-glusterfsd-mgmt: Exhausted all volfile servers
W [glusterfsd.c:1429:cleanup_and_exit] (-->/usr/lib64/libgfrpc.so.0(+0xfbfb) [0x7f2628fcabfb] --> /usr/sbin/glusterfs(+0x140d7) [0x556eae2140d7] -->/usr/sbin/glusterfs(cleanup_and_exit+0x54) [0x556eae208524] ) 0-: received signum(1), shutting down

It's not really an option to use an newer glusterfs version because this version is part of the distribution, but it should be possible to let the SUSE team fix this issue if it is a bug or if there are other workarounds like config changes or compile options.

[manfredw]

Solved the issue by using a gusterfs repository from SUSE Build Service which offers a newer version of glusterfsd that worked.

[stale[bot]]

Thank you for your contributions. Noticed that this issue is not having any activity in last ~6 months! We are marking this issue as stale because it has not had recent activity. It will be closed in 2 weeks if no one responds with a comment here.

[black-dragon74]

Closing it as the issue is solved.