convoy icon indicating copy to clipboard operation
convoy copied to clipboard

Published 20 hours ago • verified publisher icon rancher

Couldn't get GlusterFS Convoy with v0.50.2 Rancher to work

Open lukmadev opened this issue 9 years ago • 4 comments

Hi,

I coudn't get GlusterFS Convoy with v0.50.2 Rancher to work. I tried this multiple times. I've even still upgraded to the newest rancher version.

The GlusterFS Service itself is started correctly as far as I can see. But when I'm trying to create a Gluster FS Convoy Service from Rancher Catalog the whole thing hangs in initializing process.

Log output from convoy-gluster_convoy-gluster_1 (volume-agent-glusterfs):

21.12.2015 15:27:55Waiting for metadata.time="2015-12-21T14:27:55Z" level=info msg="Execing [/usr/bin/nsenter --mount=/proc/778/ns/mnt -F -- /var/lib/docker/aufs/mnt/e4e77074b8372129f30a0b9ca864e54feddf414e1ae495404897db546359a248/var/lib/rancher/convoy-agent/share-mnt --stage2 /var/lib/rancher/convoy/convoy-gluster-2cf6ae3e-205f-4892-a791-9f526adfda91 -- /launch volume-agent-glusterfs-internal]"
21.12.2015 15:27:55Registering convoy socket at /var/run/conoy-convoy-gluster.sock
21.12.2015 15:27:55time="2015-12-21T14:27:55Z" level=info msg="Listening for health checks on 0.0.0.0:10241/healthcheck"
21.12.2015 15:27:55time="2015-12-21T14:27:55Z" level=info msg="Got: root /var/lib/rancher/convoy/convoy-gluster-2cf6ae3e-205f-4892-a791-9f526adfda91"
21.12.2015 15:27:55time="2015-12-21T14:27:55Z" level=info msg="Got: drivers [glusterfs]"
21.12.2015 15:27:55time="2015-12-21T14:27:55Z" level=info msg="Got: driver-opts [glusterfs.defaultvolumepool=herocloud glusterfs.servers=glusterfs]"
21.12.2015 15:27:55time="2015-12-21T14:27:55Z" level=info msg="Launching convoy with args: [--socket=/host/var/run/conoy-convoy-gluster.sock daemon --root=/var/lib/rancher/convoy/convoy-gluster-2cf6ae3e-205f-4892-a791-9f526adfda91 --drivers=glusterfs --driver-opts=glusterfs.defaultvolumepool=herocloud --driver-opts=glusterfs.servers=glusterfs]"
21.12.2015 15:27:55time="2015-12-21T14:27:55Z" level=debug msg="Creating config at /var/lib/rancher/convoy/convoy-gluster-2cf6ae3e-205f-4892-a791-9f526adfda91" pkg=daemon
21.12.2015 15:27:55time="2015-12-21T14:27:55Z" level=debug msg= driver=glusterfs driver_opts=map[glusterfs.defaultvolumepool:herocloud glusterfs.servers:glusterfs] event=init pkg=daemon reason=prepare root="/var/lib/rancher/convoy/convoy-gluster-2cf6ae3e-205f-4892-a791-9f526adfda91"
21.12.2015 15:27:55time="2015-12-21T14:27:55Z" level=debug msg="Volume herocloud is being mounted it to /var/lib/rancher/convoy/convoy-gluster-2cf6ae3e-205f-4892-a791-9f526adfda91/glusterfs/mounts/herocloud, with option [-t glusterfs]" pkg=util
21.12.2015 15:27:56time="2015-12-21T14:27:56Z" level=error msg="Get http:///host/var/run/conoy-convoy-gluster.sock/v1/volumes/list: dial unix /host/var/run/conoy-convoy-gluster.sock: no such file or directory"
21.12.2015 15:27:57time="2015-12-21T14:27:57Z" level=error msg="Get http:///host/var/run/conoy-convoy-gluster.sock/v1/volumes/list: dial unix /host/var/run/conoy-convoy-gluster.sock: no such file or directory"
21.12.2015 15:27:58time="2015-12-21T14:27:58Z" level=error msg="Get http:///host/var/run/conoy-convoy-gluster.sock/v1/volumes/list: dial unix /host/var/run/conoy-convoy-gluster.sock: no such file or directory"
... repeats very often ...
21.12.2015 15:30:02time="2015-12-21T14:30:02Z" level=error msg="Get http:///host/var/run/conoy-convoy-gluster.sock/v1/volumes/list: dial unix /host/var/run/conoy-convoy-gluster.sock: no such file or directory"
21.12.2015 15:30:02time="2015-12-21T14:30:02Z" level=debug msg="Cleaning up environment..." pkg=daemon
21.12.2015 15:30:02time="2015-12-21T14:30:02Z" level=error msg="Failed to execute: mount [-t glusterfs glusterfs:/herocloud /var/lib/rancher/convoy/convoy-gluster-2cf6ae3e-205f-4892-a791-9f526adfda91/glusterfs/mounts/herocloud], output Mount failed. Please check the log file for more details.\n, error exit status 1"
21.12.2015 15:30:02{
21.12.2015 15:30:02 "Error": "Failed to execute: mount [-t glusterfs glusterfs:/herocloud /var/lib/rancher/convoy/convoy-gluster-2cf6ae3e-205f-4892-a791-9f526adfda91/glusterfs/mounts/herocloud], output Mount failed. Please check the log file for more details.\n, error exit status 1"
21.12.2015 15:30:02}
21.12.2015 15:30:02time="2015-12-21T14:30:02Z" level=info msg="convoy exited with error: exit status 1"
21.12.2015 15:30:02time="2015-12-21T14:30:02Z" level=info msg=Exiting.

Log Output from convoy-gluster_convoy-gluster-storagepool_1 Container:

21.12.2015 15:27:55time="2015-12-21T14:27:55Z" level=info msg="Listening for health checks on 0.0.0.0:10241/healthcheck"
21.12.2015 15:28:00time="2015-12-21T14:28:00Z" level=debug msg="storagepool event [2c5c6de9-67dd-4a88-a436-2a78b5c4b1f2]"
21.12.2015 15:41:10Waiting for metadata.

Any ideas what's going wrong?

lukmadev avatar Dec 21 '15 14:12 lukmadev

Hi @lukmadev

Could you execute shell into convoy-gluster_convoy-gluster_1 and check grep -r gluster /var/log/*?

yasker avatar Dec 21 '15 19:12 yasker

Hi @yasker ,

thanks for your help!

Here is the log

/var/log/apt/history.log:Commandline: apt-get install -y curl glusterfs-client
/var/log/apt/history.log:Install: glusterfs-common:amd64 (3.7.6-ubuntu1~wily1, automatic), libpython2.7-stdlib:amd64 (2.7.10-4ubuntu1, automatic), libaio1:amd64 (0.3.110-1, automatic), python-minimal:amd64 (2.7.9-1, automatic), fuse:amd64 (2.9.4-1ubuntu1, automatic), libacl1-dev:amd64 (2.2.52-2, automatic), liburcu2:amd64 (0.8.7-2, automatic), librdmacm1:amd64 (1.0.21-1, automatic), libattr1-dev:amd64 (2.4.47-2, automatic), libc-dev-bin:amd64 (2.21-0ubuntu4, automatic), python2.7:amd64 (2.7.10-4ubuntu1, automatic), curl:amd64 (7.43.0-1ubuntu2), libibverbs1:amd64 (1.1.8-1.1ubuntu1, automatic), glusterfs-client:amd64 (3.7.6-ubuntu1~wily1), libpython2.7:amd64 (2.7.10-4ubuntu1, automatic), python:amd64 (2.7.9-1, automatic), manpages-dev:amd64 (3.74-1ubuntu1, automatic), libdevmapper-event1.02.1:amd64 (1.02.99-1ubuntu1, automatic), libpython-stdlib:amd64 (2.7.9-1, automatic), manpages:amd64 (3.74-1ubuntu1, automatic), python2.7-minimal:amd64 (2.7.10-4ubuntu1, automatic), liblvm2app2.2:amd64 (2.02.122-1ubuntu1, automatic), linux-libc-dev:amd64 (4.2.0-19.23, automatic), libfuse2:amd64 (2.9.4-1ubuntu1, automatic), attr:amd64 (2.4.47-2, automatic), libpython2.7-minimal:amd64 (2.7.10-4ubuntu1, automatic), libc6-dev:amd64 (2.21-0ubuntu4, automatic)
/var/log/apt/term.log:Selecting previously unselected package glusterfs-common.^M
/var/log/apt/term.log:Preparing to unpack .../glusterfs-common_3.7.6-ubuntu1~wily1_amd64.deb ...^M
/var/log/apt/term.log:Unpacking glusterfs-common (3.7.6-ubuntu1~wily1) ...^M
/var/log/apt/term.log:Selecting previously unselected package glusterfs-client.^M
/var/log/apt/term.log:Preparing to unpack .../glusterfs-client_3.7.6-ubuntu1~wily1_amd64.deb ...^M
/var/log/apt/term.log:Unpacking glusterfs-client (3.7.6-ubuntu1~wily1) ...^M
/var/log/apt/term.log:Setting up glusterfs-common (3.7.6-ubuntu1~wily1) ...^M
/var/log/apt/term.log:Setting up glusterfs-client (3.7.6-ubuntu1~wily1) ...^M
/var/log/dpkg.log:2015-12-02 00:10:31 install glusterfs-common:amd64 <none> 3.7.6-ubuntu1~wily1
/var/log/dpkg.log:2015-12-02 00:10:31 status half-installed glusterfs-common:amd64 3.7.6-ubuntu1~wily1
/var/log/dpkg.log:2015-12-02 00:10:32 status unpacked glusterfs-common:amd64 3.7.6-ubuntu1~wily1
/var/log/dpkg.log:2015-12-02 00:10:32 status unpacked glusterfs-common:amd64 3.7.6-ubuntu1~wily1
/var/log/dpkg.log:2015-12-02 00:10:32 install glusterfs-client:amd64 <none> 3.7.6-ubuntu1~wily1
/var/log/dpkg.log:2015-12-02 00:10:32 status half-installed glusterfs-client:amd64 3.7.6-ubuntu1~wily1
/var/log/dpkg.log:2015-12-02 00:10:32 status unpacked glusterfs-client:amd64 3.7.6-ubuntu1~wily1
/var/log/dpkg.log:2015-12-02 00:10:32 status unpacked glusterfs-client:amd64 3.7.6-ubuntu1~wily1
/var/log/dpkg.log:2015-12-02 00:10:33 configure glusterfs-common:amd64 3.7.6-ubuntu1~wily1 <none>
/var/log/dpkg.log:2015-12-02 00:10:33 status unpacked glusterfs-common:amd64 3.7.6-ubuntu1~wily1
/var/log/dpkg.log:2015-12-02 00:10:33 status unpacked glusterfs-common:amd64 3.7.6-ubuntu1~wily1
/var/log/dpkg.log:2015-12-02 00:10:33 status half-configured glusterfs-common:amd64 3.7.6-ubuntu1~wily1
/var/log/dpkg.log:2015-12-02 00:10:34 status installed glusterfs-common:amd64 3.7.6-ubuntu1~wily1
/var/log/dpkg.log:2015-12-02 00:10:34 configure glusterfs-client:amd64 3.7.6-ubuntu1~wily1 <none>
/var/log/dpkg.log:2015-12-02 00:10:34 status unpacked glusterfs-client:amd64 3.7.6-ubuntu1~wily1
/var/log/dpkg.log:2015-12-02 00:10:34 status unpacked glusterfs-client:amd64 3.7.6-ubuntu1~wily1
/var/log/dpkg.log:2015-12-02 00:10:34 status half-configured glusterfs-client:amd64 3.7.6-ubuntu1~wily1
/var/log/dpkg.log:2015-12-02 00:10:34 status installed glusterfs-client:amd64 3.7.6-ubuntu1~wily1
/var/log/glusterfs/var-lib-rancher-convoy-convoy-gluster-2cf6ae3e-205f-4892-a791-9f526adfda91-glusterfs-mounts-herocloud.log:[2015-12-21 14:27:55.334690] I [MSGID: 100030] [glusterfsd.c:2318:main] 0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version 3.7.6 (args: /usr/sbin/glusterfs --volfile-server=glusterfs --volfile-id=/herocloud /var/lib/rancher/convoy/convoy-gluster-2cf6ae3e-205f-4892-a791-9f526adfda91/glusterfs/mounts/herocloud)
/var/log/glusterfs/var-lib-rancher-convoy-convoy-gluster-2cf6ae3e-205f-4892-a791-9f526adfda91-glusterfs-mounts-herocloud.log:[2015-12-21 14:30:02.548957] E [socket.c:2278:socket_connect_finish] 0-glusterfs: connection to 10.42.120.108:24007 failed (Connection timed out)
/var/log/glusterfs/var-lib-rancher-convoy-convoy-gluster-2cf6ae3e-205f-4892-a791-9f526adfda91-glusterfs-mounts-herocloud.log:[2015-12-21 14:30:02.549003] E [glusterfsd-mgmt.c:1818:mgmt_rpc_notify] 0-glusterfsd-mgmt: failed to connect with remote-host: glusterfs (Transport endpoint is not connected)
/var/log/glusterfs/var-lib-rancher-convoy-convoy-gluster-2cf6ae3e-205f-4892-a791-9f526adfda91-glusterfs-mounts-herocloud.log:[2015-12-21 14:30:02.549010] I [glusterfsd-mgmt.c:1824:mgmt_rpc_notify] 0-glusterfsd-mgmt: Exhausted all volfile servers
/var/log/glusterfs/var-lib-rancher-convoy-convoy-gluster-2cf6ae3e-205f-4892-a791-9f526adfda91-glusterfs-mounts-herocloud.log:[2015-12-21 14:30:02.549152] W [glusterfsd.c:1236:cleanup_and_exit] (-->/usr/lib/x86_64-linux-gnu/libgfrpc.so.0(rpc_clnt_notify+0xd2) [0x7f01b2210402] -->/usr/sbin/glusterfs(+0xf29f) [0x7f01b291529f] -->/usr/sbin/glusterfs(cleanup_and_exit+0x56) [0x7f01b290f3f6] ) 0-: received signum (1), shutting down
/var/log/glusterfs/var-lib-rancher-convoy-convoy-gluster-2cf6ae3e-205f-4892-a791-9f526adfda91-glusterfs-mounts-herocloud.log:[2015-12-21 14:30:02.549174] I [fuse-bridge.c:5683:fini] 0-fuse: Unmounting '/var/lib/rancher/convoy/convoy-gluster-2cf6ae3e-205f-4892-a791-9f526adfda91/glusterfs/mounts/herocloud'.
/var/log/glusterfs/var-lib-rancher-convoy-convoy-gluster-2cf6ae3e-205f-4892-a791-9f526adfda91-glusterfs-mounts-herocloud.log:[2015-12-21 14:30:06.793426] I [MSGID: 100030] [glusterfsd.c:2318:main] 0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version 3.7.6 (args: /usr/sbin/glusterfs --volfile-server=glusterfs --volfile-id=/herocloud /var/lib/rancher/convoy/convoy-gluster-2cf6ae3e-205f-4892-a791-9f526adfda91/glusterfs/mounts/herocloud)
/var/log/glusterfs/var-lib-rancher-convoy-convoy-gluster-2cf6ae3e-205f-4892-a791-9f526adfda91-glusterfs-mounts-herocloud.log:[2015-12-21 14:32:14.004985] E [socket.c:2278:socket_connect_finish] 0-glusterfs: connection to 10.42.120.108:24007 failed (Connection timed out)
/var/log/glusterfs/var-lib-rancher-convoy-convoy-gluster-2cf6ae3e-205f-4892-a791-9f526adfda91-glusterfs-mounts-herocloud.log:[2015-12-21 14:32:14.005022] E [glusterfsd-mgmt.c:1818:mgmt_rpc_notify] 0-glusterfsd-mgmt: failed to connect with remote-host: glusterfs (Transport endpoint is not connected)
/var/log/glusterfs/var-lib-rancher-convoy-convoy-gluster-2cf6ae3e-205f-4892-a791-9f526adfda91-glusterfs-mounts-herocloud.log:[2015-12-21 14:32:14.005028] I [glusterfsd-mgmt.c:1824:mgmt_rpc_notify] 0-glusterfsd-mgmt: Exhausted all volfile servers
/var/log/glusterfs/var-lib-rancher-convoy-convoy-gluster-2cf6ae3e-205f-4892-a791-9f526adfda91-glusterfs-mounts-herocloud.log:[2015-12-21 14:32:14.005154] W [glusterfsd.c:1236:cleanup_and_exit] (-->/usr/lib/x86_64-linux-gnu/libgfrpc.so.0(rpc_clnt_notify+0xd2) [0x7f323762b402] -->/usr/sbin/glusterfs(+0xf29f) [0x7f3237d3029f] -->/usr/sbin/glusterfs(cleanup_and_exit+0x56) [0x7f3237d2a3f6] ) 0-: received signum (1), shutting down
/var/log/glusterfs/var-lib-rancher-convoy-convoy-gluster-2cf6ae3e-205f-4892-a791-9f526adfda91-glusterfs-mounts-herocloud.log:[2015-12-21 14:32:14.005175] I [fuse-bridge.c:5683:fini] 0-fuse: Unmounting '/var/lib/rancher/convoy/convoy-gluster-2cf6ae3e-205f-4892-a791-9f526adfda91/glusterfs/mounts/herocloud'.

... etc ...

I've digged a bit more in the logs and found that in glusterfs_glusterfs-server_glusterfs-peer_3

22.12.2015 07:46:15Containers are starting...
22.12.2015 07:46:15Waiting for Gluster Daemons to come up
22.12.2015 07:46:50gluster peer probe 10.42.120.108
22.12.2015 07:48:50Error : Request timed out
22.12.2015 07:48:51Waiting for all service containers to start...
...

lukmadev avatar Dec 22 '15 07:12 lukmadev

See:

/var/log/glusterfs/var-lib-rancher-convoy-convoy-gluster-2cf6ae3e-205f-4892-a791-9f526adfda91-glusterfs-mounts-herocloud.log:[2015-12-21 14:27:55.334690] I [MSGID: 100030] [glusterfsd.c:2318:main] 0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version 3.7.6 (args: /usr/sbin/glusterfs --volfile-server=glusterfs --volfile-id=/herocloud /var/lib/rancher/convoy/convoy-gluster-2cf6ae3e-205f-4892-a791-9f526adfda91/glusterfs/mounts/herocloud)
/var/log/glusterfs/var-lib-rancher-convoy-convoy-gluster-2cf6ae3e-205f-4892-a791-9f526adfda91-glusterfs-mounts-herocloud.log:[2015-12-21 14:30:02.548957] E [socket.c:2278:socket_connect_finish] 0-glusterfs: connection to 10.42.120.108:24007 failed (Connection timed out)
/var/log/glusterfs/var-lib-rancher-convoy-convoy-gluster-2cf6ae3e-205f-4892-a791-9f526adfda91-glusterfs-mounts-herocloud.log:[2015-12-21 14:30:02.549003] E [glusterfsd-mgmt.c:1818:mgmt_rpc_notify] 0-glusterfsd-mgmt: failed to connect with remote-host: glusterfs (Transport endpoint is not connected)
/var/log/glusterfs/var-lib-rancher-convoy-convoy-gluster-2cf6ae3e-205f-4892-a791-9f526adfda91-glusterfs-mounts-herocloud.log:[2015-12-21 14:30:02.549010] I [glusterfsd-mgmt.c:1824:mgmt_rpc_notify] 0-glusterfsd-mgmt: Exhausted all volfile servers

This means the server is not connected correctly. Also you found out that glusterfs server seems has trouble starting.

Could you go to glusterfs server container, and try

gluster volume list
gluster volume status

It's better to execute these in glusterfs server of the same host of Convoy failing.

yasker avatar Dec 28 '15 02:12 yasker

Mark as low priority because we don't actively support glusterfs now.

yasker avatar Aug 30 '16 21:08 yasker