glusterd2 icon indicating copy to clipboard operation
glusterd2 copied to clipboard

Published 20 hours ago • verified publisher icon gluster

Volume status showing port as -1 for few volumes in gcs setup

Open ksandha opened this issue 6 years ago • 6 comments

Checking the volume status of the volume in gd2, the port number to few of the volume is set as -1

Observed behavior

Created 90+ PVC and check the volume status for all the volumes. the ports of few of the volumes are showing -1

Expected/desired behavior

port number should be correct

Details on how to reproduce (minimal and precise)

  1. create 90 PVC on a gcs setup
  2. check the status of the volumes by logging in the gd2 pod.
Volume : pvc-6036b1a9-f6fd-11e8-a878-5254009d5095
+--------------------------------------+-------------------------------+-----------------------------------------------------------------------------------------+--------+-------+------+
|               BRICK ID               |             HOST              |                                          PATH                                           | ONLINE | PORT  | PID  |
+--------------------------------------+-------------------------------+-----------------------------------------------------------------------------------------+--------+-------+------+
| 3b32acca-2b3d-43d8-afcb-2e997a5d4f3a | gluster-kube3-0.glusterd2.gcs | /var/run/glusterd2/bricks/pvc-6036b1a9-f6fd-11e8-a878-5254009d5095/subvol1/brick1/brick | true   | 44176 | 8307 |
| ac9bb36f-7a6d-4922-b7e0-92d737add530 | gluster-kube2-0.glusterd2.gcs | /var/run/glusterd2/bricks/pvc-6036b1a9-f6fd-11e8-a878-5254009d5095/subvol1/brick2/brick | true   |    -1 | 4426 |
| cb5bb00e-1f1b-4fb2-bda8-88196809935f | gluster-kube1-0.glusterd2.gcs | /var/run/glusterd2/bricks/pvc-6036b1a9-f6fd-11e8-a878-5254009d5095/subvol1/brick3/brick | true   | 36991 | 4442 |
+--------------------------------------+-------------------------------+-----------------------------------------------------------------------------------------+--------+-------+------+
Volume : pvc-606f3fb9-f6fd-11e8-a878-5254009d5095
+--------------------------------------+-------------------------------+-----------------------------------------------------------------------------------------+--------+-------+------+
|               BRICK ID               |             HOST              |                                          PATH                                           | ONLINE | PORT  | PID  |
+--------------------------------------+-------------------------------+-----------------------------------------------------------------------------------------+--------+-------+------+
| bf526798-5b8e-46b8-bb86-d7b8454a7d67 | gluster-kube3-0.glusterd2.gcs | /var/run/glusterd2/bricks/pvc-606f3fb9-f6fd-11e8-a878-5254009d5095/subvol1/brick1/brick | true   | 34987 | 8393 |
| a61a03df-ab97-4561-aa83-d70a5de67287 | gluster-kube2-0.glusterd2.gcs | /var/run/glusterd2/bricks/pvc-606f3fb9-f6fd-11e8-a878-5254009d5095/subvol1/brick2/brick | true   | 39301 | 4459 |
| b7b777c9-f4b3-4ebb-9b07-87ddec632c3c | gluster-kube1-0.glusterd2.gcs | /var/run/glusterd2/bricks/pvc-606f3fb9-f6fd-11e8-a878-5254009d5095/subvol1/brick3/brick | true   | 39196 | 4475 |
+--------------------------------------+-------------------------------+-----------------------------------------------------------------------------------------+--------+-------+------+
Volume : pvc-60b0ee97-f6fd-11e8-a878-5254009d5095
+--------------------------------------+-------------------------------+-----------------------------------------------------------------------------------------+--------+-------+------+
|               BRICK ID               |             HOST              |                                          PATH                                           | ONLINE | PORT  | PID  |
+--------------------------------------+-------------------------------+-----------------------------------------------------------------------------------------+--------+-------+------+
| 77c7b7f4-54cc-4745-81bd-1e1930fb6653 | gluster-kube3-0.glusterd2.gcs | /var/run/glusterd2/bricks/pvc-60b0ee97-f6fd-11e8-a878-5254009d5095/subvol1/brick1/brick | true   | 39776 | 8507 |
| c47cad22-d333-4c3d-bae0-b8e73ab3a0ee | gluster-kube2-0.glusterd2.gcs | /var/run/glusterd2/bricks/pvc-60b0ee97-f6fd-11e8-a878-5254009d5095/subvol1/brick2/brick | true   | 33785 | 4514 |
| 72479e48-5972-4f0f-87fe-3bc0fef95032 | gluster-kube1-0.glusterd2.gcs | /var/run/glusterd2/bricks/pvc-60b0ee97-f6fd-11e8-a878-5254009d5095/subvol1/brick3/brick | true   | 35567 | 4530 |
+--------------------------------------+-------------------------------+-----------------------------------------------------------------------------------------+--------+-------+------+
Volume : pvc-60f1d413-f6fd-11e8-a878-5254009d5095
+--------------------------------------+-------------------------------+-----------------------------------------------------------------------------------------+--------+-------+------+
|               BRICK ID               |             HOST              |                                          PATH                                           | ONLINE | PORT  | PID  |
+--------------------------------------+-------------------------------+-----------------------------------------------------------------------------------------+--------+-------+------+
| 34b14c2c-0e5f-474d-a767-5cba979599d3 | gluster-kube3-0.glusterd2.gcs | /var/run/glusterd2/bricks/pvc-60f1d413-f6fd-11e8-a878-5254009d5095/subvol1/brick1/brick | true   | 38840 | 8556 |
| caa1b141-4195-42a0-a8cc-6708a71d8cd6 | gluster-kube2-0.glusterd2.gcs | /var/run/glusterd2/bricks/pvc-60f1d413-f6fd-11e8-a878-5254009d5095/subvol1/brick2/brick | true   | 43316 | 4547 |
| ba53c070-d7db-49f4-a967-a0c1620da2dc | gluster-kube1-0.glusterd2.gcs | /var/run/glusterd2/bricks/pvc-60f1d413-f6fd-11e8-a878-5254009d5095/subvol1/brick3/brick | true   | 40124 | 4561 |
+--------------------------------------+-------------------------------+-----------------------------------------------------------------------------------------+--------+-------+------+
Volume : pvc-6130f5a4-f6fd-11e8-a878-5254009d5095
+--------------------------------------+-------------------------------+-----------------------------------------------------------------------------------------+--------+-------+------+
|               BRICK ID               |             HOST              |                                          PATH                                           | ONLINE | PORT  | PID  |
+--------------------------------------+-------------------------------+-----------------------------------------------------------------------------------------+--------+-------+------+
| c335249c-0845-450c-92d5-e6d9484949f8 | gluster-kube1-0.glusterd2.gcs | /var/run/glusterd2/bricks/pvc-6130f5a4-f6fd-11e8-a878-5254009d5095/subvol1/brick1/brick | true   | 37957 | 4585 |
| 70e40c59-99a8-4960-9a9d-fc6218b9cf05 | gluster-kube3-0.glusterd2.gcs | /var/run/glusterd2/bricks/pvc-6130f5a4-f6fd-11e8-a878-5254009d5095/subvol1/brick2/brick | true   | 36395 | 8599 |
| 7292a74f-6dbe-4bfc-9efc-407f4cbc5ace | gluster-kube2-0.glusterd2.gcs | /var/run/glusterd2/bricks/pvc-6130f5a4-f6fd-11e8-a878-5254009d5095/subvol1/brick3/brick | true   | 34887 | 4570 |
+--------------------------------------+-------------------------------+-----------------------------------------------------------------------------------------+--------+-------+------+
Volume : pvc-616ba96f-f6fd-11e8-a878-5254009d5095
+--------------------------------------+-------------------------------+-----------------------------------------------------------------------------------------+--------+-------+------+
|               BRICK ID               |             HOST              |                                          PATH                                           | ONLINE | PORT  | PID  |
+--------------------------------------+-------------------------------+-----------------------------------------------------------------------------------------+--------+-------+------+
| fa3e19c3-7eb6-4323-8ae0-2f77a8b20d16 | gluster-kube3-0.glusterd2.gcs | /var/run/glusterd2/bricks/pvc-616ba96f-f6fd-11e8-a878-5254009d5095/subvol1/brick1/brick | true   | 42473 | 8632 |
| 145e365b-267f-4535-818d-c46a84871568 | gluster-kube2-0.glusterd2.gcs | /var/run/glusterd2/bricks/pvc-616ba96f-f6fd-11e8-a878-5254009d5095/subvol1/brick2/brick | true   | 35133 | 4594 |
| 4937306e-460b-485c-8d99-ddb1529c7ede | gluster-kube1-0.glusterd2.gcs | /var/run/glusterd2/bricks/pvc-616ba96f-f6fd-11e8-a878-5254009d5095/subvol1/brick3/brick | true   |    -1 | 4609 |
+--------------------------------------+-------------------------------+-----------------------------------------------------------------------------------------+--------+-------+------+
Volume : pvc-61a92c31-f6fd-11e8-a878-5254009d5095
+--------------------------------------+-------------------------------+-----------------------------------------------------------------------------------------+--------+-------+------+
|               BRICK ID               |             HOST              |                                          PATH                                           | ONLINE | PORT  | PID  |
+--------------------------------------+-------------------------------+-----------------------------------------------------------------------------------------+--------+-------+------+
| 0aae5cf0-f6ef-4c06-9117-55eb20e62298 | gluster-kube1-0.glusterd2.gcs | /var/run/glusterd2/bricks/pvc-61a92c31-f6fd-11e8-a878-5254009d5095/subvol1/brick1/brick | true   | 41861 | 4668 |
| 98c57bdf-16dd-4910-8fed-fa3aec6ed9cd | gluster-kube3-0.glusterd2.gcs | /var/run/glusterd2/bricks/pvc-61a92c31-f6fd-11e8-a878-5254009d5095/subvol1/brick2/brick | true   | 41190 | 8743 |
| 01679ba4-e941-4bb3-9ab3-85c2dbd52b67 | gluster-kube2-0.glusterd2.gcs | /var/run/glusterd2/bricks/pvc-61a92c31-f6fd-11e8-a878-5254009d5095/subvol1/brick3/brick | true   | 44505 | 4653 |
+--------------------------------------+-------------------------------+-----------------------------------------------------------------------------------------+--------+-------+------+
Volume : pvc-61f65936-f6fd-11e8-a878-5254009d5095
+--------------------------------------+-------------------------------+-----------------------------------------------------------------------------------------+--------+-------+------+
|               BRICK ID               |             HOST              |                                          PATH                                           | ONLINE | PORT  | PID  |
+--------------------------------------+-------------------------------+-----------------------------------------------------------------------------------------+--------+-------+------+
| 2ce82314-bce2-411b-bc96-8dab847f2436 | gluster-kube2-0.glusterd2.gcs | /var/run/glusterd2/bricks/pvc-61f65936-f6fd-11e8-a878-5254009d5095/subvol1/brick3/brick | true   | 46171 | 4689 |
| b35a9481-9d13-4615-9641-2a7dbbe7756b | gluster-kube1-0.glusterd2.gcs | /var/run/glusterd2/bricks/pvc-61f65936-f6fd-11e8-a878-5254009d5095/subvol1/brick1/brick | true   | 43430 | 4703 |
| 6d9bdd44-b6ec-41ee-891c-16a2f8fa1bd6 | gluster-kube3-0.glusterd2.gcs | /var/run/glusterd2/bricks/pvc-61f65936-f6fd-11e8-a878-5254009d5095/subvol1/brick2/brick | true   | 36463 | 8798 |
+--------------------------------------+-------------------------------+-----------------------------------------------------------------------------------------+--------+-------+------+
Volume : pvc-6230e700-f6fd-11e8-a878-5254009d5095
+--------------------------------------+-------------------------------+-----------------------------------------------------------------------------------------+--------+-------+------+
|               BRICK ID               |             HOST              |                                          PATH                                           | ONLINE | PORT  | PID  |
+--------------------------------------+-------------------------------+-----------------------------------------------------------------------------------------+--------+-------+------+
| aef1da4f-9e4e-4d21-b8bf-d2669920aa5f | gluster-kube1-0.glusterd2.gcs | /var/run/glusterd2/bricks/pvc-6230e700-f6fd-11e8-a878-5254009d5095/subvol1/brick3/brick | true   | 33642 | 4738 |
| a59e998a-fad3-45ac-99af-fece37682143 | gluster-kube3-0.glusterd2.gcs | /var/run/glusterd2/bricks/pvc-6230e700-f6fd-11e8-a878-5254009d5095/subvol1/brick1/brick | true   |    -1 | 8857 |
| ebbca4fb-d2e5-48e2-95f4-9bd478739db1 | gluster-kube2-0.glusterd2.gcs | /var/run/glusterd2/bricks/pvc-6230e700-f6fd-11e8-a878-5254009d5095/subvol1/brick2/brick | true   | 38375 | 4727 |
+--------------------------------------+-------------------------------+-----------------------------------------------------------------------------------------+--------+-------+------+
Volume : pvc-62749092-f6fd-11e8-a878-5254009d5095

Information about the environment:

  • Glusterd2 version used (e.g. v4.1.0 or master): glusterd2-5.0-0.dev.48.gitd996927.el7.x86_64
  • Operating system used: Centos
  • Glusterd2 compiled from sources, as a package (rpm/deb), or container: container
  • Using External ETCD: (yes/no, if yes ETCD version): yes
  • If container, which container image:
  • Using kubernetes, openshift, or direct install: kubernetes
  • If kubernetes/openshift, is gluster running inside kubernetes/openshift or outside: outside

Other useful information

  • glusterd2 config files from all nodes (default /etc/glusterd2/glusterd2.toml)
  • glusterd2 log files from all nodes (default /var/log/glusterd2/glusterd2.log)
  • ETCD configuration
  • Contents of uuid.toml from all nodes (default /var/lib/glusterd2/uuid.toml)
  • Output of statedump from any one of the node

Useful commands

  • To get glusterd2 version 
    [root@gluster-kube2-0 /]# glusterd2 --version

glusterd version: v6.0-dev.48.gitd996927 git SHA: d996927 go version: go1.9.4 go OS/arch: linux/amd64 [root@gluster-kube2-0 /]#

```
  • To get ETCD version 
    etcd --version
  • To get output of statedump 
    curl http://glusterd2-IP:glusterd2-Port/statedump

ksandha avatar Dec 03 '18 13:12 ksandha

@ksandha Could you attach the glusterd2.log? This seems like pmap_signin didn't happen?

atinmu avatar Dec 04 '18 10:12 atinmu

glusterd2_port.log

ksandha avatar Dec 07 '18 08:12 ksandha

From the attached log:

atin@dhcp35-96:~/Downloads$ grep -irns "volume started" glusterd2_port.log 770:time="2018-12-07 07:32:19.767744" level=info msg="volume started" reqid=4eeb98bc-351c-41cb-af67-224dbf9fd757 source="[volume-start.go:176:volumes.volumeStartHandler]" volume-name=pvc-38f2c390-f9f2-11e8-9e9c-52540044da85 771:time="2018-12-07 07:32:19.777919" level=info msg="volume started" reqid=24a30046-61dd-4b7d-b977-dd4466ee875b source="[volume-start.go:176:volumes.volumeStartHandler]" volume-name=pvc-395bce12-f9f2-11e8-9e9c-52540044da85 798:time="2018-12-07 07:32:22.441129" level=info msg="volume started" reqid=e0416413-0b0a-4cfb-9689-0eea02dafcb4 source="[volume-start.go:176:volumes.volumeStartHandler]" volume-name=pvc-3a7b3983-f9f2-11e8-9e9c-52540044da85 803:time="2018-12-07 07:32:22.684476" level=info msg="volume started" reqid=94dc1ff1-18ba-4ead-80e0-d4f58c96e1ab source="[volume-start.go:176:volumes.volumeStartHandler]" volume-name=pvc-3ab4e396-f9f2-11e8-9e9c-52540044da85 813:time="2018-12-07 07:32:24.156058" level=info msg="volume started" reqid=a0225070-d384-417c-8214-ca5beb9cac94 source="[volume-start.go:176:volumes.volumeStartHandler]" volume-name=pvc-3b8026fc-f9f2-11e8-9e9c-52540044da85 818:time="2018-12-07 07:32:24.795960" level=info msg="volume started" reqid=c8fcf45d-a1f7-46e5-af30-a62eb45f00ab source="[volume-start.go:176:volumes.volumeStartHandler]" volume-name=pvc-3c69c76a-f9f2-11e8-9e9c-52540044da85 822:time="2018-12-07 07:32:25.255692" level=info msg="volume started" reqid=f1b1a6e6-f680-4780-ba2e-acf24dae8a28 source="[volume-start.go:176:volumes.volumeStartHandler]" volume-name=pvc-3ca06eb5-f9f2-11e8-9e9c-52540044da85 18037:time="2018-12-07 08:25:57.638766" level=info msg="volume started" reqid=36986753-7528-48ad-8707-e5a0b0455f9a source="[volume-start.go:176:volumes.volumeStartHandler]" volume-name=pvc-b72dab2e-f9f9-11e8-9e9c-52540044da85 18047:time="2018-12-07 08:25:59.423166" level=info msg="volume started" reqid=9d07c336-7a77-4bfb-bd47-6c452cd328fe source="[volume-start.go:176:volumes.volumeStartHandler]" volume-name=pvc-b8cbc54d-f9f9-11e8-9e9c-52540044da85 18085:time="2018-12-07 08:26:02.472205" level=info msg="volume started" reqid=7cff7a14-a07a-4739-aab6-4a4d97c7fed0 source="[volume-start.go:176:volumes.volumeStartHandler]" volume-name=pvc-ba1272cb-f9f9-11e8-9e9c-52540044da85 18104:time="2018-12-07 08:26:04.712503" level=info msg="volume started" reqid=9abf1c4e-884b-4436-9a5e-bbae9f73d00a source="[volume-start.go:176:volumes.volumeStartHandler]" volume-name=pvc-bb5cd5d1-f9f9-11e8-9e9c-52540044da85 18117:time="2018-12-07 08:26:07.247192" level=info msg="volume started" reqid=4c7fd0ef-bbbb-45e9-8f0b-60887faded17 source="[volume-start.go:176:volumes.volumeStartHandler]" volume-name=pvc-bcae0c89-f9f9-11e8-9e9c-52540044da85 18122:time="2018-12-07 08:26:07.474766" level=info msg="volume started" reqid=7f252a67-3109-4f7d-a097-6c9ff62000dd source="[volume-start.go:176:volumes.volumeStartHandler]" volume-name=pvc-bd4c88d9-f9f9-11e8-9e9c-52540044da85 18158:time="2018-12-07 08:26:10.358618" level=info msg="volume started" reqid=73d58c6d-d05f-4359-a5c6-e7c44f6e1c55 source="[volume-start.go:176:volumes.volumeStartHandler]" volume-name=pvc-bdf7bc5a-f9f9-11e8-9e9c-52540044da85 18162:time="2018-12-07 08:26:10.575301" level=info msg="volume started" reqid=86b2f502-699e-4dff-aba7-1f9f6650f13d source="[volume-start.go:176:volumes.volumeStartHandler]" volume-name=pvc-be2fe1d2-f9f9-11e8-9e9c-52540044da85 18181:time="2018-12-07 08:26:11.992914" level=info msg="volume started" reqid=0226b5a1-f954-460b-98a5-66b9cbd81cb9 source="[volume-start.go:176:volumes.volumeStartHandler]" volume-name=pvc-c03d2b0d-f9f9-11e8-9e9c-52540044da85 18204:time="2018-12-07 08:26:14.784843" level=info msg="volume started" reqid=9c4d455a-6f6a-49fc-b54a-0442fbbea100 source="[volume-start.go:176:volumes.volumeStartHandler]" volume-name=pvc-bb999b3a-f9f9-11e8-9e9c-52540044da85 18219:time="2018-12-07 08:26:17.155912" level=info msg="volume started" reqid=0f0f06ec-653f-4766-9023-937e0a807dc5 source="[volume-start.go:176:volumes.volumeStartHandler]" volume-name=pvc-be65a7f1-f9f9-11e8-9e9c-52540044da85 18272:time="2018-12-07 08:26:20.618948" level=info msg="volume started" reqid=f1ca54be-7587-437f-a784-8df1414a7bc2 source="[volume-start.go:176:volumes.volumeStartHandler]" volume-name=pvc-bce35bff-f9f9-11e8-9e9c-52540044da85 18333:time="2018-12-07 08:26:24.727759" level=info msg="volume started" reqid=7e835436-5ab3-4ddb-a1df-0f1b58f4e8e9 source="[volume-start.go:176:volumes.volumeStartHandler]" volume-name=pvc-b9738086-f9f9-11e8-9e9c-52540044da85 18335:time="2018-12-07 08:26:24.930461" level=info msg="volume started" reqid=619cf657-ff3a-465b-a186-e74d58b6d9b3 source="[volume-start.go:176:volumes.volumeStartHandler]" volume-name=pvc-ba4ebccb-f9f9-11e8-9e9c-52540044da85 18367:time="2018-12-07 08:26:29.179987" level=info msg="volume started" reqid=fe0f67ed-bc6c-4bc9-8b78-cc75ed5f6036 source="[volume-start.go:176:volumes.volumeStartHandler]" volume-name=pvc-c2978a49-f9f9-11e8-9e9c-52540044da85 18401:time="2018-12-07 08:26:32.228216" level=info msg="volume started" reqid=779f27f2-841b-4423-9173-d8cb4d2dd536 source="[volume-start.go:176:volumes.volumeStartHandler]" volume-name=pvc-c337cce4-f9f9-11e8-9e9c-52540044da85 18415:time="2018-12-07 08:26:32.833569" level=info msg="volume started" reqid=cb3be059-a93d-4cf7-9104-4d0ead36348b source="[volume-start.go:176:volumes.volumeStartHandler]" volume-name=pvc-c36f1c95-f9f9-11e8-9e9c-52540044da85 18425:time="2018-12-07 08:26:33.286549" level=info msg="volume started" reqid=39acf236-6d10-42e7-9159-7150b0709d92 source="[volume-start.go:176:volumes.volumeStartHandler]" volume-name=pvc-c3a3b299-f9f9-11e8-9e9c-52540044da85 18435:time="2018-12-07 08:26:33.803032" level=info msg="volume started" reqid=d587e2ee-b8f7-47b7-9502-af3a4f89491d source="[volume-start.go:176:volumes.volumeStartHandler]" volume-name=pvc-c402d517-f9f9-11e8-9e9c-52540044da85 18512:time="2018-12-07 08:26:39.487224" level=info msg="volume started" reqid=e12545a1-768d-46eb-aacf-391b9bd50d3e source="[volume-start.go:176:volumes.volumeStartHandler]" volume-name=pvc-c5855480-f9f9-11e8-9e9c-52540044da85 18550:time="2018-12-07 08:26:42.495368" level=info msg="volume started" reqid=ce57e843-9ee4-4774-92f2-746838942675 source="[volume-start.go:176:volumes.volumeStartHandler]" volume-name=pvc-c66187d2-f9f9-11e8-9e9c-52540044da85 18571:time="2018-12-07 08:26:46.364876" level=info msg="volume started" reqid=8de12868-9d41-4e4c-9032-e9ddc1a7da7e source="[volume-start.go:176:volumes.volumeStartHandler]" volume-name=pvc-c73f99f6-f9f9-11e8-9e9c-52540044da85 18573:time="2018-12-07 08:26:46.607683" level=info msg="volume started" reqid=1f5bacef-f32e-4ce7-b44a-2276460d310f source="[volume-start.go:176:volumes.volumeStartHandler]" volume-name=pvc-c78ccfba-f9f9-11e8-9e9c-52540044da85 18599:time="2018-12-07 08:26:48.973431" level=info msg="volume started" reqid=c78e474f-749e-443c-9dd6-dcb28d794d68 source="[volume-start.go:176:volumes.volumeStartHandler]" volume-name=pvc-c7c441a7-f9f9-11e8-9e9c-52540044da85 18613:time="2018-12-07 08:26:49.942045" level=info msg="volume started" reqid=b954ea50-3993-4a94-9952-da95eea5ba1c source="[volume-start.go:176:volumes.volumeStartHandler]" volume-name=pvc-c7fb8996-f9f9-11e8-9e9c-52540044da85 18643:time="2018-12-07 08:26:51.997035" level=info msg="volume started" reqid=dfe8bafb-1cb3-4326-88cc-902e9e497303 source="[volume-start.go:176:volumes.volumeStartHandler]" volume-name=pvc-c8a01998-f9f9-11e8-9e9c-52540044da85 18659:time="2018-12-07 08:26:52.858181" level=info msg="volume started" reqid=b2acef47-530a-4e28-9bc1-fa29db9da939 source="[volume-start.go:176:volumes.volumeStartHandler]" volume-name=pvc-c8db16fc-f9f9-11e8-9e9c-52540044da85 18679:time="2018-12-07 08:26:54.002376" level=info msg="volume started" reqid=fbe147ef-90db-4992-a177-20b11bb77cac source="[volume-start.go:176:volumes.volumeStartHandler]" volume-name=pvc-c91283dc-f9f9-11e8-9e9c-52540044da85 18736:time="2018-12-07 08:26:58.180626" level=info msg="volume started" reqid=fa570876-64b1-43b8-ae45-fe8cb950333d source="[volume-start.go:176:volumes.volumeStartHandler]" volume-name=pvc-ca165f34-f9f9-11e8-9e9c-52540044da85 18749:time="2018-12-07 08:26:59.168778" level=info msg="volume started" reqid=0043490e-6caa-4c37-81f8-f999fcf420fa source="[volume-start.go:176:volumes.volumeStartHandler]" volume-name=pvc-c0e4aa7e-f9f9-11e8-9e9c-52540044da85 18764:time="2018-12-07 08:27:00.309447" level=info msg="volume started" reqid=6b3aa008-6a61-41e5-843a-f5a18d430c7e source="[volume-start.go:176:volumes.volumeStartHandler]" volume-name=pvc-c1b8625e-f9f9-11e8-9e9c-52540044da85 18799:time="2018-12-07 08:27:03.376152" level=info msg="volume started" reqid=75b4b345-41e5-47d7-95aa-a62f48f70593 source="[volume-start.go:176:volumes.volumeStartHandler]" volume-name=pvc-c17e5c12-f9f9-11e8-9e9c-52540044da85 18812:time="2018-12-07 08:27:04.253258" level=info msg="volume started" reqid=3b78b7c9-c6cf-4764-8f93-e40600a2acde source="[volume-start.go:176:volumes.volumeStartHandler]" volume-name=pvc-ca857378-f9f9-11e8-9e9c-52540044da85 18833:time="2018-12-07 08:27:07.763421" level=info msg="volume started" reqid=11f04ff5-a483-4a0a-8c48-2a1f1d2fa4b1 source="[volume-start.go:176:volumes.volumeStartHandler]" volume-name=pvc-cb2c64f2-f9f9-11e8-9e9c-52540044da85 18870:time="2018-12-07 08:27:11.419692" level=info msg="volume started" reqid=a11a9724-5e7a-47e7-9bd4-31cfb1af3720 source="[volume-start.go:176:volumes.volumeStartHandler]" volume-name=pvc-cc25c3cf-f9f9-11e8-9e9c-52540044da85 18923:time="2018-12-07 08:27:15.178867" level=info msg="volume started" reqid=2c36d96f-7dc8-4526-b5b8-c3eb1fcc05e6 source="[volume-start.go:176:volumes.volumeStartHandler]" volume-name=pvc-cd20af06-f9f9-11e8-9e9c-52540044da85 18931:time="2018-12-07 08:27:15.831228" level=info msg="volume started" reqid=d6a755b3-152a-4459-859c-dfaa20254357 source="[volume-start.go:176:volumes.volumeStartHandler]" volume-name=pvc-cd5b9e18-f9f9-11e8-9e9c-52540044da85 19008:time="2018-12-07 08:27:21.887982" level=info msg="volume started" reqid=e1eb8b60-b50c-4614-a7f6-54e96b6adf69 source="[volume-start.go:176:volumes.volumeStartHandler]" volume-name=pvc-ced9c840-f9f9-11e8-9e9c-52540044da85 19033:time="2018-12-07 08:27:23.894293" level=info msg="volume started" reqid=3945d705-3337-4cff-9382-a21aceff6ecd source="[volume-start.go:176:volumes.volumeStartHandler]" volume-name=pvc-cf791557-f9f9-11e8-9e9c-52540044da85 19050:time="2018-12-07 08:27:24.875991" level=info msg="volume started" reqid=a98383ae-978c-4e22-bfcd-acda6b38d1d3 source="[volume-start.go:176:volumes.volumeStartHandler]" volume-name=pvc-cfebb9a1-f9f9-11e8-9e9c-52540044da85 19074:time="2018-12-07 08:27:26.634744" level=info msg="volume started" reqid=ca05e7d7-1c5f-4318-bbb5-4711921c9604 source="[volume-start.go:176:volumes.volumeStartHandler]" volume-name=pvc-d05ccad4-f9f9-11e8-9e9c-52540044da85 19124:time="2018-12-07 08:27:31.330375" level=info msg="volume started" reqid=a0b96524-fc2c-496e-88cd-439ad83af88c source="[volume-start.go:176:volumes.volumeStartHandler]" volume-name=pvc-d1742cc2-f9f9-11e8-9e9c-52540044da85 21219:time="2018-12-07 08:31:57.939200" level=info msg="volume started" reqid=35eb7bb3-7550-42ed-aaba-b646c0206cb6 source="[volume-start.go:176:volumes.volumeStartHandler]" volume-name=pvc-d1e97eed-f9f9-11e8-9e9c-52540044da85 21223:time="2018-12-07 08:31:58.258278" level=info msg="volume started" reqid=e3380c00-ab7b-4b3f-9645-cf309bd5702e source="[volume-start.go:176:volumes.volumeStartHandler]" volume-name=pvc-d2c6825c-f9f9-11e8-9e9c-52540044da85 21263:time="2018-12-07 08:32:01.450502" level=info msg="volume started" reqid=0e35b5ef-f9a9-45fa-9d3c-d4ac2df9f32a source="[volume-start.go:176:volumes.volumeStartHandler]" volume-name=pvc-d3b8a3d1-f9f9-11e8-9e9c-52540044da85 21264:time="2018-12-07 08:32:01.491204" level=info msg="volume started" reqid=4ddb99fb-7bb0-4938-878d-cbf0919e6f1a source="[volume-start.go:176:volumes.volumeStartHandler]" volume-name=pvc-d37c3d6b-f9f9-11e8-9e9c-52540044da85 21335:time="2018-12-07 08:32:05.505616" level=info msg="volume started" reqid=5a8e6c65-1d40-40f8-8196-f4cde5db37ed source="[volume-start.go:176:volumes.volumeStartHandler]" volume-name=pvc-d49a0d5f-f9f9-11e8-9e9c-52540044da85 21346:time="2018-12-07 08:32:06.027302" level=info msg="volume started" reqid=f26efcb1-9e5d-46d3-a6a7-bbec05793886 source="[volume-start.go:176:volumes.volumeStartHandler]" volume-name=pvc-d5085523-f9f9-11e8-9e9c-52540044da85 21374:time="2018-12-07 08:32:07.896879" level=info msg="volume started" reqid=39b1df58-08fe-4631-a737-908f18293284 source="[volume-start.go:176:volumes.volumeStartHandler]" volume-name=pvc-d54370fd-f9f9-11e8-9e9c-52540044da85 21396:time="2018-12-07 08:32:08.963607" level=info msg="volume started" reqid=039d116f-4aff-4a4d-a12e-d6c52b4ecfef source="[volume-start.go:176:volumes.volumeStartHandler]" volume-name=pvc-d580243d-f9f9-11e8-9e9c-52540044da85 21401:time="2018-12-07 08:32:09.061783" level=info msg="volume started" reqid=a1120e0d-5cde-4357-b672-d029f7fc4adc source="[volume-start.go:176:volumes.volumeStartHandler]" volume-name=pvc-d3396e3a-f9f9-11e8-9e9c-52540044da85 21447:time="2018-12-07 08:32:12.253810" level=info msg="volume started" reqid=5acf8302-a2ce-4d16-afda-16ff6b15d1fa source="[volume-start.go:176:volumes.volumeStartHandler]" volume-name=pvc-d61d4969-f9f9-11e8-9e9c-52540044da85 21454:time="2018-12-07 08:32:12.620646" level=info msg="volume started" reqid=099a0e81-6f4d-40a7-9d05-4b5e4867b6c8 source="[volume-start.go:176:volumes.volumeStartHandler]" volume-name=pvc-d5ea4dbd-f9f9-11e8-9e9c-52540044da85 21590:time="2018-12-07 08:32:20.882103" level=info msg="volume started" reqid=3e2c73e9-ae02-4c42-a2b4-0d556b4b6b60 source="[volume-start.go:176:volumes.volumeStartHandler]" volume-name=pvc-d8b2256f-f9f9-11e8-9e9c-52540044da85 21646:time="2018-12-07 08:32:24.546108" level=info msg="volume started" reqid=96bbb874-7d41-4a4b-a495-1427452ee295 source="[volume-start.go:176:volumes.volumeStartHandler]" volume-name=pvc-d929b1d8-f9f9-11e8-9e9c-52540044da85 21691:time="2018-12-07 08:32:27.979662" level=info msg="volume started" reqid=9cbc092f-09ed-4f07-af96-1c0d476731a9 source="[volume-start.go:176:volumes.volumeStartHandler]" volume-name=pvc-d99e2ef1-f9f9-11e8-9e9c-52540044da85 21717:time="2018-12-07 08:32:28.889339" level=info msg="volume started" reqid=9dc19764-7847-4854-acb3-e439e1472f5f source="[volume-start.go:176:volumes.volumeStartHandler]" volume-name=pvc-d7862566-f9f9-11e8-9e9c-52540044da85 21720:time="2018-12-07 08:32:29.023712" level=info msg="volume started" reqid=7b70c023-bf66-452b-b085-75980ea62152 source="[volume-start.go:176:volumes.volumeStartHandler]" volume-name=pvc-d6c9246b-f9f9-11e8-9e9c-52540044da85 21723:time="2018-12-07 08:32:29.059992" level=info msg="volume started" reqid=96f1ade0-f67f-421e-9065-0fe59b49ef33 source="[volume-start.go:176:volumes.volumeStartHandler]" volume-name=pvc-d9d47546-f9f9-11e8-9e9c-52540044da85 21750:time="2018-12-07 08:32:31.450834" level=info msg="volume started" reqid=6fb178c7-03ed-4b2b-aa33-cc26301e26b6 source="[volume-start.go:176:volumes.volumeStartHandler]" volume-name=pvc-da119ed6-f9f9-11e8-9e9c-52540044da85 21791:time="2018-12-07 08:32:36.088414" level=info msg="volume started" reqid=5d462053-1459-4e15-983b-d182d268b3f8 source="[volume-start.go:176:volumes.volumeStartHandler]" volume-name=pvc-db46c7b7-f9f9-11e8-9e9c-52540044da85 21821:time="2018-12-07 08:32:38.016287" level=info msg="volume started" reqid=87718c3f-3de5-401c-99ca-62a1eb716c3b source="[volume-start.go:176:volumes.volumeStartHandler]" volume-name=pvc-dbf8ac84-f9f9-11e8-9e9c-52540044da85 21860:time="2018-12-07 08:32:40.299833" level=info msg="volume started" reqid=792366b2-d313-4310-89f8-ec54036696b7 source="[volume-start.go:176:volumes.volumeStartHandler]" volume-name=pvc-dc71ec91-f9f9-11e8-9e9c-52540044da85 21878:time="2018-12-07 08:32:41.189673" level=info msg="volume started" reqid=5433f3f4-6b6d-48d7-8249-130dc2c4387f source="[volume-start.go:176:volumes.volumeStartHandler]" volume-name=pvc-dca68c54-f9f9-11e8-9e9c-52540044da85 21993:time="2018-12-07 08:32:48.965530" level=info msg="volume started" reqid=ad361acc-a1df-4ae2-9f1f-d4f6bf25964d source="[volume-start.go:176:volumes.volumeStartHandler]" volume-name=pvc-de54f29f-f9f9-11e8-9e9c-52540044da85 22006:time="2018-12-07 08:32:49.402633" level=info msg="volume started" reqid=e28478fa-dcd0-4578-9638-3bf63cb0652c source="[volume-start.go:176:volumes.volumeStartHandler]" volume-name=pvc-de9d9d77-f9f9-11e8-9e9c-52540044da85 22072:time="2018-12-07 08:32:53.895475" level=info msg="volume started" reqid=4f896614-8577-4a8d-b1d3-db49c639b3bb source="[volume-start.go:176:volumes.volumeStartHandler]" volume-name=pvc-dfcc9f86-f9f9-11e8-9e9c-52540044da85 22202:time="2018-12-07 08:33:02.270430" level=info msg="volume started" reqid=74e9fbfd-1f77-4963-93b2-fe7ffb721803 source="[volume-start.go:176:volumes.volumeStartHandler]" volume-name=pvc-e0e313be-f9f9-11e8-9e9c-52540044da85 22279:time="2018-12-07 08:33:07.140096" level=info msg="volume started" reqid=2549afb5-0ebf-4532-b423-4dc74f12997f source="[volume-start.go:176:volumes.volumeStartHandler]" volume-name=pvc-e297a6b9-f9f9-11e8-9e9c-52540044da85 22352:time="2018-12-07 08:33:11.654053" level=info msg="volume started" reqid=bea1c162-bb77-4517-9b3a-d351527bff2b source="[volume-start.go:176:volumes.volumeStartHandler]" volume-name=pvc-e3845da6-f9f9-11e8-9e9c-52540044da85

atin@dhcp35-96:~/Downloads$ grep -irns "volume started" glusterd2_port.log | wc -l 77

^^ 77 volumes (which isn't matching with 90+ PVCs what you mentioned in the original report) are in started state and I don't find the PVs in question where one of the bricks reported as port = -1 in the above list.

@ksandha ^^ is this the correct set of logs?

atinmu avatar Dec 10 '18 05:12 atinmu 

[root@gluster-kube2-0 /]# glustercli volume status
Volume : pvc-16238f77-fc49-11e8-b05d-52540001ce00
+--------------------------------------+-------------------------------+-----------------------------------------------------------------------------------------+--------+-------+-------+
|               BRICK ID               |             HOST              |                                          PATH                                           | ONLINE | PORT  |  PID  |
+--------------------------------------+-------------------------------+-----------------------------------------------------------------------------------------+--------+-------+-------+
| 6b10e8f5-1f66-4ba2-9cb0-29026fb5a0b8 | gluster-kube3-0.glusterd2.gcs | /var/run/glusterd2/bricks/pvc-16238f77-fc49-11e8-b05d-52540001ce00/subvol1/brick1/brick | true   | 46537 |    89 |
| c96c84e2-541a-48d8-bb8f-911d4f2fb443 | gluster-kube2-0.glusterd2.gcs | /var/run/glusterd2/bricks/pvc-16238f77-fc49-11e8-b05d-52540001ce00/subvol1/brick2/brick | true   |    -1 | 16984 |
| f2de90d8-60dd-42e3-a2e6-a295386dc6b5 | gluster-kube1-0.glusterd2.gcs | /var/run/glusterd2/bricks/pvc-16238f77-fc49-11e8-b05d-52540001ce00/subvol1/brick3/brick | true   | 41318 | 16980 |
+--------------------------------------+-------------------------------+-----------------------------------------------------------------------------------------+--------+-------+-------+
Volume : pvc-16543967-fc49-11e8-b05d-52540001ce00
+--------------------------------------+-------------------------------+-----------------------------------------------------------------------------------------+--------+-------+-------+
|               BRICK ID               |             HOST              |                                          PATH                                           | ONLINE | PORT  |  PID  |
+--------------------------------------+-------------------------------+-----------------------------------------------------------------------------------------+--------+-------+-------+
| d07b9fda-60a1-4ec0-9a02-f1ed8e49e7b7 | gluster-kube3-0.glusterd2.gcs | /var/run/glusterd2/bricks/pvc-16543967-fc49-11e8-b05d-52540001ce00/subvol1/brick1/brick | true   | 39477 |   110 |
| a0a961a1-3994-437f-aec5-5fbeca71d3cf | gluster-kube2-0.glusterd2.gcs | /var/run/glusterd2/bricks/pvc-16543967-fc49-11e8-b05d-52540001ce00/subvol1/brick2/brick | true   |    -1 | 17049 |
| c5a638d0-c8f3-4608-ada5-5ac96a6103d4 | gluster-kube1-0.glusterd2.gcs | /var/run/glusterd2/bricks/pvc-16543967-fc49-11e8-b05d-52540001ce00/subvol1/brick3/brick | true   | 45450 | 17044 |
+--------------------------------------+-------------------------------+-----------------------------------------------------------------------------------------+--------+-------+-------+
Volume : pvc-1684329d-fc49-11e8-b05d-52540001ce00
+--------------------------------------+-------------------------------+-----------------------------------------------------------------------------------------+--------+-------+-------+
|               BRICK ID               |             HOST              |                                          PATH                                           | ONLINE | PORT  |  PID  |
+--------------------------------------+-------------------------------+-----------------------------------------------------------------------------------------+--------+-------+-------+
| c0e34814-b259-4b83-bc74-d401203d2889 | gluster-kube3-0.glusterd2.gcs | /var/run/glusterd2/bricks/pvc-1684329d-fc49-11e8-b05d-52540001ce00/subvol1/brick1/brick | true   | 38254 |   131 |
| dc5aba34-186f-4fa8-95e7-3346ed19f32d | gluster-kube2-0.glusterd2.gcs | /var/run/glusterd2/bricks/pvc-1684329d-fc49-11e8-b05d-52540001ce00/subvol1/brick2/brick | true   |    -1 | 16943 |
| 37bed68c-557b-4568-a42d-2e29a81fc5bf | gluster-kube1-0.glusterd2.gcs | /var/run/glusterd2/bricks/pvc-1684329d-fc49-11e8-b05d-52540001ce00/subvol1/brick3/brick | true   | 36946 | 16945 |
+--------------------------------------+-------------------------------+-----------------------------------------------------------------------------------------+--------+-------+-------+
Volume : pvc-16c26573-fc49-11e8-b05d-52540001ce00
+--------------------------------------+-------------------------------+-----------------------------------------------------------------------------------------+--------+-------+-------+
|               BRICK ID               |             HOST              |                                          PATH                                           | ONLINE | PORT  |  PID  |
+--------------------------------------+-------------------------------+-----------------------------------------------------------------------------------------+--------+-------+-------+
| e15730c3-2ae4-4346-9305-42dc019bbccd | gluster-kube2-0.glusterd2.gcs | /var/run/glusterd2/bricks/pvc-16c26573-fc49-11e8-b05d-52540001ce00/subvol1/brick2/brick | true   |    -1 | 16803 |
| 81aa92ba-c151-4538-a0fe-90a5da712c76 | gluster-kube1-0.glusterd2.gcs | /var/run/glusterd2/bricks/pvc-16c26573-fc49-11e8-b05d-52540001ce00/subvol1/brick3/brick | true   | 46715 | 16800 |
| 48773c21-f002-4185-b102-297ed28eb5d1 | gluster-kube3-0.glusterd2.gcs | /var/run/glusterd2/bricks/pvc-16c26573-fc49-11e8-b05d-52540001ce00/subvol1/brick1/brick | true   | 32907 |   152 |
+--------------------------------------+-------------------------------+-----------------------------------------------------------------------------------------+--------+-------+-------+
Volume : pvc-16f45087-fc49-11e8-b05d-52540001ce00
+--------------------------------------+-------------------------------+-----------------------------------------------------------------------------------------+--------+-------+-------+
|               BRICK ID               |             HOST              |                                          PATH                                           | ONLINE | PORT  |  PID  |
+--------------------------------------+-------------------------------+-----------------------------------------------------------------------------------------+--------+-------+-------+
| 5b66aa18-313e-47f2-97f5-411fdbee0219 | gluster-kube3-0.glusterd2.gcs | /var/run/glusterd2/bricks/pvc-16f45087-fc49-11e8-b05d-52540001ce00/subvol1/brick1/brick | true   | 37824 |   173 |
| f6efe014-998b-44ec-b60a-d639201badcf | gluster-kube2-0.glusterd2.gcs | /var/run/glusterd2/bricks/pvc-16f45087-fc49-11e8-b05d-52540001ce00/subvol1/brick2/brick | true   |    -1 | 16835 |
| 55999ccf-4e19-479d-9a34-f44bddd9bafb | gluster-kube1-0.glusterd2.gcs | /var/run/glusterd2/bricks/pvc-16f45087-fc49-11e8-b05d-52540001ce00/subvol1/brick3/brick | true   | 33743 | 16831 |
+--------------------------------------+-------------------------------+-----------------------------------------------------------------------------------------+--------+-------+-------+
Volume : pvc-171f902f-fc49-11e8-b05d-52540001ce00
+--------------------------------------+-------------------------------+-----------------------------------------------------------------------------------------+--------+-------+-------+
|               BRICK ID               |             HOST              |                                          PATH                                           | ONLINE | PORT  |  PID  |
+--------------------------------------+-------------------------------+-----------------------------------------------------------------------------------------+--------+-------+-------+
| 5bed0dd3-5364-4900-ad05-c4481ae8d5e5 | gluster-kube3-0.glusterd2.gcs | /var/run/glusterd2/bricks/pvc-171f902f-fc49-11e8-b05d-52540001ce00/subvol1/brick1/brick | true   | 36441 |   194 |
| fccb45b0-e770-4dbc-8d03-0aa9d61adca5 | gluster-kube2-0.glusterd2.gcs | /var/run/glusterd2/bricks/pvc-171f902f-fc49-11e8-b05d-52540001ce00/subvol1/brick2/brick | true   |    -1 | 16921 |
| c5db4bb7-2f98-4291-9528-d25ef3dcf64d | gluster-kube1-0.glusterd2.gcs | /var/run/glusterd2/bricks/pvc-171f902f-fc49-11e8-b05d-52540001ce00/subvol1/brick3/brick | true   | 42477 | 16917 |
+--------------------------------------+-------------------------------+-----------------------------------------------------------------------------------------+--------+-------+-------+
Volume : pvc-17535529-fc49-11e8-b05d-52540001ce00
+--------------------------------------+-------------------------------+-----------------------------------------------------------------------------------------+--------+-------+-------+
|               BRICK ID               |             HOST              |                                          PATH                                           | ONLINE | PORT  |  PID  |
+--------------------------------------+-------------------------------+-----------------------------------------------------------------------------------------+--------+-------+-------+
| 1b5814f9-78e1-46df-b47e-50bce9f82dd8 | gluster-kube3-0.glusterd2.gcs | /var/run/glusterd2/bricks/pvc-17535529-fc49-11e8-b05d-52540001ce00/subvol1/brick1/brick | true   | 36634 |   215 |
| a8d706ac-556b-4ca9-ac55-0c211fb8c315 | gluster-kube2-0.glusterd2.gcs | /var/run/glusterd2/bricks/pvc-17535529-fc49-11e8-b05d-52540001ce00/subvol1/brick2/brick | true   |    -1 | 16857 |
| 9dc8770d-95a9-45e4-a27a-0c82a1435c48 | gluster-kube1-0.glusterd2.gcs | /var/run/glusterd2/bricks/pvc-17535529-fc49-11e8-b05d-52540001ce00/subvol1/brick3/brick | true   | 41720 | 16852 |
+--------------------------------------+-------------------------------+-----------------------------------------------------------------------------------------+--------+-------+-------+
Volume : pvc-17a2f0fe-fc49-11e8-b05d-52540001ce00
+--------------------------------------+-------------------------------+-----------------------------------------------------------------------------------------+--------+-------+-------+
|               BRICK ID               |             HOST              |                                          PATH                                           | ONLINE | PORT  |  PID  |
+--------------------------------------+-------------------------------+-----------------------------------------------------------------------------------------+--------+-------+-------+
| e926f72c-d973-4cbd-ad8a-db29a57ec964 | gluster-kube1-0.glusterd2.gcs | /var/run/glusterd2/bricks/pvc-17a2f0fe-fc49-11e8-b05d-52540001ce00/subvol1/brick3/brick | true   | 32902 | 17154 |
| f11cdf0c-6f01-4b47-899f-d370defd7fbc | gluster-kube3-0.glusterd2.gcs | /var/run/glusterd2/bricks/pvc-17a2f0fe-fc49-11e8-b05d-52540001ce00/subvol1/brick1/brick | true   | 43492 |   236 |
| a226be88-90ba-448f-b82d-46ee208365db | gluster-kube2-0.glusterd2.gcs | /var/run/glusterd2/bricks/pvc-17a2f0fe-fc49-11e8-b05d-52540001ce00/subvol1/brick2/brick | true   |    -1 | 17157 |
+--------------------------------------+-------------------------------+-----------------------------------------------------------------------------------------+--------+-------+-------+
Volume : pvc-17e1891c-fc49-11e8-b05d-52540001ce00
+--------------------------------------+-------------------------------+-----------------------------------------------------------------------------------------+--------+-------+-------+
|               BRICK ID               |             HOST              |                                          PATH                                           | ONLINE | PORT  |  PID  |
+--------------------------------------+-------------------------------+-----------------------------------------------------------------------------------------+--------+-------+-------+
| 60e9e82c-5daa-4524-a57b-68b793a8cf8e | gluster-kube3-0.glusterd2.gcs | /var/run/glusterd2/bricks/pvc-17e1891c-fc49-11e8-b05d-52540001ce00/subvol1/brick1/brick | true   | 45188 |   257 |
| 1da0dda6-f923-4669-bf73-1e18a683bcff | gluster-kube2-0.glusterd2.gcs | /var/run/glusterd2/bricks/pvc-17e1891c-fc49-11e8-b05d-52540001ce00/subvol1/brick2/brick | true   |    -1 | 16949 |
| 43a8eade-d0b9-42c9-81a3-7ed479851352 | gluster-kube1-0.glusterd2.gcs | /var/run/glusterd2/bricks/pvc-17e1891c-fc49-11e8-b05d-52540001ce00/subvol1/brick3/brick | true   | 35237 | 16939 |
+--------------------------------------+-------------------------------+-----------------------------------------------------------------------------------------+--------+-------+-------+
Volume : pvc-1815bd3b-fc49-11e8-b05d-52540001ce00
+--------------------------------------+-------------------------------+-----------------------------------------------------------------------------------------+--------+-------+-------+
|               BRICK ID               |             HOST              |                                          PATH                                           | ONLINE | PORT  |  PID  |
+--------------------------------------+-------------------------------+-----------------------------------------------------------------------------------------+--------+-------+-------+
| a25ae8b0-d75b-4e58-b291-ab0776de32c1 | gluster-kube3-0.glusterd2.gcs | /var/run/glusterd2/bricks/pvc-1815bd3b-fc49-11e8-b05d-52540001ce00/subvol1/brick1/brick | true   | 39530 |   278 |
| 08c9a5ec-3059-4a62-8bd5-dcd220a60151 | gluster-kube2-0.glusterd2.gcs | /var/run/glusterd2/bricks/pvc-1815bd3b-fc49-11e8-b05d-52540001ce00/subvol1/brick2/brick | true   |    -1 | 17070 |
| df52b06f-d88d-4884-b586-75d4c996c168 | gluster-kube1-0.glusterd2.gcs | /var/run/glusterd2/bricks/pvc-1815bd3b-fc49-11e8-b05d-52540001ce00/subvol1/brick3/brick | true   | 41833 | 17065 |
+--------------------------------------+-------------------------------+-----------------------------------------------------------------------------------------+--------+-------+-------+
Volume : pvc-1848109b-fc49-11e8-b05d-52540001ce00
+--------------------------------------+-------------------------------+-----------------------------------------------------------------------------------------+--------+-------+-------+
|               BRICK ID               |             HOST              |                                          PATH                                           | ONLINE | PORT  |  PID  |
+--------------------------------------+-------------------------------+-----------------------------------------------------------------------------------------+--------+-------+-------+
| b3304859-da3b-4df7-b73e-99577bc534cc | gluster-kube2-0.glusterd2.gcs | /var/run/glusterd2/bricks/pvc-1848109b-fc49-11e8-b05d-52540001ce00/subvol1/brick2/brick | true   |    -1 | 17112 |
| 8fdf6e4a-c828-4739-8ed5-f406982febd0 | gluster-kube1-0.glusterd2.gcs | /var/run/glusterd2/bricks/pvc-1848109b-fc49-11e8-b05d-52540001ce00/subvol1/brick3/brick | true   | 42632 | 17108 |
| 4930f015-5582-4338-a8da-8f954acd600f | gluster-kube3-0.glusterd2.gcs | /var/run/glusterd2/bricks/pvc-1848109b-fc49-11e8-b05d-52540001ce00/subvol1/brick1/brick | true   | 40835 |   299 |
+--------------------------------------+-------------------------------+-----------------------------------------------------------------------------------------+--------+-------+-------+
Volume : pvc-187c7db8-fc49-11e8-b05d-52540001ce00
+--------------------------------------+-------------------------------+-----------------------------------------------------------------------------------------+--------+-------+-------+
|               BRICK ID               |             HOST              |                                          PATH                                           | ONLINE | PORT  |  PID  |
+--------------------------------------+-------------------------------+-----------------------------------------------------------------------------------------+--------+-------+-------+
| 931e647a-a513-4701-821a-0ed07dc47a54 | gluster-kube3-0.glusterd2.gcs | /var/run/glusterd2/bricks/pvc-187c7db8-fc49-11e8-b05d-52540001ce00/subvol1/brick1/brick | true   | 44594 |   320 |
| b19614f0-4e4e-4bfe-b340-c8d7a0a04617 | gluster-kube2-0.glusterd2.gcs | /var/run/glusterd2/bricks/pvc-187c7db8-fc49-11e8-b05d-52540001ce00/subvol1/brick2/brick | true   |    -1 | 17091 |
| 41702753-14e2-43f5-9988-155337f046a5 | gluster-kube1-0.glusterd2.gcs | /var/run/glusterd2/bricks/pvc-187c7db8-fc49-11e8-b05d-52540001ce00/subvol1/brick3/brick | true   | 33413 | 17086 |
+--------------------------------------+-------------------------------+-----------------------------------------------------------------------------------------+--------+-------+-------+
Volume : pvc-18acb68a-fc49-11e8-b05d-52540001ce00
+--------------------------------------+-------------------------------+-----------------------------------------------------------------------------------------+--------+-------+-------+
|               BRICK ID               |             HOST              |                                          PATH                                           | ONLINE | PORT  |  PID  |
+--------------------------------------+-------------------------------+-----------------------------------------------------------------------------------------+--------+-------+-------+
| c968e010-2524-4ca8-a74e-3fd27e3820cc | gluster-kube3-0.glusterd2.gcs | /var/run/glusterd2/bricks/pvc-18acb68a-fc49-11e8-b05d-52540001ce00/subvol1/brick1/brick | true   | 37459 |   341 |
| 56630da9-2aa1-4cba-9992-059050cbb10d | gluster-kube2-0.glusterd2.gcs | /var/run/glusterd2/bricks/pvc-18acb68a-fc49-11e8-b05d-52540001ce00/subvol1/brick2/brick | true   |    -1 | 17313 |
| ad14a9ee-4930-48ec-a2f1-8b986b2d4926 | gluster-kube1-0.glusterd2.gcs | /var/run/glusterd2/bricks/pvc-18acb68a-fc49-11e8-b05d-52540001ce00/subvol1/brick3/brick | true   | 39676 | 17314 |
+--------------------------------------+-------------------------------+-----------------------------------------------------------------------------------------+--------+-------+-------+
Volume : pvc-18dae6f4-fc49-11e8-b05d-52540001ce00
+--------------------------------------+-------------------------------+-----------------------------------------------------------------------------------------+--------+-------+-------+
|               BRICK ID               |             HOST              |                                          PATH                                           | ONLINE | PORT  |  PID  |
+--------------------------------------+-------------------------------+-----------------------------------------------------------------------------------------+--------+-------+-------+
| c978f37b-82da-49cc-a77b-302ea65101f7 | gluster-kube3-0.glusterd2.gcs | /var/run/glusterd2/bricks/pvc-18dae6f4-fc49-11e8-b05d-52540001ce00/subvol1/brick1/brick | true   | 37105 |   362 |
| 6e4c0127-93a0-4de8-8f48-cf94647a39a5 | gluster-kube2-0.glusterd2.gcs | /var/run/glusterd2/bricks/pvc-18dae6f4-fc49-11e8-b05d-52540001ce00/subvol1/brick2/brick | true   |    -1 | 17437 |
| f617a6c4-17f9-4de8-a1cd-dc4b1bd027f9 | gluster-kube1-0.glusterd2.gcs | /var/run/glusterd2/bricks/pvc-18dae6f4-fc49-11e8-b05d-52540001ce00/subvol1/brick3/brick | true   | 43834 | 17430 |
+--------------------------------------+-------------------------------+-----------------------------------------------------------------------------------------+--------+-------+-------+
Volume : pvc-190c48a4-fc49-11e8-b05d-52540001ce00
+--------------------------------------+-------------------------------+-----------------------------------------------------------------------------------------+--------+-------+-------+
|               BRICK ID               |             HOST              |                                          PATH                                           | ONLINE | PORT  |  PID  |
+--------------------------------------+-------------------------------+-----------------------------------------------------------------------------------------+--------+-------+-------+
| c1c3d33d-d1f4-4abe-985c-baa30efde4c7 | gluster-kube3-0.glusterd2.gcs | /var/run/glusterd2/bricks/pvc-190c48a4-fc49-11e8-b05d-52540001ce00/subvol1/brick1/brick | true   | 38353 |   383 |
| 1e56c085-824e-4a45-bdc9-0363be67f7fb | gluster-kube2-0.glusterd2.gcs | /var/run/glusterd2/bricks/pvc-190c48a4-fc49-11e8-b05d-52540001ce00/subvol1/brick2/brick | true   |    -1 | 17376 |
| c1dcd6f6-6b6d-4e74-aea1-8f606a906d8e | gluster-kube1-0.glusterd2.gcs | /var/run/glusterd2/bricks/pvc-190c48a4-fc49-11e8-b05d-52540001ce00/subvol1/brick3/brick | true   | 45270 | 17370 |
+--------------------------------------+-------------------------------+-----------------------------------------------------------------------------------------+--------+-------+-------+
Volume : pvc-193bf5ba-fc49-11e8-b05d-52540001ce00
+--------------------------------------+-------------------------------+-----------------------------------------------------------------------------------------+--------+-------+-------+
|               BRICK ID               |             HOST              |                                          PATH                                           | ONLINE | PORT  |  PID  |
+--------------------------------------+-------------------------------+-----------------------------------------------------------------------------------------+--------+-------+-------+
| dd57adb4-3f84-4fb1-8688-0973dba5ad7e | gluster-kube3-0.glusterd2.gcs | /var/run/glusterd2/bricks/pvc-193bf5ba-fc49-11e8-b05d-52540001ce00/subvol1/brick1/brick | true   | 45959 |   404 |
| e6862c91-de9d-4f09-8bbc-08ef3eb9023f | gluster-kube2-0.glusterd2.gcs | /var/run/glusterd2/bricks/pvc-193bf5ba-fc49-11e8-b05d-52540001ce00/subvol1/brick2/brick | true   |    -1 | 17322 |
| b0a2cbc4-d4f3-491d-be7b-691385ff1948 | gluster-kube1-0.glusterd2.gcs | /var/run/glusterd2/bricks/pvc-193bf5ba-fc49-11e8-b05d-52540001ce00/subvol1/brick3/brick | true   | 44934 | 17308 |
+--------------------------------------+-------------------------------+-----------------------------------------------------------------------------------------+--------+-------+-------+
Volume : pvc-1970caaf-fc49-11e8-b05d-52540001ce00
+--------------------------------------+-------------------------------+-----------------------------------------------------------------------------------------+--------+-------+-------+
|               BRICK ID               |             HOST              |                                          PATH                                           | ONLINE | PORT  |  PID  |
+--------------------------------------+-------------------------------+-----------------------------------------------------------------------------------------+--------+-------+-------+
| a04bb2e3-642f-40fd-9995-33e1092eac9b | gluster-kube3-0.glusterd2.gcs | /var/run/glusterd2/bricks/pvc-1970caaf-fc49-11e8-b05d-52540001ce00/subvol1/brick1/brick | true   | 44037 |   425 |
| 1aebca4e-d48d-481a-a775-3bb104f468e3 | gluster-kube2-0.glusterd2.gcs | /var/run/glusterd2/bricks/pvc-1970caaf-fc49-11e8-b05d-52540001ce00/subvol1/brick2/brick | true   |    -1 | 17184 |
| e4e70f81-1352-4551-b52f-5133f0266d7c | gluster-kube1-0.glusterd2.gcs | /var/run/glusterd2/bricks/pvc-1970caaf-fc49-11e8-b05d-52540001ce00/subvol1/brick3/brick | true   | 41577 | 17181 |
+--------------------------------------+-------------------------------+-----------------------------------------------------------------------------------------+--------+-------+-------+
Volume : pvc-19a4470f-fc49-11e8-b05d-52540001ce00
+--------------------------------------+-------------------------------+-----------------------------------------------------------------------------------------+--------+-------+-------+
|               BRICK ID               |             HOST              |                                          PATH                                           | ONLINE | PORT  |  PID  |
+--------------------------------------+-------------------------------+-----------------------------------------------------------------------------------------+--------+-------+-------+
| c526866c-603b-4f94-8808-ce8e50e31ded | gluster-kube1-0.glusterd2.gcs | /var/run/glusterd2/bricks/pvc-19a4470f-fc49-11e8-b05d-52540001ce00/subvol1/brick3/brick | true   | 44531 | 17409 |
| 30e08328-a7c5-4238-ab87-4c3256b2a026 | gluster-kube3-0.glusterd2.gcs | /var/run/glusterd2/bricks/pvc-19a4470f-fc49-11e8-b05d-52540001ce00/subvol1/brick1/brick | true   | 42277 |   446 |
| 7ec45214-a215-4219-8163-f04f3f54453a | gluster-kube2-0.glusterd2.gcs | /var/run/glusterd2/bricks/pvc-19a4470f-fc49-11e8-b05d-52540001ce00/subvol1/brick2/brick | true   |    -1 | 17416 |
+--------------------------------------+-------------------------------+-----------------------------------------------------------------------------------------+--------+-------+-------+
Volume : pvc-19ed87e6-fc49-11e8-b05d-52540001ce00
+--------------------------------------+-------------------------------+-----------------------------------------------------------------------------------------+--------+-------+-------+
|               BRICK ID               |             HOST              |                                          PATH                                           | ONLINE | PORT  |  PID  |
+--------------------------------------+-------------------------------+-----------------------------------------------------------------------------------------+--------+-------+-------+
| 6f3eac2b-a128-4ada-9146-6b9feff5ecca | gluster-kube2-0.glusterd2.gcs | /var/run/glusterd2/bricks/pvc-19ed87e6-fc49-11e8-b05d-52540001ce00/subvol1/brick2/brick | true   |    -1 | 17207 |
| 17b40a4a-2327-4980-ac3d-6b8ae8cbd1e5 | gluster-kube1-0.glusterd2.gcs | /var/run/glusterd2/bricks/pvc-19ed87e6-fc49-11e8-b05d-52540001ce00/subvol1/brick3/brick | true   | 33163 | 17203 |
| 82c25be2-1c6d-49d9-850d-94534e84db9e | gluster-kube3-0.glusterd2.gcs | /var/run/glusterd2/bricks/pvc-19ed87e6-fc49-11e8-b05d-52540001ce00/subvol1/brick1/brick | true   | 38980 |   467 |
+--------------------------------------+-------------------------------+-----------------------------------------------------------------------------------------+--------+-------+-------+
Volume : pvc-1a26b05e-fc49-11e8-b05d-52540001ce00
+--------------------------------------+-------------------------------+-----------------------------------------------------------------------------------------+--------+-------+-------+
|               BRICK ID               |             HOST              |                                          PATH                                           | ONLINE | PORT  |  PID  |
+--------------------------------------+-------------------------------+-----------------------------------------------------------------------------------------+--------+-------+-------+
| f6aed7c4-c5cf-4aa4-a185-102a56c6efe7 | gluster-kube1-0.glusterd2.gcs | /var/run/glusterd2/bricks/pvc-1a26b05e-fc49-11e8-b05d-52540001ce00/subvol1/brick3/brick | true   | 44482 | 17349 |
| 6d8020ad-5dbe-4004-afe0-306a7c0564e7 | gluster-kube3-0.glusterd2.gcs | /var/run/glusterd2/bricks/pvc-1a26b05e-fc49-11e8-b05d-52540001ce00/subvol1/brick1/brick | true   | 46863 |   488 |
| 8fa28a3f-5537-40d0-b0cb-129878717784 | gluster-kube2-0.glusterd2.gcs | /var/run/glusterd2/bricks/pvc-1a26b05e-fc49-11e8-b05d-52540001ce00/subvol1/brick2/brick | true   |    -1 | 17355 |
+--------------------------------------+-------------------------------+-----------------------------------------------------------------------------------------+--------+-------+-------+
[root@gluster-kube2-0 /]#

the new logs are placed on the rhsqa-virt05.lab.eng.blr.redhat.com:/glusterd2_newport.log

ksandha avatar Dec 10 '18 09:12 ksandha

If you have the setup, can you please check if the pid 17355 is still running on gluster-kube2-0.glusterd2.gcs ? Also can you take the glusterd statedump and share the output?

atinmu avatar Dec 10 '18 11:12 atinmu

This issue is still seen on glusterd version: v6.0-dev.114.gitd51f60b.

One such output, Volume : pvc-a703978d-1d70-11e9-9b03-5254006fcc4e +--------------------------------------+-------------------------------+-----------------------------------------------------------------------------------------+--------+-------+------+ | BRICK ID | HOST | PATH | ONLINE | PORT | PID | +--------------------------------------+-------------------------------+-----------------------------------------------------------------------------------------+--------+-------+------+ | 2117315e-adb7-4b81-b583-d932283669d2 | gluster-kube3-0.glusterd2.gcs | /var/run/glusterd2/bricks/pvc-a703978d-1d70-11e9-9b03-5254006fcc4e/subvol1/brick1/brick | true | 41554 | 4091 | | 8f336ca3-c58c-4966-b99f-186b5a359408 | gluster-kube1-0.glusterd2.gcs | /var/run/glusterd2/bricks/pvc-a703978d-1d70-11e9-9b03-5254006fcc4e/subvol1/brick2/brick | true | -1 | 4668 | | 2096c96b-6e57-4273-a228-b389ad8442c6 | gluster-kube2-0.glusterd2.gcs | /var/run/glusterd2/bricks/pvc-a703978d-1d70-11e9-9b03-5254006fcc4e/subvol1/brick3/brick | true | 40314 | 4227 | +--------------------------------------+-------------------------------+-----------------------------------------------------------------------------------------+--------+-------+------+

Process "4668" is running on gluster-kube1-0.glusterd2.gcs

[root@gluster-kube1-0 /]# ps -ef | grep -i 4668 root 4668 1 0 11:36 ? 00:00:00 /usr/sbin/glusterfsd --volfile-server gluster-kube1-0.glusterd2.gcs --volfile-server-port 24007 --volfile-id pvc-a703978d-1d70-11e9-9b03-5254006fcc4e.0aea396e-4401-4bd7-9b40-66662b521112.var-run-glusterd2-bricks-pvc-a703978d-1d70-11e9-9b03-5254006fcc4e-subvol1-brick2-brick -p /var/run/glusterd2/0aea396e-4401-4bd7-9b40-66662b521112-var-run-glusterd2-bricks-pvc-a703978d-1d70-11e9-9b03-5254006fcc4e-subvol1-brick2-brick.pid -S /var/run/glusterd2/47b340d5c3c65469.socket --brick-name /var/run/glusterd2/bricks/pvc-a703978d-1d70-11e9-9b03-5254006fcc4e/subvol1/brick2/brick -l /var/log/glusterd2/glusterfs/bricks/var-run-glusterd2-bricks-pvc-a703978d-1d70-11e9-9b03-5254006fcc4e-subvol1-brick2-brick.log --xlator-option *-posix.glusterd-uuid=0aea396e-4401-4bd7-9b40-66662b521112

Attaching glusterd2 statedump statedump_new.txt

PrasadDesala avatar Jan 21 '19 12:01 PrasadDesala