Device is not listing which was previously added after forming cluster

[Akarsha-rai]

Observed behavior

Device is not listing which was previously added after forming cluster

Expected/desired behavior

Device should be listed successfully

Details on how to reproduce (minimal and precise)

1. Have 3 nodes which are not in cluster.
2. Add a device on each nodes and list the device.

[root@dhcp35-121 ~]# glustercli device add e299e410-3e9d-45cb-8d14-0164d1d2b538 /dev/vdb
Device add successful

[root@dhcp35-121 ~]# curl -X GET http://10.70.35.121:24007/v1/devices | python -m json.tool
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   177  100   177    0     0  48760      0 --:--:-- --:--:-- --:--:-- 88500
[
    {
        "available-size": 20836352,
        "device-used": false,
        "extent-size": 4096,
        "name": "/dev/vdb",
        "peer-id": "e299e410-3e9d-45cb-8d14-0164d1d2b538",
        "state": "enabled",
        "vg-name": "vg-dev-vdb"
    }
]

[root@dhcp35-122 ~]# glustercli device add 5764c4ff-0624-48da-ad19-f1ab4b16e064 /dev/vdb
Device add successful

[root@dhcp35-122 ~]# curl -X GET http://10.70.35.122:24007/v1/devices | python -m json.tool
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   177  100   177    0     0  59435      0 --:--:-- --:--:-- --:--:-- 88500
[
    {
        "available-size": 20836352,
        "device-used": false,
        "extent-size": 4096,
        "name": "/dev/vdb",
        "peer-id": "5764c4ff-0624-48da-ad19-f1ab4b16e064",
        "state": "enabled",
        "vg-name": "vg-dev-vdb"
    }
]

[root@dhcp35-4 ~]# glustercli device add 6bf28bfa-8b4f-45ba-857d-9c3c021f43b0 /dev/vdb
Device add successful

[root@dhcp35-4 ~]# curl -X GET http://10.70.35.4:24007/v1/devices | python -m json.tool
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   177  100   177    0     0  62544      0 --:--:-- --:--:-- --:--:-- 88500
[
    {
        "available-size": 20836352,
        "device-used": false,
        "extent-size": 4096,
        "name": "/dev/vdb",
        "peer-id": "6bf28bfa-8b4f-45ba-857d-9c3c021f43b0",
        "state": "enabled",
        "vg-name": "vg-dev-vdb"
    }
]

3. Add a peers and form cluster.

[root@dhcp35-122 ~]# glustercli peer status
+--------------------------------------+-----------------------------------+--------------------+--------------------+--------+------+
|                  ID                  |               NAME                |  CLIENT ADDRESSES  |   PEER ADDRESSES   | ONLINE | PID  |
+--------------------------------------+-----------------------------------+--------------------+--------------------+--------+------+
| 3e3d7a74-f8a0-49dc-8ec4-e71fd1906dea | dhcp35-4.lab.eng.blr.redhat.com   | 127.0.0.1:24007    | 10.70.35.4:24008   | yes    | 1174 |
|                                      |                                   | 10.70.35.4:24007   |                    |        |      |
|                                      |                                   | 10.70.35.46:24007  |                    |        |      |
| 5764c4ff-0624-48da-ad19-f1ab4b16e064 | dhcp35-122.lab.eng.blr.redhat.com | 127.0.0.1:24007    | 10.70.35.122:24008 | yes    | 1138 |
|                                      |                                   | 10.70.35.122:24007 |                    |        |      |
| ecb44f85-1b8b-4f6e-8a96-1412d7721968 | dhcp35-121.lab.eng.blr.redhat.com | 127.0.0.1:24007    | 10.70.35.121:24008 | yes    | 1178 |
|                                      |                                   | 10.70.35.121:24007 |                    |        |      |
|                                      |                                   | 10.70.35.80:24007  |                    |        |      |
+--------------------------------------+-----------------------------------+--------------------+--------------------+--------+------+

4. List the devices, devices will be listed only on the node from which we performed peer add.

[root@dhcp35-122 ~]# curl -X GET http://10.70.35.122:24007/v1/devices | python -m json.tool
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   177  100   177    0     0  59078      0 --:--:-- --:--:-- --:--:-- 88500
[
    {
        "available-size": 20836352,
        "device-used": false,
        "extent-size": 4096,
        "name": "/dev/vdb",
        "peer-id": "5764c4ff-0624-48da-ad19-f1ab4b16e064",
        "state": "enabled",
        "vg-name": "vg-dev-vdb"
    }
]

[root@dhcp35-121 ~]# curl -X GET http://10.70.35.121:24007/v1/devices | python -m json.tool
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   177  100   177    0     0  48760      0 --:--:-- --:--:-- --:--:-- 88500
[
    {
        "available-size": 20836352,
        "device-used": false,
        "extent-size": 4096,
        "name": "/dev/vdb",
        "peer-id": "5764c4ff-0624-48da-ad19-f1ab4b16e064",
        "state": "enabled",
        "vg-name": "vg-dev-vdb"
    }
]

[root@dhcp35-4 ~]# curl -X GET http://10.70.35.4:24007/v1/devices | python -m json.tool
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   177  100   177    0     0  62544      0 --:--:-- --:--:-- --:--:-- 88500
[
    {
        "available-size": 20836352,
        "device-used": false,
        "extent-size": 4096,
        "name": "/dev/vdb",
        "peer-id": "5764c4ff-0624-48da-ad19-f1ab4b16e064",
        "state": "enabled",
        "vg-name": "vg-dev-vdb"
    }
]

Information about the environment:

- Glusterd2 version used (e.g. v4.1.0 or master): 
[root@dhcp35-121 ~]# glusterd2 --version
glusterd version: v6.0-dev.28.git1b19aeb
git SHA: 1b19aeb
go version: go1.9.4
go OS/arch: linux/amd64

- Operating system used: 
[root@dhcp35-229 ~]# cat /etc/centos-release
CentOS Linux release 7.5.1804 (Core)

Glusterd2 compiled from sources, as a package (rpm/deb), or container:
package

Using External ETCD: (yes/no, if yes ETCD version):
yes, etcdmain: etcd Version: 3.3.8

If container, which container image:

Using kubernetes, openshift, or direct install:
direct install

If kubernetes/openshift, is gluster running inside kubernetes/openshift or outside 


### Other useful information

- glusterd2 config files from all nodes (default /etc/glusterd2/glusterd2.toml)

[root@dhcp35-121 ~]# cat /etc/glusterd2/glusterd2.toml localstatedir = "/var/lib/glusterd2" logdir = "/var/log/glusterd2" logfile = "glusterd2.log" loglevel = "INFO" rundir = "/var/run/glusterd2" defaultpeerport = "24008" peeraddress = ":24008" clientaddress = ":24007" #restauth should be set to false to disable REST authentication in glusterd2 restauth = false etcdendpoints = "http://10.70.35.10:2379" noembed = true

[root@dhcp35-122 ~]# cat /etc/glusterd2/glusterd2.toml localstatedir = "/var/lib/glusterd2" logdir = "/var/log/glusterd2" logfile = "glusterd2.log" loglevel = "INFO" rundir = "/var/run/glusterd2" defaultpeerport = "24008" peeraddress = ":24008" clientaddress = ":24007" #restauth should be set to false to disable REST authentication in glusterd2 restauth = false etcdendpoints = "http://10.70.35.10:2379" noembed = true

[root@dhcp35-4 ~]# cat /etc/glusterd2/glusterd2.toml localstatedir = "/var/lib/glusterd2" logdir = "/var/log/glusterd2" logfile = "glusterd2.log" loglevel = "INFO" rundir = "/var/run/glusterd2" defaultpeerport = "24008" peeraddress = ":24008" clientaddress = ":24007" #restauth should be set to false to disable REST authentication in glusterd2 restauth = false etcdendpoints = "http://10.70.35.10:2379" noembed = true

[rishubhjain]

@Akarsha-rai Thanks for reporting this, this issue is because every time we add a peer to a cluster we assign a new PeerID and all the information like devices or volumes related to the old peerID is lost as we do not copy it to the new peer Info. We need to change the workflow a bit, instead of assigning new peerID we can use the same peerID or if there is a conflict then we can assign a new peerID and copy the details (it will take more than copy). This will work just for external etcd. Will discuss this with @aravindavk and @kshlm and update this issue.