veenadong

This run is microk8s v1.28.12, disconnected Node 1 network: ``` `core@glop-nm-115-mem2:~$ kubectl get nodes -o wide NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME glop-nm-115-mem1.glcpdev.cloud.hpe.com NotReady 70m v1.28.12...

Seems when agent-core crashed, the api-rest pod somehow holds on to the old connection and does not release it. Does increasing the number of api-rest pods or agent-core pods help...

We tried upgrading to 2.7.1, after 1 node failure, trying to get volumes/pools/nodes information: ``` core@sc-os-72-node1:~$ kubectl mayastor get volumes Failed to list volumes. Error error in request: request timed...

"2024-10-08T17:08:26.487185768" is around when we had to restart all the nodes hoping to resolve this issue. Scaled the api-rest to replicas=1 -- still getting timed out Restarted agent-core -- still...

Further observations after more testing, restarting the api-rest pod does work. However, after certain amount of time (40 -60 minutes), the pod hung (ie. cannot exec into the pod). If...

One of the node in this cluster of 3 nodes was rebooted.

We ran into another issue where the NexusSpec in etcd is missing. Missing entry is related to: ``` /openebs.io/mayastor/apis/v0/clusters/1013e263-e0ba-48b2-ae78-52d51b5da9c8/namespaces/mayastor/volume/4e01a5bc-d532-4186-a4ec-ab0b689a8e44/nexus/8f5d061c-82f1-46bb-8bd1-c4164573e1da/info ``` The related key: /openebs.io/mayastor/apis/v0/clusters/1013e263-e0ba-48b2-ae78-52d51b5da9c8/namespaces/mayastor/NexusSpec/8f5d061c-82f1-46bb-8bd1-c4164573e1da is not present, causing CSI not...

@tiagolobocastro The key that's missing is: ``` /openebs.io/mayastor/apis/v0/clusters/1013e263-e0ba-48b2-ae78-52d51b5da9c8/namespaces/mayastor/NexusSpec/8f5d061c-82f1-46bb-8bd1-c4164573e1da ``` The following is the command ran while in the error state: ``` core@sc-os-160-node2:~/mayastor-2024-08-29--17-47-43-UTC$ kubectl exec -n mayastor mayastor-etcd-0 -- etcdctl get...

> You can try to scale down the app trying to use the volume. When scaling down the volume should be "reset" to not expecting nexus to be present. Then...

> hmm no logs are included in the bundle @veenadong, which plugin version did you use to create the bundle? We are running 2.5.1 and the plugin version is: `Kubectl...