PhanLe1010

@rajivml Could you help us troubleshoot by providing the reproducing steps and env information (or provide us an env)? ## Environment - Longhorn version: - Installation method (e.g. Rancher Catalog...

> User "USER" cannot get resource "volumeattachments" in API group "storage.k8s.io" at the cluster scope This error is not related to this issue. It indicate that the client is missing...

@derekbit Is it correct the problem here is: the filesystem inside the pod become read-only due to many io errors (not from Longhorn layer)? Therefore, Longhorn doesn't notice the issue...

Thanks @olljanat > pod restart does not trigger volume unmount + remount on host After a quick reviewing the kubelet flow of unmount/mount volume, I am not sure if it...

@sharanbabumg In Longhorn v1.2.4, when Kuberlet restarts, Longhorn doesn't kill the instance-manager-xxx pods. This should in theory help your case. Please let us know if you have feedback after upgrading...

@sharanbabumg It is weird. Can you provide us: --- * Reproduce steps * Your env details ``` - Longhorn version: - Installation method (e.g. Rancher Catalog App/Helm/Kubectl): - Kubernetes distro...

Note: we might want to take a look at [this project](https://github.com/kubernetes-csi/livenessprobe) to allow kubelet to restart the longhorn csi plugin

We need more information about the volume to understand the current state of the volume. Can you past the output of: * `kubectl get volumes.longhorn.io pvc-3cc715b2-aaa2-4c1d-a788-ffc71905874c -o yaml -n longhorn-system`...

Can you manually try to salvage the volume by: 1. Make sure the node `node1 ` and the disk `/data/longhorn` are schedulable 2. Scale down the statefulset `lab-shihs` to 0...

Can you go to Longhorn UI -> click on node tap -> take a screenshot and post it here. The log message `Message: the disk default-disk-7f5c98b5a858e751(/data/longhorn) on the node node1...