seaweedfs-csi-driver icon indicating copy to clipboard operation
seaweedfs-csi-driver copied to clipboard

Pod in Terminating State won't Unmount

Open washcycle opened this issue 1 year ago • 1 comments

Mounted storage can get into a state where the weed mount isn't unmounted when the pod is terminating.

I don't have a reproducibe example right now, but wanted to document the issue still.

Work around is to go into the k8s node and kill the pvc mount there.

root@aspen7:/# ps -aux | grep pvc-cebb2309-cdaa-4edd-a81d-bc75e6b5b15d
root     1334193  0.0  0.0 774944 54492 ?        Sl   May08   0:21 weed -logtostderr=true mount -dirAutoCreate=true -umask=000 -dir=/var/lib/k0s/kubelet/plugins/kubernetes.io/csi/seaweedfs-csi-driver/a28177b3447eebd81ee4b8be4305
7212220e657c2f1e8f7cbc696edbc3d8fb4f/globalmount -localSocket=/tmp/seaweedfs-mount-1226006495.sock -cacheDir=/var/cache/seaweedfs/pvc-cebb2309-cdaa-4edd-a81d-bc75e6b5b15d -collectionQuotaMB=25600 -concurrentWriters=15 -collectio
n=pvc-cebb2309-cdaa-4edd-a81d-bc75e6b5b15d -filer=seaweedfs-filer-client.seaweedfs:8888 -cacheCapacityMB=1024 -filer.path=/buckets/pvc-cebb2309-cdaa-4edd-a81d-bc75e6b5b15d

sudo kill 1334193

The pod will then terminate.

I wonder if there is something we can add to the csi driver that checks for terminating pods and will kill the seaweedfs mount, i'm not super good with the Go k8s stack, but perhaps inspiration from some other projects that have solved this would help.

washcycle avatar May 09 '23 15:05 washcycle

Check whether NodeUnpublishVolume or NodeUnstageVolume is invoked in https://github.com/seaweedfs/seaweedfs-csi-driver/blob/master/pkg/driver/nodeserver.go

Maybe add some println.

chrislusf avatar May 09 '23 15:05 chrislusf