After redeploy csi-s3-test-nginx pod occur `Transport endpoint is not connected`

[dyrnq]

Steps to reproduce

1. deploy k8s-cis-s3
2. create sc

cat <<EOF | kubectl apply -f -
# Source: csi-s3/templates/storageclass.yaml
kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: csi-s3
provisioner: ru.yandex.s3.csi
parameters:
  mounter: "geesefs"
  options: "--memory-limit 1000 --dir-mode 0777 --file-mode 0666"
  csi.storage.k8s.io/provisioner-secret-name: csi-s3-secret
  csi.storage.k8s.io/provisioner-secret-namespace: kube-system
  csi.storage.k8s.io/controller-publish-secret-name: csi-s3-secret
  csi.storage.k8s.io/controller-publish-secret-namespace: kube-system
  csi.storage.k8s.io/node-stage-secret-name: csi-s3-secret
  csi.storage.k8s.io/node-stage-secret-namespace: kube-system
  csi.storage.k8s.io/node-publish-secret-name: csi-s3-secret
  csi.storage.k8s.io/node-publish-secret-namespace: kube-system
reclaimPolicy: Delete
EOF

3. deploy example pvc and pod

cat <<EOF | kubectl apply -f -
# https://github.com/yandex-cloud/k8s-csi-s3/blob/master/deploy/kubernetes/examples/pvc.yaml
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: csi-s3-pvc
  namespace: default
spec:
  accessModes:
  - ReadWriteMany
  resources:
    requests:
      storage: 5Gi
  storageClassName: csi-s3
# https://github.com/yandex-cloud/k8s-csi-s3/blob/master/deploy/kubernetes/examples/pod.yaml
---
apiVersion: v1
kind: Pod
metadata:
  name: csi-s3-test-nginx
  namespace: default
spec:
  containers:
   - name: csi-s3-test-nginx
     image: nginx
     volumeMounts:
       - mountPath: /usr/share/nginx/html/s3
         name: webroot
  volumes:
   - name: webroot
     persistentVolumeClaim:
       claimName: csi-s3-pvc
       readOnly: false
EOF

kubectl exec -it pod/csi-s3-test-nginx -- bash -c "echo hello >> /usr/share/nginx/html/s3/index.html"
kubectl exec -it pod/csi-s3-test-nginx -- bash -c "curl http://127.0.0.1/s3/index.html"

systemctl list-units --no-pager |grep gee
  geesefs-pvc_2d7451f1a3_2ddcd1_2d4c08_2db57b_2dc146a8b66105.service                                                                                                     loaded active running   GeeseFS mount for Kubernetes volume pvc-7451f1a3-dcd1-4c08-b57b-c146a8b66105

4. delete csi-s3-test-nginx pod

kubectl delete pod csi-s3-test-nginx

5. redo step3
6. occur Transport endpoint is not connected

kubectl exec -it pod/csi-s3-test-nginx -- bash -c "ls -l /usr/share/nginx/html/s3"
ls: cannot access '/usr/share/nginx/html/s3': Transport endpoint is not connected
command terminated with exit code 2

I'm not sure if this is a problem, Could you please help confirm this? TKS.

[dyrnq]

When this error occurs, the pod is Running but the out systemd unit like geesefs-pvc can't be found use systemctl list-units --no-pager |grep gee on that pod`s node.

[dyrnq]

root@debian:~# mount |grep gee
pvc-7451f1a3-dcd1-4c08-b57b-c146a8b66105 on /var/lib/kubelet/pods/2eb6faa5-c36e-46a8-bf79-c2d7e054ac52/volumes/kubernetes.io~csi/pvc-7451f1a3-dcd1-4c08-b57b-c146a8b66105/mount type fuse.geesefs (rw,nosuid,nodev,relatime,user_id=65534,group_id=0,default_permissions,allow_other)
root@debian:~# df /var/lib/kubelet/pods/2eb6faa5-c36e-46a8-bf79-c2d7e054ac52/volumes/kubernetes.io~csi/pvc-7451f1a3-dcd1-4c08-b57b-c146a8b66105/mount
df: /var/lib/kubelet/pods/2eb6faa5-c36e-46a8-bf79-c2d7e054ac52/volumes/kubernetes.io~csi/pvc-7451f1a3-dcd1-4c08-b57b-c146a8b66105/mount: Transport endpoint is not connected

[jidckii]

Same problem, if mount 1 pvc from many pods on many nodes.

[vitalif]

Hi, I'm not sure why you experience this bug, I checked it myself and didn't reproduce it I think logs from csi-s3 containers could help... please attach them? :-)

[dyrnq]

Hi, I'm not sure why you experience this bug, I checked it myself and didn't reproduce it I think logs from csi-s3 containers could help... please attach them? :-)

tks reply

This error does not occur just once. It needs to be executed repeatedly. It is accidental.

redo step3 ---> step4 may times, then the error occur

[dyrnq]

@vitalif this is my test case

test-csi-s3.sh.md

kubectl logs -l app=csi-s3 -c csi-s3 -n kube-system

csi-s3.log

[dyrnq]

Finally I found a solution, which can be solvedusing the following startupProbe and terminationGracePeriodSeconds: 2 for quick fail

     #livenessProbe:
     startupProbe:
       exec:
         command:
         - ls
         - /usr/share/nginx/html/s3
       initialDelaySeconds: 2
       periodSeconds: 2
       successThreshold: 1
       failureThreshold: 1
       terminationGracePeriodSeconds: 2