kubernetes
rsync via kubectl hangs
What happened:
rsync with kubectl as remote shell gets stuck. After transfering a couple of files nothing happens. The sync was canceled after 20 minutes. When repeating the command it might get stuck after a different file. The files are mostly smaller than 500kb, some have 1 - 100MB. The command was run on one of the kubernetes master nodes. The vm of the master node and the vm where the pod is running are on the same vmware esx host. So it shouldn't be a problem caused by a network component. Encreasing the debug level in the kube-apiserver and the kubelet to 9 didn't reveal any problems. When the rsync command is cancel via ctrl+c, the rsync server command in the pod is not terminated. Sometimes the rsync finishes without any problems.
rsync command:
rsync -aOv --progress --blocking-io --rsh="kubectl exec my-pod-name -c my-container-name -n my-namespace -i -- " rsync:/mnt/content/ /mnt
What you expected to happen:
About 45.000 (8.4GB) should be transfered without problems.
How to reproduce it (as minimally and precisely as possible):
I didn't try, but it should be reproducable by creating lot's of random files (ca. 45.000) with a size of 500kb.
Anything else we need to know?:
The virus scanner was disabled.
Environment:
kubernetes version:
$ kubectl version
Client Version: v1.30.1+rke2r1
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.30.1+rke2r1
os: Red Hat Enterprise Linux 8.10 (Ootpa) on VMware vSphere 8
This issue is currently awaiting triage.
SIG CLI takes a lead on issue triage for this repo, but any Kubernetes member can accept issues by applying the triage/accepted label.
The triage/accepted label can be added by org members by writing /triage accepted in a comment.
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.
SPDY pings cause such problems, if the connection remains open longer (for reference https://github.com/kubernetes/kubectl/issues/1606#issuecomment-2149219184). I'd recommend testing the same command execution after websockets are in beta in next releases.
Sorry to ask this but we just enabled websockets as the default protocol for port forwarding, which should (I think) resolve this. It graduated to Beta in 1.31 (so it's on by default) could you try upgrading to that version or running this on that version? If not you can try enabling the alpha version of the feature on a 1.30 cluster, but you need access to be able to edit the flags provided to the api-server to enable it fully.
PortForwardWebsockets=true for the api-server flag and KUBECTL_REMOTE_COMMAND_WEBSOCKETS=true for the kubectl environment variable to enable the feature in 1.30.
Because of the newly beta feature and the question about whether it has helped:
/triage needs-information
/close Due to no follow up closing issue can reopen if there is more information provided.
@mpuckett159: Closing this issue.
In response to this:
/close Due to no follow up closing issue can reopen if there is more information provided.
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.