kubectl
kubectl copied to clipboard
kubernetes
kubectl exec & port forward TLS handshake error when https_proxy environment variable contains special characters
kubectl version
- Client Version: version.Info{Major:"1", Minor:"13", GitVersion:"v1.13.12", GitCommit:"a8b52209ee172232b6db7a6e0ce2adc77458829f", GitTreeState:"clean", BuildDate:"2019-10-15T12:12:15Z", GoVersion:"go1.11.13", Compiler:"gc", Platform:"windows/amd64"}
- Server Version: version.Info{Major:"1", Minor:"13", GitVersion:"v1.13.12", GitCommit:"524c3a1238422529d62f8e49506df658fa9c8b8c", GitTreeState:"clean", BuildDate:"2019-11-14T05:26:24Z", GoVersion:"go1.11.13", Compiler:"gc", Platform:"linux/amd64"}
Issue
When running the following behind a corporate proxy:
kubectl exec -it grafana-8665c55f6b-k7jpb ls -n monitoring
I'm seeing:
error: error sending request: Post https://k8s-fqdn:443/api/v1/namespaces/monitoring/pods/grafana-8665c55f6b-k7jpb/exec?command=ls&container=grafana&container=grafana&stdin=true&stdout=true&tty=true: tls: first record does not look like a TLS handshake
The same issue also happens with port forwarding e.g.:
kubectl -n monitoring port-forward prometheus-server-657c6d8f59-s7mjz 9090
error: error upgrading connection: error sending request: Post https://k8s-fqdn:443/api/v1/namespaces/monitoring/pods/prometheus-server-657c6d8f59-s7mjz/portforward: tls: first record does not look like a TLS handshake
How to reproduce
If the password for the https_proxy environment variable contains a hash symbol (URL encoded this is %23), it doesn't work and we get a TLS handshake error. If the password for the https_proxy environment variable contains a dollar symbol (URL encoded this is %24) it works fine.
The following does not work and gives the TLS handshake error above:
http_proxy=http://username:password%23@proxyhost:proxy-port https_proxy=http://username:password%23@proxyhost:proxy-port
The following does work:
http_proxy=http://username:password%24@proxyhost:proxy-port https_proxy=http://username:password%24@proxyhost:proxy-port
and we see:
kubectl exec -it grafana-8665c55f6b-k7jpb ls -n monitoring LICENSE README.md bin public tools NOTICE.md VERSION conf scripts
OR
kubectl -n monitoring port-forward prometheus-server-657c6d8f59-s7mjz 9090 Forwarding from 127.0.0.1:9090 -> 9090 Forwarding from [::1]:9090 -> 9090
as expected.
Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
If this issue is safe to close now please do so with /close.
Send feedback to sig-testing, kubernetes/test-infra and/or fejta. /lifecycle stale
Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.
If this issue is safe to close now please do so with /close.
Send feedback to sig-testing, kubernetes/test-infra and/or fejta. /lifecycle rotten
Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.
Send feedback to sig-testing, kubernetes/test-infra and/or fejta. /close
@fejta-bot: Closing this issue.
In response to this:
Rotten issues close after 30d of inactivity. Reopen the issue with
/reopen. Mark the issue as fresh with/remove-lifecycle rotten.Send feedback to sig-testing, kubernetes/test-infra and/or fejta. /close
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.
We recently experienced this error when the HTTP PROXY password contained ! in the string.
The kubectl version was : Client Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.0", GitCommit:"cb303e613a121a29364f75cc67d3d580833a7479", GitTreeState:"clean", BuildDate:"2021-04-08T16:31:21Z", GoVersion:"go1.16.1", Compiler:"gc", Platform:"darwin/amd64"}
The server version was: AWS EKS 1.19
@mjoshi89: You can't reopen an issue/PR unless you authored it or you are a collaborator.
In response to this:
/reopen
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.
/reopen
@mjoshi89 please tell more about the proxy you're using. Which one? Version? Anything else of note?
@eddiezane: Reopened this issue.
In response to this:
/reopen
@mjoshi89 please tell more about the proxy you're using. Which one? Version? Anything else of note?
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.
@mwalexander82: This issue is currently awaiting triage.
SIG CLI takes a lead on issue triage for this repo, but any Kubernetes member can accept issues by applying the triage/accepted label.
The triage/accepted label can be added by org members by writing /triage accepted in a comment.
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.
Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.
Send feedback to sig-contributor-experience at kubernetes/community. /close
@fejta-bot: Closing this issue.
In response to this:
Rotten issues close after 30d of inactivity. Reopen the issue with
/reopen. Mark the issue as fresh with/remove-lifecycle rotten.Send feedback to sig-contributor-experience at kubernetes/community. /close
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.
/reopen
kubectl version Client Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.0", GitCommit:"cb303e613a121a29364f75cc67d3d580833a7479", GitTreeState:"clean", BuildDate:"2021-04-08T16:31:21Z", GoVersion:"go1.16.1", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"19+", GitVersion:"v1.19.13-eks-8df270", GitCommit:"8df2700a72a2598fa3a67c05126fa158fd839620", GitTreeState:"clean", BuildDate:"2021-07-31T01:36:57Z", GoVersion:"go1.15.14", Compiler:"gc", Platform:"linux/amd64"} WARNING: version difference between client (1.21) and server (1.19) exceeds the supported minor version skew of +/-1
I have no idea what kind of corporate proxy I am behind
@dimitriosstander: You can't reopen an issue/PR unless you authored it or you are a collaborator.
In response to this:
/reopen
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.
@vrevelas: You can't reopen an issue/PR unless you authored it or you are a collaborator.
In response to this:
/reopen
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.
@eddiezane could you re-open please? a co-worker is experiencing this issue: They have an exclamation mark in their password in the proxy-url key of their kubeconfig, url encoded as %21 - the result when running 'kubectl exec' is "tls: first record does not look like a TLS handshake" with kubectl 1.24.3 on our corporate proxy, which is squid/3.5.27. It's worth noting that 'kubectl get' works fine, only 'kubectl exec' seems affected.
@dims: Reopened this issue.
In response to this:
/reopen
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.
+1, I'm facing the same issue
kubectl version
- Client Version: version.Info{Major:"1", Minor:"22", GitVersion:"v1.22.15", GitCommit:"1d79bc3bcccfba7466c44cc2055d6e7442e140ea", GitTreeState:"clean", BuildDate:"2022-09-21T12:18:10Z", GoVersion:"go1.16.15", Compiler:"gc", Platform:"windows/amd64"}
- Server Version: version.Info{Major:"1", Minor:"21+", GitVersion:"v1.21.14-eks-6d3986b", GitCommit:"8877a3e28d597e1184c15e4b5d543d5dc36b083b", GitTreeState:"clean", BuildDate:"2022-07-20T22:05:32Z", GoVersion:"go1.16.15", Compiler:"gc", Platform:"linux/amd64"}
The fix for this is now available in the latest patch versions of kubectl 1.23 and newer (specifically 1.23.16, 1.24.10, 1.25.6, 1.26.1, and 1.27+). Note at time of writing 1.24.10 hadn't been released yet but it should be out today, and 1.27 is a future release. Binaries are available at https://kubectl.docs.kubernetes.io/installation/kubectl/binaries/