[v0.24] vCluster connect background proxy is not port-forwarding to the localhost (WSL + Docker Desktop)
What happened?
Hi, i have a problem with vcluster v0.24.x that makes me unable to connect to the vcluster in my remote cluster. I can connect just fine in vcluster v0.23.x before.
Here's the details of my local environment and logs:
OS: Windows 11 running WSL2 (Ubuntu 22.04) Docker: v28.0.4 running using docker desktop
$ vcluster version
vcluster version 0.24.1
If I run the connect command, it's unable to connect to the vcluster in the remote cluster
$ vcluster connect vcluster-staging -n vcluster-staging --debug
11:34:49 debug Found vCluster vcluster-staging/vcluster-staging
11:34:49 debug Successfully found vCluster pod for connecting vcluster-staging-0
11:34:49 done vCluster is up and running
11:34:49 debug Successfully retrieved vCluster kube config
11:34:49 info Stopping background proxy...
11:34:50 info Starting background proxy container...
11:34:50 debug failed to list default service account Get "https://127.0.0.1:12228/api/v1/namespaces/default/serviceaccounts/default": dial tcp 127.0.0.1:12228: connect: connection refused
11:34:50 debug failed to list default service account Get "https://127.0.0.1:12228/api/v1/namespaces/default/serviceaccounts/default": dial tcp 127.0.0.1:12228: connect: connection refused
11:34:50 debug failed to list default service account Get "https://127.0.0.1:12228/api/v1/namespaces/default/serviceaccounts/default": dial tcp 127.0.0.1:12228: connect: connection refused
11:34:50 debug failed to list default service account Get "https://127.0.0.1:12228/api/v1/namespaces/default/serviceaccounts/default": dial tcp 127.0.0.1:12228: connect: connection refused
11:34:51 debug failed to list default service account Get "https://127.0.0.1:12228/api/v1/namespaces/default/serviceaccounts/default": dial tcp 127.0.0.1:12228: connect: connection refused
11:34:51 debug failed to list default service account Get "https://127.0.0.1:12228/api/v1/namespaces/default/serviceaccounts/default": dial tcp 127.0.0.1:12228: connect: connection refused
11:34:51 debug failed to list default service account Get "https://127.0.0.1:12228/api/v1/namespaces/default/serviceaccounts/default": dial tcp 127.0.0.1:12228: connect: connection refused
11:34:51 debug failed to list default service account Get "https://127.0.0.1:12228/api/v1/namespaces/default/serviceaccounts/default": dial tcp 127.0.0.1:12228: connect: connection refused
11:34:51 debug failed to list default service account Get "https://127.0.0.1:12228/api/v1/namespaces/default/serviceaccounts/default": dial tcp 127.0.0.1:12228: connect: connection refused
11:34:52 debug failed to list default service account Get "https://127.0.0.1:12228/api/v1/namespaces/default/serviceaccounts/default": dial tcp 127.0.0.1:12228: connect: connection refused
11:34:52 debug failed to list default service account Get "https://127.0.0.1:12228/api/v1/namespaces/default/serviceaccounts/default": dial tcp 127.0.0.1:12228: connect: connection refused
^
I noticed Starting background proxy container here, so I run docker ps and find that the port is not forwarded in my case
That's the reason that I'm unable to connect; the port has not been forwarded to my localhost. Again, this happened after I upgraded to version 0.24.x, it was working fine in v0.23.x
I also try to run using service account. It seems to run fine at first, but the same issue occurs when I'm running kubectl get ns
$ vcluster connect vcluster-staging -n vcluster-staging --service-account admin --cluster-role cluster-admin
11:57:15 done vCluster is up and running
11:57:16 info Stopping background proxy...
11:57:16 info Starting background proxy container...
11:57:16 info Create service account token for kube-system/admin
11:57:18 done Switched active kube context to vcluster_vcluster-staging_vcluster-staging_269504772373670133-cb5fb81a92dad4b638af7d734a46ee145
- Use `vcluster disconnect` to return to your previous kube context
- Use `kubectl get namespaces` to access the vcluster
$ kubectl get namespaces
E0429 11:57:57.857810 45897 memcache.go:265] "Unhandled Error" err="couldn't get current server API group list: Get \"https://127.0.0.1:12900/api?timeout=32s\": dial tcp 127.0.0.1:12900: connect: connection refused"
E0429 11:57:57.859682 45897 memcache.go:265] "Unhandled Error" err="couldn't get current server API group list: Get \"https://127.0.0.1:12900/api?timeout=32s\": dial tcp 127.0.0.1:12900: connect: connection refused"
E0429 11:57:57.863896 45897 memcache.go:265] "Unhandled Error" err="couldn't get current server API group list: Get \"https://127.0.0.1:12900/api?timeout=32s\": dial tcp 127.0.0.1:12900: connect: connection refused"
E0429 11:57:57.866098 45897 memcache.go:265] "Unhandled Error" err="couldn't get current server API group list: Get \"https://127.0.0.1:12900/api?timeout=32s\": dial tcp 127.0.0.1:12900: connect: connection refused"
E0429 11:57:57.867596 45897 memcache.go:265] "Unhandled Error" err="couldn't get current server API group list: Get \"https://127.0.0.1:12900/api?timeout=32s\": dial tcp 127.0.0.1:12900: connect: connection refused"
The connection to the server 127.0.0.1:12900 was refused - did you specify the right host or port?
Please let me know if you need more details. Thank you.
What did you expect to happen?
I can connect to vcluster using vcluster v0.24.x or above
How can we reproduce it (as minimally and precisely as possible)?
- Test in the same environment that I have (I mentioned it above)
- Install vCluster v0.24.x
- Try to connect to your vcluster using
vcluster connect vcluster-staging -n vcluster-staging --debugcommand
Anything else we need to know?
I'm using this workaround for now to be able to connect to my vcluster. But I hope that this issue can be fixed so I can connect only using vcluster cli just fine like before
Workaround:
- Manual port forward from vcluster pod
$ kubectl port-forward -n vcluster-staging pod/vcluster-staging-0 8443:8443
- Switch the context manually to your vcluster
$ kubectl config use-context vcluster_vcluster-staging_vcluster-staging_269504772373670133-cb5fb81a92dad4b638af7d734a46ee145
Switched to context "vcluster_vcluster-staging_vcluster-staging_269504772373670133-cb5fb81a92dad4b638af7d734a46ee145".
- Run
kubectl get namespaces
Host cluster Kubernetes version
$ kubectl version
Client Version: v1.32.3
Kustomize Version: v5.5.0
Server Version: v1.32.1-aliyun.1
vcluster version
$ vcluster --version
vcluster version 0.24.1
VCluster Config
controlPlane:
distro:
k8s:
enabled: true
backingStore:
database:
embedded:
enabled: true
statefulSet:
persistence:
volumeClaim:
size: 20Gi
storageClass: "alicloud-disk-topology-alltype"
I'm using this workaround for now to be able to connect to my vcluster. But I hope that this issue can be fixed so I can connect only using vcluster cli just fine like before
Workaround:
- Manual port forward from vcluster pod
$ kubectl port-forward -n vcluster-staging pod/vcluster-staging-0 8443:8443
- Switch the context manually to your vcluster
$ kubectl config use-context vcluster_vcluster-staging_vcluster-staging_269504772373670133-cb5fb81a92dad4b638af7d734a46ee145
Switched to context "vcluster_vcluster-staging_vcluster-staging_269504772373670133-cb5fb81a92dad4b638af7d734a46ee145".
- Run
kubectl get namespaces
Hi @wahyudibo I've got a PR https://github.com/loft-sh/vcluster/pull/2774 that might fix your issue, would you be able to build the CLI and check if it works for you?
to build the CLI you just need to run just build-cli-snapshot from the root of the repo
Hi @jjaferson, thank you for the response and PR. I have built the binaries just like you've suggested but unfortunately, the issue still persists.
$ vcluster version
vcluster version 0.26.0-next
Here's my vcluster connect command:
$ vcluster connect vcluster-staging -n vcluster-staging --service-account admin --cluster-role cluster-admin --local-port 8443 --address 0.0.0.0 --debug
12:31:04 debug VirtualClusterInstance resources are not available on the server.
12:31:04 debug Found vCluster vcluster-staging/vcluster-staging
12:31:05 debug Successfully found vCluster pod for connecting vcluster-staging-0
12:31:05 done vCluster is up and running
12:31:05 debug Successfully retrieved vCluster kube config
12:31:05 info Stopping background proxy...
12:31:05 info Starting background proxy container...
12:31:05 info Create service account token for kube-system/admin
12:31:10 done Switched active kube context to vcluster_vcluster-staging_vcluster-staging_269504772373670133-cb5fb81a92dad4b638af7d734a46ee145
- Use `vcluster disconnect` to return to your previous kube context
- Use `kubectl get namespaces` to access the vcluster
Then, i run kubectl get namespaces
$ kubectl get namespaces
E0521 12:32:16.546823 69560 memcache.go:265] "Unhandled Error" err="couldn't get current server API group list: Get \"https://127.0.0.1:8443/api?timeout=32s\": dial tcp 127.0.0.1:8443: connect: connection refused"
E0521 12:32:16.548302 69560 memcache.go:265] "Unhandled Error" err="couldn't get current server API group list: Get \"https://127.0.0.1:8443/api?timeout=32s\": dial tcp 127.0.0.1:8443: connect: connection refused"
E0521 12:32:16.549809 69560 memcache.go:265] "Unhandled Error" err="couldn't get current server API group list: Get \"https://127.0.0.1:8443/api?timeout=32s\": dial tcp 127.0.0.1:8443: connect: connection refused"
E0521 12:32:16.551303 69560 memcache.go:265] "Unhandled Error" err="couldn't get current server API group list: Get \"https://127.0.0.1:8443/api?timeout=32s\": dial tcp 127.0.0.1:8443: connect: connection refused"
E0521 12:32:16.552876 69560 memcache.go:265] "Unhandled Error" err="couldn't get current server API group list: Get \"https://127.0.0.1:8443/api?timeout=32s\": dial tcp 127.0.0.1:8443: connect: connection refused"
The connection to the server 127.0.0.1:8443 was refused - did you specify the right host or port?
here's the result when I inspected background proxy container
$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
b66014e58a79 bitnami/kubectl:1.33 "kubectl port-forwar..." 2 minutes ago Up 2 minutes vcluster_vcluster-staging_vcluster-staging_269504772373670133-cb5fb81a92dad4b638af7d734a46ee145_background_proxy
$ docker logs -f vcluster_vcluster-staging_vcluster-staging_269504772373670133-cb5fb81a92dad4b638af7d734a46ee145_background_proxy
Forwarding from 0.0.0.0:8443 -> 8443
As seen in the above logs, the PORTS section still empty. I'm expecting something like 0.0.0.0:8443->8443/tcp in that section that indicates the port-forwarding from container to host. Please let me know if you need more details from me.
Hey @wahyudibo ! How does the host kube-config look like, especially the clusters[*].server part for the cluster where the vCluster was created in? Maybe we don't correctly rewrite that.
Hi @jjaferson, thank you for the response and PR. I have built the binaries just like you've suggested but unfortunately, the issue still persists.
$ vcluster version vcluster version 0.26.0-nextHere's my vcluster connect command:
$ vcluster connect vcluster-staging -n vcluster-staging --service-account admin --cluster-role cluster-admin --local-port 8443 --address 0.0.0.0 --debug 12:31:04 debug VirtualClusterInstance resources are not available on the server. 12:31:04 debug Found vCluster vcluster-staging/vcluster-staging 12:31:05 debug Successfully found vCluster pod for connecting vcluster-staging-0 12:31:05 done vCluster is up and running 12:31:05 debug Successfully retrieved vCluster kube config 12:31:05 info Stopping background proxy... 12:31:05 info Starting background proxy container... 12:31:05 info Create service account token for kube-system/admin 12:31:10 done Switched active kube context to vcluster_vcluster-staging_vcluster-staging_269504772373670133-cb5fb81a92dad4b638af7d734a46ee145 - Use `vcluster disconnect` to return to your previous kube context - Use `kubectl get namespaces` to access the vclusterThen, i run
kubectl get namespaces$ kubectl get namespaces E0521 12:32:16.546823 69560 memcache.go:265] "Unhandled Error" err="couldn't get current server API group list: Get \"https://127.0.0.1:8443/api?timeout=32s\": dial tcp 127.0.0.1:8443: connect: connection refused" E0521 12:32:16.548302 69560 memcache.go:265] "Unhandled Error" err="couldn't get current server API group list: Get \"https://127.0.0.1:8443/api?timeout=32s\": dial tcp 127.0.0.1:8443: connect: connection refused" E0521 12:32:16.549809 69560 memcache.go:265] "Unhandled Error" err="couldn't get current server API group list: Get \"https://127.0.0.1:8443/api?timeout=32s\": dial tcp 127.0.0.1:8443: connect: connection refused" E0521 12:32:16.551303 69560 memcache.go:265] "Unhandled Error" err="couldn't get current server API group list: Get \"https://127.0.0.1:8443/api?timeout=32s\": dial tcp 127.0.0.1:8443: connect: connection refused" E0521 12:32:16.552876 69560 memcache.go:265] "Unhandled Error" err="couldn't get current server API group list: Get \"https://127.0.0.1:8443/api?timeout=32s\": dial tcp 127.0.0.1:8443: connect: connection refused" The connection to the server 127.0.0.1:8443 was refused - did you specify the right host or port?here's the result when I inspected background proxy container
$ docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES b66014e58a79 bitnami/kubectl:1.33 "kubectl port-forwar..." 2 minutes ago Up 2 minutes vcluster_vcluster-staging_vcluster-staging_269504772373670133-cb5fb81a92dad4b638af7d734a46ee145_background_proxy $ docker logs -f vcluster_vcluster-staging_vcluster-staging_269504772373670133-cb5fb81a92dad4b638af7d734a46ee145_background_proxy Forwarding from 0.0.0.0:8443 -> 8443As seen in the above logs, the PORTS section still empty. I'm expecting something like
0.0.0.0:8443->8443/tcpin that section that indicates the port-forwarding from container to host. Please let me know if you need more details from me.
it looks like the kubeconfig in the host isn't been rewritten correctly, it's forwarding traffic to 127.0.0.1:8443 as opposed to 0.0.0.0:8443. As Fabian mentioned can you check the address in host kubeconfig ?
Could you try v0.24.2-rc.3
Thank you for your reply @FabianKramm @jjaferson @cbron. I run connect command but this time I don't set the --local-port 8443
$ vcluster connect vcluster-staging -n vcluster-staging --service-account admin --cluster-role cluster-admin
12:24:54 done vCluster is up and running
12:24:54 info Stopping background proxy...
12:24:54 info Starting background proxy container...
12:24:55 info Create service account token for kube-system/admin
12:25:05 done Switched active kube context to vcluster_vcluster-staging_vcluster-staging_269504772373670133-cb5fb81a92dad4b638af7d734a46ee145
- Use `vcluster disconnect` to return to your previous kube context
- Use `kubectl get namespaces` to access the vcluster
$ kubectl get namespaces
E0527 12:25:10.592707 26940 memcache.go:265] "Unhandled Error" err="couldn't get current server API group list: Get \"https://127.0.0.1:10615/api?timeout=32s\": dial tcp 127.0.0.1:10615: connect: connection refused"
E0527 12:25:10.594279 26940 memcache.go:265] "Unhandled Error" err="couldn't get current server API group list: Get \"https://127.0.0.1:10615/api?timeout=32s\": dial tcp 127.0.0.1:10615: connect: connection refused"
E0527 12:25:10.595796 26940 memcache.go:265] "Unhandled Error" err="couldn't get current server API group list: Get \"https://127.0.0.1:10615/api?timeout=32s\": dial tcp 127.0.0.1:10615: connect: connection refused"
E0527 12:25:10.597678 26940 memcache.go:265] "Unhandled Error" err="couldn't get current server API group list: Get \"https://127.0.0.1:10615/api?timeout=32s\": dial tcp 127.0.0.1:10615: connect: connection refused"
E0527 12:25:10.599341 26940 memcache.go:265] "Unhandled Error" err="couldn't get current server API group list: Get \"https://127.0.0.1:10615/api?timeout=32s\": dial tcp 127.0.0.1:10615: connect: connection refused"
The connection to the server 127.0.0.1:10615 was refused - did you specify the right host or port?
$ docker logs -f vcluster_vcluster-staging_vcluster-staging_269504772373670133-cb5fb81a92dad4b638af7d734a46ee145_background_proxy
Forwarding from 0.0.0.0:10615 -> 8443
Here's my kubeconfig after the command
- cluster:
certificate-authority-data: [REDACTED]
server: https://127.0.0.1:10615
name: vcluster_vcluster-staging_vcluster-staging_269504772373670133-cb5fb81a92dad4b638af7d734a46ee145
Could you try v0.24.2-rc.3
I have tried and the result still the same @cbron
$ ~/vcluster-linux-amd64 connect vcluster-staging -n vcluster-staging --service-account admin --cluster-role cluster-admin
12:31:52 warn There is a newer version of vcluster: v0.25.0. Run `vcluster upgrade` to upgrade to the newest version.
12:31:53 done vCluster is up and running
12:31:53 info Stopping background proxy...
12:31:53 info Starting background proxy container...
12:31:53 info Create service account token for kube-system/admin
12:32:07 done Switched active kube context to vcluster_vcluster-staging_vcluster-staging_269504772373670133-cb5fb81a92dad4b638af7d734a46ee145
- Use `vcluster disconnect` to return to your previous kube context
- Use `kubectl get namespaces` to access the vcluster
$ kubectl get namespaces
E0527 12:32:12.655880 39826 memcache.go:265] "Unhandled Error" err="couldn't get current server API group list: Get \"https://127.0.0.1:11042/api?timeout=32s\": dial tcp 127.0.0.1:11042: connect: connection refused"
E0527 12:32:12.657442 39826 memcache.go:265] "Unhandled Error" err="couldn't get current server API group list: Get \"https://127.0.0.1:11042/api?timeout=32s\": dial tcp 127.0.0.1:11042: connect: connection refused"
E0527 12:32:12.659098 39826 memcache.go:265] "Unhandled Error" err="couldn't get current server API group list: Get \"https://127.0.0.1:11042/api?timeout=32s\": dial tcp 127.0.0.1:11042: connect: connection refused"
E0527 12:32:12.661126 39826 memcache.go:265] "Unhandled Error" err="couldn't get current server API group list: Get \"https://127.0.0.1:11042/api?timeout=32s\": dial tcp 127.0.0.1:11042: connect: connection refused"
E0527 12:32:12.663753 39826 memcache.go:265] "Unhandled Error" err="couldn't get current server API group list: Get \"https://127.0.0.1:11042/api?timeout=32s\": dial tcp 127.0.0.1:11042: connect: connection refused"
The connection to the server 127.0.0.1:11042 was refused - did you specify the right host or port?
$ ~/vcluster-linux-amd64 --version
vcluster version 0.24.2-rc.3
Thank you for your reply @FabianKramm @jjaferson @cbron. I run connect command but this time I don't set the
--local-port 8443$ vcluster connect vcluster-staging -n vcluster-staging --service-account admin --cluster-role cluster-admin 12:24:54 done vCluster is up and running 12:24:54 info Stopping background proxy... 12:24:54 info Starting background proxy container... 12:24:55 info Create service account token for kube-system/admin 12:25:05 done Switched active kube context to vcluster_vcluster-staging_vcluster-staging_269504772373670133-cb5fb81a92dad4b638af7d734a46ee145 - Use `vcluster disconnect` to return to your previous kube context - Use `kubectl get namespaces` to access the vcluster $ kubectl get namespaces E0527 12:25:10.592707 26940 memcache.go:265] "Unhandled Error" err="couldn't get current server API group list: Get \"https://127.0.0.1:10615/api?timeout=32s\": dial tcp 127.0.0.1:10615: connect: connection refused" E0527 12:25:10.594279 26940 memcache.go:265] "Unhandled Error" err="couldn't get current server API group list: Get \"https://127.0.0.1:10615/api?timeout=32s\": dial tcp 127.0.0.1:10615: connect: connection refused" E0527 12:25:10.595796 26940 memcache.go:265] "Unhandled Error" err="couldn't get current server API group list: Get \"https://127.0.0.1:10615/api?timeout=32s\": dial tcp 127.0.0.1:10615: connect: connection refused" E0527 12:25:10.597678 26940 memcache.go:265] "Unhandled Error" err="couldn't get current server API group list: Get \"https://127.0.0.1:10615/api?timeout=32s\": dial tcp 127.0.0.1:10615: connect: connection refused" E0527 12:25:10.599341 26940 memcache.go:265] "Unhandled Error" err="couldn't get current server API group list: Get \"https://127.0.0.1:10615/api?timeout=32s\": dial tcp 127.0.0.1:10615: connect: connection refused" The connection to the server 127.0.0.1:10615 was refused - did you specify the right host or port? $ docker logs -f vcluster_vcluster-staging_vcluster-staging_269504772373670133-cb5fb81a92dad4b638af7d734a46ee145_background_proxy Forwarding from 0.0.0.0:10615 -> 8443Here's my kubeconfig after the command
- cluster: certificate-authority-data: [REDACTED] server: https://127.0.0.1:10615 name: vcluster_vcluster-staging_vcluster-staging_269504772373670133-cb5fb81a92dad4b638af7d734a46ee145Could you try v0.24.2-rc.3
I have tried and the result still the same @cbron
$ ~/vcluster-linux-amd64 connect vcluster-staging -n vcluster-staging --service-account admin --cluster-role cluster-admin 12:31:52 warn There is a newer version of vcluster: v0.25.0. Run `vcluster upgrade` to upgrade to the newest version. 12:31:53 done vCluster is up and running 12:31:53 info Stopping background proxy... 12:31:53 info Starting background proxy container... 12:31:53 info Create service account token for kube-system/admin 12:32:07 done Switched active kube context to vcluster_vcluster-staging_vcluster-staging_269504772373670133-cb5fb81a92dad4b638af7d734a46ee145 - Use `vcluster disconnect` to return to your previous kube context - Use `kubectl get namespaces` to access the vcluster $ kubectl get namespaces E0527 12:32:12.655880 39826 memcache.go:265] "Unhandled Error" err="couldn't get current server API group list: Get \"https://127.0.0.1:11042/api?timeout=32s\": dial tcp 127.0.0.1:11042: connect: connection refused" E0527 12:32:12.657442 39826 memcache.go:265] "Unhandled Error" err="couldn't get current server API group list: Get \"https://127.0.0.1:11042/api?timeout=32s\": dial tcp 127.0.0.1:11042: connect: connection refused" E0527 12:32:12.659098 39826 memcache.go:265] "Unhandled Error" err="couldn't get current server API group list: Get \"https://127.0.0.1:11042/api?timeout=32s\": dial tcp 127.0.0.1:11042: connect: connection refused" E0527 12:32:12.661126 39826 memcache.go:265] "Unhandled Error" err="couldn't get current server API group list: Get \"https://127.0.0.1:11042/api?timeout=32s\": dial tcp 127.0.0.1:11042: connect: connection refused" E0527 12:32:12.663753 39826 memcache.go:265] "Unhandled Error" err="couldn't get current server API group list: Get \"https://127.0.0.1:11042/api?timeout=32s\": dial tcp 127.0.0.1:11042: connect: connection refused" The connection to the server 127.0.0.1:11042 was refused - did you specify the right host or port? $ ~/vcluster-linux-amd64 --version vcluster version 0.24.2-rc.3
Are you using KinD? Can you try to change your clusters[*].server to point to host.docker.internal and see if it works?
Just curious - if the background proxy feature is not yet stable for all use cases, why not make --background-proxy=false the default (since it continues to work for everyone)?
@jeremyj563 without proxy we cannot connect to vclusters hosted on Cloud Providers on remote machines, when the clusters don't have public dns
@jeremyj563 without proxy we cannot connect to vclusters hosted on Cloud Providers on remote machines, when the clusters don't have public dns
But... You just use --background-proxy=true then for that corner case!