for-win icon indicating copy to clipboard operation
for-win copied to clipboard

Cannot connect from WSL2 to Docker Desktop Kubernetes via kubectl

Open RemekGdansk opened this issue 2 years ago • 30 comments

  • [x] I have tried with the latest version of Docker Desktop
  • [x] I have tried disabling enabled experimental features
  • [x] I have uploaded Diagnostics
  • Diagnostics ID: D7D6AD08-4AEE-4ED1-804A-460EC61D577A/20230414085504

Actual behavior

I cannot connect from WSL 2 (Ubuntu) to Docker Desktop Kubernnetes via kubectl. I can connect via kubectl from PowerShell.

Expected behavior

I should be able to connect via kubectl from WSL2 as well as from PowerShell.

Information

  • Is it reproducible? Yes

  • Is the problem new? Yes, it worked before

  • Did the problem appear with an update? Possibly, since it worked before

  • Windows Version: Windows 11 Home 22H2 22621.1555

  • Docker Desktop Version: 4.18.0

  • WSL2 or Hyper-V backend? WSL2

  • Are you running inside a virtualized Windows e.g. on a cloud server or a VM: not a virtualized Windows

Output of & "C:\Program Files\Docker\Docker\resources\com.docker.diagnose.exe" check

I have provided the Diagnostics ID, the information should be in that report.

Steps to reproduce the behavior

Docker & Kubernetes on Docker Desktop are working correctly. The problem is to connect via kubectl from WSL2. Example:

$ kubectl cluster-info

To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.
The connection to the server kubernetes.docker.internal:6443 was refused - did you specify the right host or port?

The same command issued from PowerShell works:

PS kubectl cluster-info
Kubernetes control plane is running at https://kubernetes.docker.internal:6443
CoreDNS is running at https://kubernetes.docker.internal:6443/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy

To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.

When I uninstall Docker for Windows correctly, remove all .cache folders & kubeconfigs, after fresh install the connection is working. However, after the computer restart, the problem reappears.

Edit: I am providing full information on the versions of WSL, Docker and Kubernetes (mine are the same as those provided by @BarclayHill in the comments below):

PS C:\> wsl --version
WSL version: 1.2.0.0
Kernel version: 5.15.90.1
WSLg version: 1.0.51
MSRDC version: 1.2.3770
Direct3D version: 1.608.2-61064218
DXCore version: 10.0.25131.1002-220531-1700.rs-onecore-base2-hyp
Windows version: 10.0.22621.1555

PS C:\> docker --version
Docker version 20.10.24, build 297e128

PS C:\> kubectl version --short
Client Version: v1.25.4
Kustomize Version: v4.5.7
Server Version: v1.25.4

RemekGdansk avatar Apr 14 '23 08:04 RemekGdansk

I have the same problem. Initially, I found this: https://github.com/docker/for-win/issues/13271. The .wlsconfig "fix" didn't work for me, and it looks like the OP upgraded. I'm fully up to date on the latest stable windows and WSL2.

edit I ran upstairs to test on my desktop that is running Insider beta, and it works fine.

brettcurtis avatar Apr 15 '23 23:04 brettcurtis

Ran into this same issue 2 days ago and could not figure out but found this thread and followed it.

Today I tried connecting to to local K8s running on docker from WSL2 Ubuntu via kubectl and found it be working which was surprising.

Nothing changed other than a few reboots and caching my MSFT Acct Creds locally to resolve RDP issue from macOS.

BarclayHill avatar Apr 16 '23 19:04 BarclayHill

Same issue here.

Rebooting my machine did not help.

Uninstalling, deleting ~/.docker + ~/.kube, then reinstalling also did not help.

@RemekGdansk do you happen to remember the list of files/folders that you needed to delete before reinstalling?

fedenusy avatar Apr 19 '23 13:04 fedenusy

Got the same issue on a fresh install on Windows 11 with wsl2. It's working fine from the host (VS Code Kubernetes Addon, Powershell) but fails to connect from the debian distribution running on wsl2. Docker itself works as expected.

lordpayder avatar Apr 19 '23 13:04 lordpayder

I experienced this issue twice within this week, both times after rebooting Windows. Only re-installing Docker for Desktop and re-installing WSL solved the issue for me.

SoerenHenning avatar Apr 19 '23 16:04 SoerenHenning

Haven't hit the issue recently since last week. Applicable versions of what I'm running to help others.

docker --version
Docker version 20.10.24, build 297e128

wsl --version
WSL version: 1.2.0.0
Kernel version: 5.15.90.1
WSLg version: 1.0.51
MSRDC version: 1.2.3770
Direct3D version: 1.608.2-61064218
DXCore version: 10.0.25131.1002-220531-1700.rs-onecore-base2-hyp
Windows version: 10.0.22621.1555

kubectl version --short
Flag --short has been deprecated, and will be removed in the future. The --short output will become the default.
Client Version: v1.25.4
Kustomize Version: v4.5.7
Server Version: v1.25.4

BarclayHill avatar Apr 19 '23 16:04 BarclayHill

I'm getting exactly the same responses as @BarclayHill, both on Win11 and WSL Ubuntu, except for kubectl in WSL:

$ kubectl version --short
Flag --short has been deprecated, and will be removed in the future. The --short output will become the default.
Client Version: v1.25.4
Kustomize Version: v4.5.7
The connection to the server kubernetes.docker.internal:6443 was refused - did you specify the right host or port?

To summarize:

Docker Kubernetes
Windows 11
WSL Ubuntu

I once again tried uninstalling Docker Desktop, deleting .docker, AppData\Local\Docker and AppData\Roaming\Docker from the user directory and re-installing Docker Desktop + its Kubernetes, but the issue remains.

SoerenHenning avatar Apr 19 '23 17:04 SoerenHenning

Same issue, this started happening after I've updated the wsl using wsl.exe --update

kubectl get all
The connection to the server kubernetes.docker.internal:6443 was refused - did you specify the right host or port?

8o8inCodes avatar Apr 19 '23 18:04 8o8inCodes

Same issue here.

Rebooting my machine did not help.

Uninstalling, deleting ~/.docker + ~/.kube, then reinstalling also did not help.

@RemekGdansk do you happen to remember the list of files/folders that you needed to delete before reinstalling?

Hi @fedenusy

Not that I remember exactly, but most probably removed the following (my usual list when I need to do a hard-reset on Docker):

Windows

  • C:\Users\<User>\.docker
  • C:\Users\<User>\.kube
  • C:\Users\<User>\AppData\Local\Docker
  • C:\Users\<User>\AppData\Roaming\Docker
  • C:\Users\<User>\AppData\Roaming\Docker Desktop

WSL

  • \\wsl.localhost\Ubuntu\home\<user>\.docker
  • \\wsl.localhost\Ubuntu\home\<user>\.kube

RemekGdansk avatar Apr 19 '23 18:04 RemekGdansk

The last post reminded me of something I did originally when I first encountered the issue. Not sure this will help others or explain why my setup is working.

Stop Docker Desktop

$ cd /home/<user>/.kube
~/.kube$ cp config config.back
~/.kube$ cp /mnt/c/users/<user>/.kube/config config

Start Docker Desktop

~/.kube$ kubectl get all
NAME                 TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)   AGE
service/kubernetes   ClusterIP   10.96.0.1    <none>        443/TCP   5d2h

BarclayHill avatar Apr 19 '23 20:04 BarclayHill

Thank you @BarclayHill for your comment. The solution did not work for me, however.

The only difference I had between the config in Windows and the config in WSL were in properties:

  • users.user.client-certificate-data
  • users.user.client-key-data

Do you still have your old config from WSL (the one renamed to config.back)? Diff between the not working and working config could provide some insight.

RemekGdansk avatar Apr 19 '23 20:04 RemekGdansk

@RemekGdansk - No, sorry, I didn't keep it around or compare them. Like I said in the original post. I could not resolve the issue even copying the config from the windows side. It was few days later when I noticed it was working as expected. What changed between was only the reboots and running the following command to resolve an unrelated RDP issue.

PS> runas /u:MicrosoftAccount\<msftaccount> winver

BarclayHill avatar Apr 19 '23 20:04 BarclayHill

I uninstalled & reinstalled Docker Desktop in order to compare changes between kube configs and this time I could not connect from WSL using kubectl after reinstallation. So it seems that the previous success with uninstall/reinstall was rather random and not a reproducible way to prevent the problem even for a short time.

RemekGdansk avatar Apr 19 '23 21:04 RemekGdansk

I just tried a factory reset of Windows 11 (that's how badly I needed to get WSL k8s working...)

...and after reinstalling everything from scratch, WSL k8s worked!

Now going to buy a UPS-- not taking any chances of a reboot.

fedenusy avatar Apr 19 '23 21:04 fedenusy

I just tried a factory reset of Windows 11 (that's how badly I needed to get WSL k8s working...)

...and after reinstalling everything from scratch, WSL k8s worked!

Now going to buy a UPS-- not taking any chances of a reboot.

lol, reconfiguring everything is gonna take ages for me.. we need to find a solution without a full reset

8o8inCodes avatar Apr 20 '23 12:04 8o8inCodes

I'm curious, are all the folks who are struggling on stable windows release vs. an insider build? It works fine on my insider build but not stable.

brettcurtis avatar Apr 20 '23 12:04 brettcurtis

I'm curious, are all the folks who are struggling on stable windows release vs. an insider build? It works fine on my insider build but not stable.

stable

fedenusy avatar Apr 20 '23 12:04 fedenusy

I have the same issue, it suddenly stopped working. Could solve it by adding following lines in .wslconfig

[wsl2]
localhostForwarding=false

But if I do so and do a port forwarding within wsl against my local Kubernetes cluster in Docker Desktop, I can't reach the port anymore from my host outside from wsl.

tuncaytunc-zf avatar Apr 20 '23 19:04 tuncaytunc-zf

Thanks @tuncaytunc-zf , it worked on my side as well!

To all readers of this thread - .wslconfig file is located in your Windows user directory. If you don't have the file already, you need to create it.

I am leaving the thread open, it is only a workaround (unless it is going to be the official Docker Desktop instruction to do so).

RemekGdansk avatar Apr 20 '23 20:04 RemekGdansk

I had this issue last week. IIRC I went only far as uninstalling docker desktop and deleting C:\Users\<me>\.docker\ and restarting before reinstalling docker desktop.

The issue for me really seemed related to localhostForwarding (except I think mine works when true, lol). I'd recommend people check if they have a port 6443 open in the docker-desktop WSL backend PS> wsl -d docker-desktop -- netstat -tulpn. If that's there, but not in your default WSL distrobution (PS> wsl -- netstat -tulpn) then something is up with port forwarding.

zevisert avatar Apr 20 '23 23:04 zevisert

I have the same issue, it suddenly stopped working. Could solve it by adding following lines in .wslconfig

[wsl2]
localhostForwarding=false

But if I do so and do a port forwarding within wsl against my local Kubernetes cluster in Docker Desktop, I can't reach the port anymore from my host outside from wsl.

Ok that works, however that blocks the access to my servers that are running under wsl..

8o8inCodes avatar Apr 21 '23 07:04 8o8inCodes

I can also replicate this issue on version 22H2, build 22621.1555.

ivanvmoreno avatar Apr 21 '23 12:04 ivanvmoreno

Windows 11 Docker/K8s and WSL Transient Issue Notes

My machine bug checked last night and this morning this issue returned.

Here is how I worked around the issue and got it going again.

It's nonsense like this thats defeating my interest of working in a Windows enviroment again.


 $ kubectl get all
E0421 07:52:45.150246     237 memcache.go:265] couldnt get current server API group list: Get "https://kubernetes.docker.internal:6443/api?timeout=32s": dial tcp 127.0.0.1:6443: connect: connection refused

WTF!!!!

$ exit
PS > wsl --list -v
  NAME                   STATE           VERSION
* Ubuntu                 Running         2
  docker-desktop         Running         2
  docker-desktop-data    Running         2
PS > wsl --shutdown
PS > wsl --list -v
  NAME                   STATE           VERSION
* Ubuntu                 Running         2
  docker-desktop         Stopped         2
  docker-desktop-data    Stopped         2
PS > wsl --shutdown ubuntu
PS > wsl --list -v
  NAME                   STATE           VERSION
* Ubuntu                 Stopped         2
  docker-desktop         Stopped         2
  docker-desktop-data    Stopped         2

Think for a second and regret messing with Windows11. Wait, I bet this has something to do with something running post bug check that I shutdown previously trying to get this resolved.

Sort through event log looking at events occurriung after bug check. Aha! Hyper-V has started after bug check doing something with a default network switch called "WSL". Make note to uninstall Hyper-V nonsense for later.

Look in running processes and find orphaned docker desktop client and kill it. Shutdown Hyper-V Service...Start Docker Desktop. Jump back to shell.


PS > wsl --list -v
  NAME                   STATE           VERSION
* Ubuntu                 Running         2
  docker-desktop         Running         2
  docker-desktop-data    Running         2
PS > kubectl get all
NAME                 TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)   AGE
service/kubernetes   ClusterIP   10.96.0.1    <none>        443/TCP   6d20h
PS > wsl
$ kubectl get all
NAME                 TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)   AGE
service/kubernetes   ClusterIP   10.96.0.1    <none>        443/TCP   6d20h

Shut down RDP and go back to macOS to get some actual work done.

BarclayHill avatar Apr 21 '23 15:04 BarclayHill

Ran into this issue with all matched version from the issue.

The .wlsconfig "fix" didn't work for me either and currently haven't found a workaround for this.

robertwenquan avatar Apr 23 '23 16:04 robertwenquan

Microsoft should make their Docker Desktop. Issues like this and issues like losing all of my settings whenever Code or DD updates is too much.

wixregiga avatar Apr 25 '23 16:04 wixregiga

I have the same issue, it suddenly stopped working. Could solve it by adding following lines in .wslconfig

[wsl2]
localhostForwarding=false

But if I do so and do a port forwarding within wsl against my local Kubernetes cluster in Docker Desktop, I can't reach the port anymore from my host outside from wsl.

This is the only solution that has consistently worked for me - however, I need the localhost forwarding! Are there any workarounds that will allow me access to servers running on wsl with this turned off?

nagaem avatar Apr 26 '23 18:04 nagaem

I have the same issue, it suddenly stopped working. Could solve it by adding following lines in .wslconfig

[wsl2]
localhostForwarding=false

But if I do so and do a port forwarding within wsl against my local Kubernetes cluster in Docker Desktop, I can't reach the port anymore from my host outside from wsl.

This is the only solution that has consistently worked for me - however, I need the localhost forwarding! Are there any workarounds that will allow me access to servers running on wsl with this turned off?

Yes, there is a workaround for that. You can do the port forwarding from your host (not in wsl), then you should be able to access the ports from your host. Of course you need the kubectl installed on your host, too.

tuncaytunc-zf avatar Apr 27 '23 10:04 tuncaytunc-zf

Today's windows update seemed to fix this for me: image

brettcurtis avatar Apr 27 '23 14:04 brettcurtis

Adding kubernetes.docker.internal host in /etc/hosts seems to temporarily fix it for me

127.0.0.1     localhost kubernetes.docker.internal

sevos avatar Jul 23 '24 08:07 sevos

this issue is haunting me too. Any resolution please? above /etc/hosts fix doesnt work for me :( No point in trying other year old comments.

Have you all moved to linux already ;)

techtel-pstevens avatar Jul 29 '24 09:07 techtel-pstevens