crc
crc copied to clipboard
crc-org
[Epic] Add functionality to communicate from WSL(2) to the CRC VM on Windows
Although it is early day, WSL2 uses Hyper-V and provides a nice experience when it comes to coding (like remote-WSL from VSCode). Although for WSL2 there is no direct way to communicate from the WSL Virtual Switch to the Default Switch. A possibility would be to add the WSL switch tothe CRC VM once the environment is enabled, and configure accordingly so oc from WSL can communicate with the CRC VM.
Note: this behaviour is different from WSL(v1) where the Windows networking stack is shared between Windows and the Linux susbsystem
Enable forwarding for the vSwitch allows traffic https://github.com/code-ready/crc/issues/374#issuecomment-803580645 however, this only works with the legacy vSwitch mode, not vsock.
Added note by @gbraad: Do NOT install CRC inside the WSL2 environment. This is a nested virtualization setup and is therefore unsupported.
Hi there, currently I am trying the provision based on wsl2 ubuntu box(using the crc binary of linux version). When I run crc setup inside wsl2 box, it shows some error messages:
david@XXXX:/mnt/c/WINDOWS/system32$ crc setup --log-level debug
INFO Checking if NetworkManager is installed
DEBU Checking if 'nmcli' is available
DEBU 'nmcli' was found in /usr/bin/nmcli
INFO Checking if NetworkManager service is running
DEBU Checking if NetworkManager.service is running
DEBU exit status 1 : System has not been booted with systemd as init system (PID 1). Can't operate.
FATA NetworkManager is required. Please make sure it is installed and running manually
I have done apt install network-manager and started the service manually to make sure it is running. It turns out that in the source code:
// https://github.com/code-ready/crc/blob/master/pkg/crc/preflight/preflight_checks_linux.go#L458
......
path, err := exec.LookPath("systemctl")
...
stdOut, stdErr, err := crcos.RunWithDefaultLocale(path, "is-active", "NetworkManager")
...
Currently the code is using the command systemctl is-active NetworkManager, which is not working now in WSL2 box, it should be like service network-manager status and check the result for "running".
Is it worthy of making a PR to cover the provision for wsl2 box? If so, I am glad to do it. Looking forward to ur idea.
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
![stale[bot] avatar](https://avatars.githubusercontent.com/in/1724?v=4 "Published by a pub.dev verified publisher"){kind=small}
The issue here talks about communication between WSL(2) and CRC-VM, like using oc, but actually targeting the underlying Hyper-V to host the CRC-vm.
@nevermosby Note that the setup you tried would never be supported as that involves nested-virtualization in which the environment runs hyper-v -> kvm -> crc-vm instead of hyper-v -> crc-vm or kvm -> crc-vm. Also note that WSL2 is a non-standard configuration that uses their own init system in-place of the distros (and modified kernel). This causes issues when something like systemd is expected. We might come back to this in the future.
Hi,
Where are we with this issue, now WSL 2 has been officially released?
Without this issue fixed I cannot use Quarkus to send s2i native builds to openshift because the quarkus-openshift extension uses Dekorate which uses OC under the wood.
I found a work around for this issue here -> https://github.com/microsoft/WSL/issues/4288#issuecomment-652259640 Basically you need to retrieve the switch IDs of the virtual switches for CRC and WSL and then enable forwarding support. Open PowerShell (run as admin)
Retrieve IDs
Get-NetIPInterface | select ifIndex,InterfaceAlias,AddressFamily,ConnectionState,Forwarding | Sort-Object -Property IfIndex | Format-Table
Enabling forwarding for WSL - change MYvEthernetWSLID to the id number of your WSL switch from the previous command
Set-NetIPInterface -ifindex MYvEthernetWSLID -Forwarding Enabled
Enable forwading for CRC - change MYvEthernetSWITCHID to the id number of your WSL switch from the previous command
Set-NetIPInterface -ifindex MYvEthernetSWITCHID -Forwarding Enabled
Immediately after running this I was able to run oc commands in WSL2 without issue.
The above workaround worked for me - thanks @mageru!
My own poor-man's workaround:
Run oc proxy on the Windows host, bound to the WSL bridge, and access it via the bridge IP from within WSL.
This is rather more involved than @mageru's workaround, but doesn't require tampering with Windows networking:
Here's the details for anyone interested:
On the Windows host:
Retrieve WSL bridge IP address:
netsh interface ip show addresses "vEthernet (WSL)"
Login to the CRC cluster:
oc login -u kubeadmin -p <token> https://api.crc.testing:6443
Start oc proxy/kubectl proxy (accept firewall prompt):
oc proxy --port=8001 --api-prefix='/' --address=<WSL bridge IP> --accept-hosts=".*"
From within WSL:
Retrieve WSL bridge IP address:
WSL_BRIDGE_IP=$(grep nameserver /etc/resolv.conf | sed 's/nameserver //')
Configure an oc/kubectl context to access the CRC cluster:
oc config set clusters.crc.server $WSL_BRIDGE_IP:8001
oc config set contexts.crc.cluster crc
oc config use-context crc
And you're good to go.
Note: Hyper-V bridge IPs seem to change with each system restart, so automate this by putting the corresponding commands in a Windows startup script and in your ~/.profile in WSL.
@zcahana The solution does not work for me.
Is there still an issue with WSL2 connectivity?
On Windows host:
C:\xxx>crc config view
- consent-telemetry : no
- network-mode : vsock
- pull-secret-file : C:\xxx\pull-secret.txt
C:\xxx>crc daemon
INFO listening vsock://00000400-FACB-11E6-BD58-64006A7986D3
INFO listening \\.\pipe\crc-http
2021/03/16 07:05:50 tcpproxy: for incoming conn 127.0.0.1:62444, error dialing "192.168.127.2:22": context deadline exceeded
2021/03/16 07:06:01 tcpproxy: for incoming conn 127.0.0.1:62458, error dialing "192.168.127.2:22": context deadline exceeded
2021/03/16 07:06:02 tcpproxy: for incoming conn 127.0.0.1:62459, error dialing "192.168.127.2:6443": context deadline exceeded
2021/03/16 07:06:12 tcpproxy: for incoming conn 127.0.0.1:61351, error dialing "192.168.127.2:22": context deadline exceeded
INFO new connection from 47925440-0000-0000-0000-000000000000:ded05241-facb-11e6-bd58-64006a7986d3 to 00000000-0000-0000-0000-000000000000:00000400-facb-11e6-bd58-64006a7986d3
INFO assigning 192.168.127.2/24 to 47925440-0000-0000-0000-000000000000:ded05241-facb-11e6-bd58-64006a7986d3
C:\xxx>crc status
CRC VM: Running
OpenShift: Running (v4.7.0)
Disk Usage: 14.23GB of 32.74GB (Inside the CRC VM)
Cache Usage: 25.26GB
Cache Directory: C:\xxx\.crc\cache
C:\xxx>netsh interface ip show addresses "vEthernet (WSL)"
Configuration for interface "vEthernet (WSL)"
DHCP enabled: No
IP Address: 172.26.208.1
Subnet Prefix: 172.26.208.0/20 (mask 255.255.240.0)
InterfaceMetric: 5000
C:\xxx>oc proxy --port=8001 --api-prefix='/' --address=172.26.208.1 --accept-hosts=".*"
Starting to serve on 172.26.208.1:8001
On WSL2:
> lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 20.04.2 LTS
Release: 20.04
Codename: focal
> WSL_BRIDGE_IP=$(grep nameserver /etc/resolv.conf | sed 's/nameserver //')
> oc config set clusters.crc.server $WSL_BRIDGE_IP:8001
Property "clusters.crc.server" set.
> oc config set contexts.crc.cluster crc
Property "contexts.crc.cluster" set.
> oc config use-context crc
Switched to context "crc".
> oc version
Client Version: 4.7.1
Unable to connect to the server: dial tcp 172.26.208.1:8001: i/o timeout
Edit: After changing the inbound rule for the firewall (had to choose: New Rule -> Port -> TCP -> Specify local ports: 8001 -> Allow the connection -> Marked all -> Enter a name -> Finish) I have at least now a connection to 172.26.208.1:8001 but oc seems not to be able to establish the connection to the running proxy.
On WSL2:
> nmap -p 8001 172.26.208.1
Starting Nmap 7.80 ( https://nmap.org ) at 2021-03-16 11:23 CET
Nmap scan report for xxx (172.26.208.1)
Host is up (0.00036s latency).
PORT STATE SERVICE
8001/tcp open vcom-tunnel
> oc version
Client Version: 4.7.1
-> # Missing "Server Version: 4.7.0" here
Edit 2: Strange. There was already a rule which implies for my understanding the custom rule I've added:
This is what I have been using since: https://github.com/gbraad/windows-dotfiles/blob/main/setup-wsl.ps1
# Allow traffic from Hyper-V VMs to WSL and vice versa
Set-NetIPInterface -ifAlias "vEthernet (WSL)" -Forwarding Enabled
Set-NetIPInterface -ifAlias "vEthernet (Default Switch)" -Forwarding Enabled
I almost wanna close this issue as this is NOT an issue with CRC, but the defaults of Windows. I have spoken to several of the WSL engineers and they are aware of it and looking towards a solution. There is nothing we can do to improve this besides pointing out the forwarding rule.
@zcahana The solution does not work for me.
- network-mode : vsock
You are using vsock which binds port 22 for SSH to a local port anyway, which is 2222:
https://github.com/code-ready/crc/blob/899cffc759d8019f7993c978598641fd488c49e9/pkg/crc/constants/constants.go#L37
This same is for the API, etc. These ports should be enabled for access. WSL1 has no issues here, but WSL2 prevents access to the local environment as it runs as a VM. See previous note ^^^
When I run crc setup inside wsl2 box, it shows some error messages:
Do NOT install or run CRC from the WSL2 environment. This requires nested virtualization; this is unsupported as it impacts performance. Plus, the used kernel did not originally come with this functionality enabled. Install as a native Hyper-V VM from the Windows host and merely use oc from the WSL2 environment.
/cc: @nevermosby
@gbraad Still not working. Last try :). Do you see what's wrong here?
When you use vsock you can not use it as if you described by those before here. Since i do not know where these IP addresses come from, I have no idea what to look for. 172. Indicate external (like DHCP assigned), so that is never gonna work as the endpoint as these aren't able to connect to the localhost listener. With vsock you bind 2222 for ssh on localhost (and api, etc) so you need netsh to expose this for external use. (and yes, a VM is regarded as external to the host itself).
I might have a look at this later in the week. Currently working on something else (tray).
Greetings.
Maybe someone will find this helpful.
In my case changing port forwarding command did not help. and after reboot adapter status returns to initial state = Disabled. I meant this one "Set-NetIPInterface -ifindex MYvEthernetSWITCHID -Forwarding Enabled".
In my case helped to use "oc.exe" command under WSL2 shell suddenly this command start to work "oc.exe login -u developer -p developer https://api.crc.testing:6443" Login successful.
after this step, I just crated a symbolic link to oc.exe sudo ln --symbolic /mnt/c/Hyper-V/IBM-OpenShift/oc.exe /usr/sbin/oc
and now I'm able to use OC under WSL2 shell.
my previous attempts to use OC for Linux Download oc for Linux for x86_64 faced with issue
The connection to the server api.crc.testing:6443 was refused - did you specify the right host or port?
So essentially you arejust using a direct connection from windows. Since this isnt the Linux client zome things might not work or different
From: TarasHalakaItera @.> Sent: Tuesday, May 3, 2022 1:40:10 PM To: code-ready/crc @.> Cc: Gerard Braad @.>; Mention @.> Subject: Re: [code-ready/crc] [Epic] Add functionality to communicate from WSL(2) to the CRC VM on Windows (#374)
Greetings.
Maybe someone will find this helpful.
In my case changing port forwarding command did not help. and after reboot adapter status returns to initial state = Disabled. I meant this one "Set-NetIPInterface -ifindex MYvEthernetSWITCHID -Forwarding Enabled".
In my case helped to use "oc.exe" command under WSL2 shell suddenly this command start to work "oc.exe login -u developer -p developer https://api.crc.testing:6443" Login successful.
after this step, I just crated a symbolic link to oc.exe sudo ln --symbolic /mnt/c/Hyper-V/IBM-OpenShift/oc.exe /usr/sbin/oc
and now I'm able to use OC under WSL2 shell.
my previous attempts to use OC for Linux Download oc for Linux for x86_64https://downloads-openshift-console.apps-crc.testing/amd64/linux/oc.tar faced with issue
The connection to the server api.crc.testing:6443 was refused - did you specify the right host or port?
— Reply to this email directly, view it on GitHubhttps://github.com/code-ready/crc/issues/374#issuecomment-1115755699, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AAAAOZV33XKS7WQC6PGHLW3VIC33VANCNFSM4IJDNFNA. You are receiving this because you were mentioned.Message ID: @.***>
So essentially you arejust using a direct connection from windows. Since this isnt the Linux client zome things might not work or different
Looks like so. but so far so good. at least for my study-cases with CRC.
autocomplete also works under WSL2.
Actually, in my case, neither the IP forwarding nor the port forwarding nor using oc.exe within WSL2 really solve the problem. The two former never work, and the latter may work for oc commands but not for other tools like helm. Essentially, crc is still UNUSABLE with WSL2, which is a shame as it defeats the purpose of having a working testing / development environment for OpenShift on a Windows system. Are there any plans at all to resolve this? Maybe offer an option during crc install to use the WSL2 network instead of the Hyper-V Default switch (which is a rather lame implementation of a virtual network, compared to VirtualBox)? Thanks.
With the latest WSL2 on Windows 11; I think the problem has changed. Even with the forwarding with between network switches; and with or without the kubectl proxy above a new error is
error: couldn't get https://172.22.6.140:6443/.well-known/oauth-authorization-server: unexpected response status 401
Where 172... is the IP of the windows host from the perspective of wsl2.
I've reproduced @TarasHalakaItera approach of using oc.exe rather than the Linux oc command. Looking in the OC logs, there is a notable difference.
oc.exe uses
I0303 14:05:41.015011 25436 request_token.go:98] SSPI Enabled
where as oc
I0303 13:49:11.734592 61491 request_token.go:93] GSSAPI Enabled
Having OC working is ok-ish - but then if want to run any kubectl command I can't.
@mbwhite
What happens is that oc.exe is invoked outside the 'modified' network stack of Linux. This means that the command works directly from the host, outside the network assignment.
SSPI is a Windows specific security provider, as the alternative to GSSAPI. This is not what causes the network related issues. The 401 might be returned as for authentication, it is not the expected endpoint and it concludes 'Unauthorized'. Would need more information what you have tried or set up.
Note: You can do the same for kubectl.exe.
To add another solution into the mix this worked for me:
- In WSL2, get the IP of host machine by running
cat /etc/resolv.conf. This may change when WSL2 is rebooted. - In WSL2, edit
/etc/hostsand replace127.0.0.1for the crc related entries with the host machine IP (e.g.172.31.48.1) - In Windows, in powershell run
netsh interface portproxy add v4tov4 listenport=6443 listenaddress=0.0.0.0 connectport=6443 connectaddress=127.0.0.1to listen globally for api connections and forward them. - In Windows, add a firewall rule to allow this traffic from the private network CIDR used by Hyper-V
New-NetFirewallRule -DisplayName "CRC API from WSL2" -Direction Inbound -LocalPort 6443 -Protocol TCP -Action Allow -RemoteAddress 172.16.0.0/12. - In WSL2, run
curl -k https://api.crc.testing:6443to check connectivity.
Edit: ~~with this method you need to delete the portproxy with netsh interface portproxy delete v4tov4 listenport=6443 listenaddress=0.0.0.0 before you start crc and then add it back after crc has started.~~ I no longer have to do this with the latest version of crc.
Edit2: updated with more relevant IPs.
This should be fixed when the new mirrored network mode becomes GA https://devblogs.microsoft.com/commandline/windows-subsystem-for-linux-september-2023-update/#new-networking-mode-mirrored
I can confirm I've successfully used CRC OKD with mirrored network mode and (superficially - ie could talk to cluster API) connectivity worked out of the box from WSL2 environment
Can confirm @hoo29's solution works for me.
However, even though I'm using the latest version of crc, I still need to delete the port proxy with netsh before starting crc again, and then add it back again after.
i.e.
netsh interface portproxy delete v4tov4 listenport=6443 listenaddress=0.0.0.0
crc start
netsh interface portproxy add v4tov4 listenport=6443 listenaddress=0.0.0.0 connectport=6443 connectaddress=127.0.0.1
Remember to regularly get the IP address for the host from /etc/resolv.conf and put it in /etc/hosts, as windows constantly regenerates both of those files.
At least the firewall rule only has to be added once!
@tim-stasse - I don't use crc at the moment so not sure about the port proxy rules but I had this helper script to auto update my hosts file
#!/bin/bash
set -euo pipefail
NAMESERVER=$(grep nameserver /etc/resolv.conf | tail -n 1 | cut -d ' ' -f 2)
HOSTS_LINE=$(grep "api.crc.testing" /etc/hosts)
DOMAINS=$(echo "$HOSTS_LINE" | sed "s/\s\+/ /g" | cut -d " " -f 2-)
sed -i "/api.crc.testing/c\\$NAMESERVER $DOMAINS" /etc/hosts
Save this to /root/update_hosts.sh, chmod +x it, and then echo "@reboot root /root/update_hosts.sh" >> /etc/crontab to automatically run it on every reboot. I found it can take a couple of mins after a reboot to run but gets there eventually.
Mirrored mode networking appears to be GA (despite what the docs say).
With the below .wslconfig on Windows 11 23H2 build 22631.2715 and WSL 2.0.9, everything just works now 😄
[wsl2]
networkingMode=mirrored
firewall=true
dnsTunneling=true
Adding resources for others to understand networkingMode and dnsTunneling: Accessing network applications with WSL
and Advanced settings configuration in WSL