Enabling a `shared` or `bridged` network causes `lima kubectl cluster-info` to fail

[spkane]

Description

colima is unable to verify that the Kubernetes has started correctly when a shared or bridged network is used for the underlying VM.

Version

Colima Version: 0.4.4 Lima Version: 0.11.2 Qemu Version: 7.0.0

Operating System

• [X] macOS Intel
• [ ] macOS M1
• [ ] Linux

Reproduction Steps

This used to work, but started failing at some point.

I have set up networking for lima as discussed here:

https://github.com/lima-vm/lima/blob/master/docs/network.md

And then in this file: .lima/_config/override.yaml I have this:

networks:
  - lima: bridged

When I try to start colima with the network uncommented then I get:

INFO[0000] starting colima
INFO[0000] runtime: containerd+k3s
INFO[0000] preparing network ...                         context=vm
INFO[0000] starting ...                                  context=vm
INFO[0030] provisioning ...                              context=containerd
INFO[0030] starting ...                                  context=containerd
INFO[0036] provisioning ...                              context=kubernetes
INFO[0037] starting ...                                  context=kubernetes
FATA[0062] error starting kubernetes: error running [lima kubectl cluster-info], output: "The connection to the server 127.0.0.1:6443 was refused - did you specify the right host or port?", err: "exit status 1"

Expected behaviour

I would expect the VM to come up with no errors and everything in a useable state.

Additional context

I am able to reproduce this even after recompiling and installing vde-2 and vde_vmnet.

[abiosoft]

Why not simply use --network-address flag? Is there anything extra you have in your Lima override config other than specifying the network?

[spkane]

I do not have anything else in the override file. As a matter of fact, I completely removed my lima and colima installs and started from scratch while I was trying to troubleshoot this.

This had been working fine in the past, so I was surprised that it stopped working and caused colima to generate an error during startup. I'll take a look at that flag and see if I can make it work.

https://github.com/abiosoft/colima/blob/main/docs/FAQ.md#the-virtual-machines-ip-is-not-reachable

[abiosoft]

If all you need is a reachable IP address, I would recommend you do that with Colima directly. Lima overrides are for advanced use-cases and they are not always guaranteed to work, especially when Colima also generates similar config.

I'll take a look at that flag and see if I can make it work.

It is easier than you think. Simply start with colima start --network-address and view the IP address with colima list.

[spkane]

I am a bit confused with the solution that you are recommending, only because this ticket is about the fact the colima generates an error during startup in this situation.

• FATA[0062] error starting kubernetes: error running [lima kubectl cluster-info], output: "The connection to the server 127.0.0.1:6443 was refused - did you specify the right host or port?", err: "exit status 1"

I am not trying to run lima kubectl cluster-info myself.

[abiosoft]

Yeah, colima runs the command as part of the startup for kubernetes to ensure that the startup is successful. Does it work if you comment out the network config in the lima override file?

What I suspect is that k3s bootstraps using the vmnet IP address and thereby unreachable without using the IP.

[spkane]

Yes. It works when the network override is commented out. That additional info is helpful. Thanks.

Adding --network-address seems to have resolved the error, although I see INFO[0000] dependencies missing for setting up reachable IP address context=vm in the output. It still isn't clear to me exactly what is going on, but I'll experiment a bit with it.

colima start --cpu 8 --memory 8 --disk 100 --runtime containerd --with-kubernetes --network-address

[ipv1337]

This is the closest issue I can find related to what I'm trying to achieve setting up a bridged network mode.

I have yet to see how to configure colima to start up the VM using my own network dhcp so I can get a real IP off my network. When I do try to add the network override.yaml for lima, I can start colima fine, but my network override in lima does not setup a bridge the way I'd expect it to.

My override is simple:

networks:
   - lima: bridged

Is this even possible? I've googled this many times and have found little information on this topic. Any help is much appreciated.

[jandubois]

This should work, as long as your host network interface is en0. Maybe you are using a different one. You can find the most likely candidate with (assuming you have jq installed; otherwise you have to pick it out yourself):

$ system_profiler SPNetworkDataType -json | jq -r 'first(.SPNetworkDataType[] | select(.ip_address) | .interface)'
en0

Then check your ~/.lima/_config/networks.yaml file for the definition of the bridged network:

networks:
  bridged:
    mode: bridged
    interface: en0

Make sure your host interface name matches the one in the bridged definition.

[ipv1337]

Hi @jandubois thanks for you quick response. I am using en0 actually as you can see here also:

└[~]> system_profiler SPNetworkDataType -json | jq -r 'first(.SPNetworkDataType[] | select(.ip_address) | .interface)'
en0

Here's what my ~/.lima/_config/networks.yaml file looks like which is the default:

└[~]> cat ~/.lima/_config/networks.yaml
# Path to socket_vmnet executable. Because socket_vmnet is invoked via sudo it should be
# installed where only root can modify/replace it. This means also none of the
# parent directories should be writable by the user.
#
# The varRun directory also must not be writable by the user because it will
# include the socket_vmnet pid file. Those will be terminated via sudo, so replacing
# the pid file would allow killing of arbitrary privileged processes. varRun
# however MUST be writable by the daemon user.
#
# None of the paths segments may be symlinks, why it has to be /private/var
# instead of /var etc.
paths:
# socketVMNet requires Lima >= 0.12 .
# socketVMNet has precedence over vdeVMNet.
  socketVMNet: "/opt/socket_vmnet/bin/socket_vmnet"
# vdeSwitch and vdeVMNet are DEPRECATED.
  vdeSwitch: /opt/vde/bin/vde_switch
  vdeVMNet: /opt/vde/bin/vde_vmnet
  varRun: /private/var/run/lima
  sudoers: /private/etc/sudoers.d/lima

group: everyone

networks:
  shared:
    mode: shared
    gateway: 192.168.105.1
    dhcpEnd: 192.168.105.254
    netmask: 255.255.255.0
  bridged:
    mode: bridged
    interface: en0
    # bridged mode doesn't have a gateway; dhcp is managed by outside network
  host:
    mode: host
    gateway: 192.168.106.1
    dhcpEnd: 192.168.106.254
    netmask: 255.255.255.0

This is how I'm trying to start colima:

$ colima start --cpu 8 --memory 8 --disk 100 --vm-type qemu --runtime containerd --kubernetes --activate --network-driver slirp --network-address

Here's something I observed from /etc/sudoers.d/colima that might help?

└[/etc/sudoers.d]> cat colima
# starting vmnet daemon
%staff ALL=(root:wheel) NOPASSWD:NOSETENV: /opt/colima/bin/socket_vmnet --vmnet-mode shared --socket-group staff --vmnet-gateway 192.168.106.1 --vmnet-dhcp-end 192.168.106.254 *
# terminating vmnet daemon
%staff ALL=(root:wheel) NOPASSWD:NOSETENV: /usr/bin/pkill -F /opt/colima/run/*.pid
# validating vmnet daemon
%staff ALL=(root:wheel) NOPASSWD:NOSETENV: /usr/bin/pkill -0 -F /opt/colima/run/*.pid

That is the network IP my colima instance keeps getting instead of my own home network IP that en0 gets.

└[~]> colima status
INFO[0000] colima is running using QEMU
INFO[0000] arch: x86_64
INFO[0000] runtime: containerd
INFO[0000] mountType: virtiofs
INFO[0000] address: 192.168.106.2
INFO[0000] kubernetes: enabled

If I attempt to change /etc/sudoers.d/colima, it just reverts itself when I stop/start colima. I don't see how or where I can persist a change there for colima.

[jandubois]

I'm sorry @ipv1337, I don't know how colima manages the networking setup; I was just responding why the bridged network might not get an IP address if you bridge through the wrong host adapter, but that doesn't seem to be a problem for you. Maybe @abiosoft can chime in with the correct commandline options needed to use the bridged networking.

One (unrelated) thing I find curious that you are using virtiofs with QEMU; I thought that would only work with VZ. Or maybe that is also some colima-specific extension.

[ipv1337]

I'm sorry @ipv1337, I don't know how colima manages the networking setup; I was just responding why the bridged network might not get an IP address if you bridge through the wrong host adapter, but that doesn't seem to be a problem for you. Maybe @abiosoft can chime in with the correct commandline options needed to use the bridged networking.

One (unrelated) thing I find curious that you are using virtiofs with QEMU; I thought that would only work with VZ. Or maybe that is also some colima-specific extension.

We can ignore the virtiofs, that I know is incorrect since I originally wanted to use vz but I was giving QEMU a try to see if they made any difference with the network setup I wanted.

[abiosoft]

If I attempt to change /etc/sudoers.d/colima, it just reverts itself when I stop/start colima. I don't see how or where I can persist a change there for colima.

Yeah, true. It is not expected a user would modify the file directly.

Maybe an introduction of --network-type flag (and config) would help with the options being shared (default) and bridged.

One (unrelated) thing I find curious that you are using virtiofs with QEMU; I thought that would only work with VZ. Or maybe that is also some colima-specific extension.

It is a misinformation from Colima's end, it is already corrected in main and would be part of the imminent next release.

[zack-is-cool]

Easy bridged networking would be an awesome feature. I have a usecase where I need to run x86 containers in k8s on an m1 mac and have services be exposed to another host on my network.

I was able to get bridged networking working with this config en7 is my interface that I have via ethernet:

system_profiler SPNetworkDataType -json | jq -r 'first(.SPNetworkDataType[] | select(.ip_address) | .interface)'
en7

~/.colima/default/colima.yaml:

...
network:
  address: true

~/.lima/_config/override.yaml:

networks:
- lima: bridged
  interface: en7
- vzNAT: false
  interface: col0

~/.lima/_config/networks.yaml:

paths:
# socketVMNet requires Lima >= 0.12 .
# socketVMNet has precedence over vdeVMNet.
  socketVMNet: "/opt/homebrew/Cellar/socket_vmnet/1.1.2/bin/socket_vmnet"
# vdeSwitch and vdeVMNet are DEPRECATED.
  vdeSwitch: /opt/vde/bin/vde_switch
  vdeVMNet: /opt/vde/bin/vde_vmnet
  varRun: /private/var/run/lima
  sudoers: /private/etc/sudoers.d/lima

group: everyone

networks:
  user-v2:
    mode: user-v2
    # user-v2 network is experimental network mode which supports all functionalities of default usernet network and also allows vm -> vm communication.
    # Doesn't support configuration of custom gateway; hardcoded to 192.168.5.0/24
  shared:
    mode: shared
    gateway: 192.168.105.1
    dhcpEnd: 192.168.105.254
    netmask: 255.255.255.0
  bridged:
    mode: bridged
    interface: en7
    # bridged mode doesn't have a gateway; dhcp is managed by outside network
  host:
    mode: host
    gateway: 192.168.106.1
    dhcpEnd: 192.168.106.254
    netmask: 255.255.255.0