Ways to get ICMP to work

[tugtugtug]

First of all, I apologize if this is not the right place to ask questions, as I'm not sure where is the best place to do so.

After some digging, on MacOS, it seems the only way to get ICMP to work with QEMU is to use a tap with a bridged network. This requires root access plus a no longer supported tuntap network extension. So I'm wondering if lima would be able to support a networking mode that would overcome this limitation? I've tried vde vmnet with both the normal and bridge mode, neither seems to make a difference.

The reason I'm asking is, tools like gradle would fail requests with network unreachable when ICMP is blocked, and this is a show stopper for our usage.

Any input to workaround this issue is appreiciated.

[jrmuizel]

What does HyperKit do for ICMP?

[jandubois]

What does HyperKit do for ICMP?

Hyperkit itself does not do networking at all. Applications normally combine it with vpnkit (also a docker component, like hyperkit), or use the vmnet framework (from Apple) to provide networking functionality.

@jrmuizel Why do you ask? Do you have a working setup based on hyperkit, that supports ICMP? Can you share any details about how it is configured?

[jrmuizel]

@jrmuizel Why do you ask? Do you have a working setup based on hyperkit, that supports ICMP? Can you share any details about how it is configured?

No, just Docker for Mac which it sounds like is using vpnkit

[jandubois]

Docker for Mac which it sounds like is using vpnkit

Yes, it is. vpnkit is only partially open-source, so it is not an option for Lima.

Also (like hyperkit) it is written in OCaml, which most people don't understand, so it would be hard to maintain/debug.

I still hope that it will be possible to support icmp in bridged mode with vde_vmnet.

[tugtugtug]

Just an update, I'm playing with this patch https://www.mail-archive.com/[email protected]/msg832954.html I wasn't able to get the bridge mode working, but the shared mode works, and I think it is potentially okay to run without root if the binary can be signed properly. Just wondering how this would differ from the vde vmnet driver lima is integrating with tho.

[AkihiroSuda]

I think it is potentially okay to run without root if the binary can be signed properly.

Apple doesn't allow signing a binary with com.apple.vm.networking entitlement.

They say This entitlement is restricted to developers of virtualization software. To request this entitlement, contact your Apple representative.

https://developer.apple.com/documentation/bundleresources/entitlements/com_apple_vm_networking

---

Just wondering how this would differ from the vde vmnet driver lima is integrating with tho.

Explained in https://github.com/lima-vm/vde_vmnet/tree/v0.5.0#how-is-vde_vmnet-related-to-qemu-builtin-vmnet-support

How is vde_vmnet related to QEMU-builtin vmnet support?

There are proposal to add builtin vmnet support for QEMU:

• https://lore.kernel.org/qemu-devel/[email protected]/
• https://lore.kernel.org/qemu-devel/[email protected]/

However, QEMU-builtin vmnet is highly likely to require running the entire QEMU process as root.

On the other hand, vde_vmnet does not require the entire QEMU process to run as root, though vde_vmnet has to run as root.

[tugtugtug]

Thank you @AkihiroSuda for the detailed response,

I think it is potentially okay to run without root if the binary can be signed properly.

Apple doesn't allow signing a binary with com.apple.vm.networking entitlement.

They say This entitlement is restricted to developers of virtualization software. To request this entitlement, contact your Apple representative.

Ah, evil Apple strikes again...

https://developer.apple.com/documentation/bundleresources/entitlements/com_apple_vm_networking

Just wondering how this would differ from the vde vmnet driver lima is integrating with tho.

Explained in https://github.com/lima-vm/vde_vmnet/tree/v0.5.0#how-is-vde_vmnet-related-to-qemu-builtin-vmnet-support

How is vde_vmnet related to QEMU-builtin vmnet support?

There are proposal to add builtin vmnet support for QEMU:

• https://lore.kernel.org/qemu-devel/[email protected]/
• https://lore.kernel.org/qemu-devel/[email protected]/

However, QEMU-builtin vmnet is highly likely to require running the entire QEMU process as root. On the other hand, vde_vmnet does not require the entire QEMU process to run as root, though vde_vmnet has to run as root.

That makes a lot of sense, I'm all in favor of having it as a separate process and run it with less privileges, the only issue right now is the current vde_vmnet implementation does not seem to offer the required functionality, i.e. ICMP. It would be wonderful if we could achieve feature parity with vde_vmnet. Until then, I'm still blocked of using lima, which is unfortunate.

[jrmuizel]

podman also seems to have working icmp. What does it use?

[jouve]

it looks like podman uses gvproxy

ref: https://github.com/containers/podman/blob/main/docs/tutorials/mac_experimental.md#using-podman-machine-on-macos-x86_64-and-apple-silicon

[guillaumerose]

ICMP in gvproxy is broken. It is just the virtual gateway that answers to all messages. The only place I know it works, it's in vpnkit.

[mfriedenhagen]

Hi, has something changed in the meantime?

• I just installed colima 0.4.3 on macOS 12.4 (Intel)
• I removed my previous files (rm -rf ~/.colima ; rm -rf ~/.lima).
• Previous versions I got fantasy times when pinging services:

❯ docker run --rm library/alpine ping -c1 repo.XXX
PING repo.XXX (10.76.16.50): 56 data bytes
64 bytes from 10.76.16.50: seq=0 ttl=254 time=2130363.442 ms
 
--- repo.XXX ping statistics ---
1 packets transmitted, 1 packets received, 0% packet loss
round-trip min/avg/max = 2130363.442/2130363.442/2130363.442 ms

• Now it looks more reasonable:

❯ docker run --rm library/alpine ping -c4 repo.XXX
PING repo.XXX (10.76.68.8): 56 data bytes
64 bytes from 10.76.68.8: seq=0 ttl=63 time=0.403 ms
64 bytes from 10.76.68.8: seq=1 ttl=63 time=0.983 ms
64 bytes from 10.76.68.8: seq=2 ttl=63 time=1.253 ms
64 bytes from 10.76.68.8: seq=3 ttl=63 time=0.898 ms

--- repo.XXX ping statistics ---
4 packets transmitted, 4 packets received, 0% packet loss
round-trip min/avg/max = 0.403/0.884/1.253 ms

[jandubois]

I think colima is using gvproxy, which might explain why ping works for it on macOS.

I've tested it again now (with Lima 0.12, on both Ubuntu and Alpine), and ping works fine on Linux (always has, afaik) and continues to be broken on macOS.

This is user-mode networking from qemu (slirp), so not related to VMNET at all.

Given that it works fine on Linux I think it should be possible to fix this in qemu for macOS as well, but don't have time to look into.

Linux (openSUSE Leap 15.2 on x86_64)

jan@localhost:~> limactl shell default ping -c 3 8.8.8.8
PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data.
64 bytes from 8.8.8.8: icmp_seq=1 ttl=255 time=8.32 ms
64 bytes from 8.8.8.8: icmp_seq=2 ttl=255 time=8.16 ms
64 bytes from 8.8.8.8: icmp_seq=3 ttl=255 time=7.73 ms

--- 8.8.8.8 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2004ms
rtt min/avg/max/mdev = 7.728/8.069/8.324/0.250 ms
jan@localhost:~> limactl shell alpine ping -c 3 8.8.8.8
PING 8.8.8.8 (8.8.8.8): 56 data bytes
64 bytes from 8.8.8.8: seq=0 ttl=42 time=9.213 ms
64 bytes from 8.8.8.8: seq=1 ttl=42 time=8.765 ms
64 bytes from 8.8.8.8: seq=2 ttl=42 time=7.774 ms

--- 8.8.8.8 ping statistics ---
3 packets transmitted, 3 packets received, 0% packet loss
round-trip min/avg/max = 7.774/8.584/9.213 ms

macOS (Catalina on x86_64)

$ limactl shell default ping -c 3 8.8.8.8
PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data.
ping: Warning: time of day goes back (-3022297129608106255us), taking countermeasures
ping: Warning: time of day goes back (-3022297129608106142us), taking countermeasures
64 bytes from 8.8.8.8: icmp_seq=0 ttl=255 time=0.000 ms
wrong data byte #16 should be 0x10 but was 0x0
#16	0 4 0 1 b2 b3 23 63 0 0 0 0 9 74 3 0 0 0 0 0 10 11 12 13 14 15 16 17 18 19 1a 1b
#48	1c 1d 1e 1f 20 21 22 23
ping: Warning: time of day goes back (-2126080803760374706us), taking countermeasures
64 bytes from 8.8.8.8: icmp_seq=0 ttl=255 time=0.000 ms (DUP!)
wrong data byte #16 should be 0x10 but was 0x0
#16	0 4 0 2 b3 b3 23 63 0 0 0 0 78 7f 3 0 0 0 0 0 10 11 12 13 14 15 16 17 18 19 1a 1b
#48	1c 1d 1e 1f 20 21 22 23
ping: Warning: time of day goes back (-1097852713835345139us), taking countermeasures
64 bytes from 8.8.8.8: icmp_seq=0 ttl=255 time=0.000 ms (DUP!)
wrong data byte #16 should be 0x10 but was 0x0
#16	0 4 0 3 b4 b3 23 63 0 0 0 0 bc 8c 3 0 0 0 0 0 10 11 12 13 14 15 16 17 18 19 1a 1b
#48	1c 1d 1e 1f 20 21 22 23

--- 8.8.8.8 ping statistics ---
3 packets transmitted, 1 received, +2 duplicates, 66.6667% packet loss, time 2006ms
rtt min/avg/max/mdev = 0.000/0.000/0.000/0.000 ms, pipe 4
$ limactl shell alpine ping -c 3 8.8.8.8
PING 8.8.8.8 (8.8.8.8): 56 data bytes
64 bytes from 8.8.8.8: seq=0 ttl=42 time=686944.898 ms
64 bytes from 8.8.8.8: seq=0 ttl=42 time=687950.062 ms (DUP!)
64 bytes from 8.8.8.8: seq=0 ttl=42 time=688950.704 ms (DUP!)
^C
--- 8.8.8.8 ping statistics ---
3 packets transmitted, 1 packets received, 2 duplicates, 66% packet loss
round-trip min/avg/max = 686944.898/687948.554/688950.704 ms

[nuta]

👋 I ran into similar issue in another OS running on top of QEMU and have submitted a fix to libslirp. I think the patch also fixes this Lima's issue. Hope it helps.

[DesktopECHO]

$ limactl shell default ping -c 3 8.8.8.8
PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data.
ping: Warning: time of day goes back (-3022297129608106255us), taking countermeasures
ping: Warning: time of day goes back (-3022297129608106142us), taking countermeasures
64 bytes from 8.8.8.8: icmp_seq=0 ttl=255 time=0.000 ms
wrong data byte #16 should be 0x10 but was 0x0
#16	0 4 0 1 b2 b3 23 63 0 0 0 0 9 74 3 0 0 0 0 0 10 11 12 13 14 15 16 17 18 19 1a 1b
#48	1c 1d 1e 1f 20 21 22 23

I built libslirp-master the other day and was pleased to see ping is now fixed in Lima.

[iamshreeram]

@nuta , @DesktopECHO - I have upgraded Lima to version 19.0.1. However, the issue persists despite merging the new fix on libslirp. Could you please guide me on building libslirp on my current Ubuntu vm? thanks