Multipass VMs don't route traffic over VPN established on host (MacOS)

[funkypenguin]

Hey folks,

On MacOS, I use Tunnelblick to establish a VPN to my datacenter. Specific routes are added to my host routing table (192.168.33.0/24, in this case), a datacenter host is reachable via the host OS.

[davidy:~] % ping 192.168.33.3
PING 192.168.33.3 (192.168.33.3): 56 data bytes
64 bytes from 192.168.33.3: icmp_seq=0 ttl=63 time=23.411 ms
64 bytes from 192.168.33.3: icmp_seq=1 ttl=63 time=27.703 ms
^C
--- 192.168.33.3 ping statistics ---
2 packets transmitted, 2 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 23.411/25.557/27.703/2.146 ms
[davidy:~] %

When I launch and shell into a Multipass VM, the VM does not have access to the datacenter host:

[davidy:~] % multipass shell batman
Welcome to Ubuntu 18.04.3 LTS (GNU/Linux 4.15.0-76-generic x86_64)
<snip>
ubuntu@batman:~$ ping 192.168.33.3
PING 192.168.33.3 (192.168.33.3) 56(84) bytes of data.
^C
--- 192.168.33.3 ping statistics ---
5 packets transmitted, 0 received, 100% packet loss, time 4336ms

ubuntu@batman:~$

Interestingly, when I switch to virtualbox support, with sudo multipass set local.driver=virtualbox, then the VM does have access to the datacenter host.

Given the above, my conclusion is that Hyperkit behaves differently with host-based VPN connections vs virtualbox, or that NAT is perhaps only applied to traffic egressing the host's default route.

Is there anything I can do to get Multipass VMs using Hyperkit to send egress traffic using the full routing table of the host?

Thanks! D

[amrox]

I just ran into this problem yesterday too!

It may be obvious, but I'll point out that an ubuntu docker image works fine.

I'd be happy to share information to help diagnose the issue.

[amrox]

Just tried again in multipass 1.1 and the same issue persists :/

[Saviq]

Hi @amrox the problem, in general, is that the VPN does not allow/configure traffic from bridge100 through the VPN - and it blocks all unknown traffic. And I don't think we should be messing with your VPN config… Not to mention that there's countless VPN solutions out there, each with different configuration, and we're unlikely to be able to implement support for all of them.

If you can find out how to configure yours to route Multipass instances' traffic through, we'd love to document that.

https://blog.netnerds.net/2016/11/share-vpn-with-os-x-sierra-internet-sharing/ seems to be a distillation of what I found on the subject.

[negletios]

Here is an approach:

Find your VPN's interface of Tunnelblick. In my case is utun2.

netstat -nr | grep utun2

Find which one you want to allow on your brigde100

Let's say is X.X.X.X./32.

Add the following line into /etc/pf.conf/ (order does matter here. Maybe after nat-anchor):

nat on utun2 from bridge100:network to X.X.X.X./32 -> (utun2)

Rewrite pf configuration:

sudo pfctl -f /etc/pf.conf

Restart multipass, not the instance.

I hope it works for you too.

[atika]

I encountered the same issue while setting up Wireguard VPN (and I was also configuring macOS PF to redirect the traffic to the VM). I ended with the same solution of @negletios for the bridge network of the VM pass through the host VPN:

Edit /etc/pf.conf (or create a new one elsewhere and start with a Launchd daemon) :

nat-anchor "com.apple/*"
nat-anchor "wireguard"
rdr-anchor "com.apple/*"

Create a script to toggle the rules:

#!/bin/bash

if [[ ${1} == "up" ]]
then
	echo "Inserting NAT Rules for Wireguard"
	cat <<-EOF | sudo pfctl -a wireguard -f -
		nat on utun1 proto {tcp, udp, icmp} from 192.168.64.0/24 to !192.168.1.0/24 -> utun1
		nat on en1 proto {tcp, udp, icmp} from 192.168.64.0/24 to 192.168.1.0/24 -> en1
	EOF
else
	echo "Removing NAT Rules for Wireguard"
	sudo pfctl -a wireguard -F all
fi

Start macOS packet filter with the new config (this flush any existing rules):

sudo pfctl -f /etc/pf.conf -e

or with a more advanced startup script:

sysctl -w net.inet.ip.forwarding=1
pfctl -d
pfctl -F all
pfctl -vf custom.conf -e

To see the rules:

sudo pfctl -s nat
sudo pfctl -a wireguard -s nat

[mozzbozz]

I found two possible solutions for this:

Option 1: Setup VPN inside VM

Install VPN inside the multipass VM. Quite simple solution that I simply never thought of until the end. But probably something that I would have done as the first thing in a VM with an actual GUI.

Option 2: Fiddling around with firewall rules

Based on the offical troubleshooting guide (scroll down to "Potential workaround for VPN conflicts (ref: #495)"), I came up with the following solution:

sudo nano /etc/pf.conf

After the nat … line (if there is one, otherwise at the end) in /etc/pf.conf, add this line:

nat on tap0 from bridge100:network to 192.168.2.1/24 -> (tap0)
nat on tap0 from bridge100:network to any -> (en0)

This assumes, I only want to access resources from the VPN in the 192.168.2.XXX ip-range. But it still allows me to access the internet and all other resources (as the VPN in my example only allows access to certain resources, but not the internet).

This also assumes, the VPN (in my case Tunnelblick) uses tap0 as the network adapter. If you have a tun-device, replace it accordingly.

Then reload PF with

sudo pfctl -f /etc/pf.conf

It should work instantly without any restarting of VMs etc.

[RussellLuo]

In my case:

• I use Tunnelblick, whose interface is utun1
• I want to access resources from somewhere (e.g. 192.168.11.22) via VPN

Finally, I got it to work by adding the following two lines into /etc/pf.conf (after the nat-anchor ... line):

# nat-anchor "com.apple/*"
nat on utun1 from bridge100:network to 192.168.11.22 -> (utun1)
nat on en0 from bridge100:network to any -> (en0)

[drackp2m]

· · ·

#!/bin/bash

if [[ ${1} == "up" ]]
then
	echo "Inserting NAT Rules for Wireguard"
	cat <<-EOF | sudo pfctl -a wireguard -f -
		nat on utun1 proto {tcp, udp, icmp} from 192.168.64.0/24 to !192.168.1.0/24 -> utun1
		nat on en1 proto {tcp, udp, icmp} from 192.168.64.0/24 to 192.168.1.0/24 -> en1
	EOF
else
	echo "Removing NAT Rules for Wireguard"
	sudo pfctl -a wireguard -F all
fi

· · ·

In my case I replace utun1 by utun3, 192.168.1.0 by 192.168.31.0 (Router Gateway) and en1 by en0 and it works perfect. ^_^

[rjcarr]

I'm still seeing this problem on an M1 Mac with multipass v1.8.1. I'll try the solutions mentioned here, but I'd like to note that I've also setup a VMware Fusion Ubuntu VM (same release as in multipass) and that properly uses the host VPN for its networking, so a solution here is possible.

[stefan-novak-brt]

@rjcarr -- I just came across the same issue. Did you have any luck?

[rjcarr]

@stefan-novak-brt: sorry, no, I just gave up and stuck with vmware fusion; it seems there is a solution involving custom packet filtering, but I'm not experienced enough to even attempt something like that.

[stefan-novak-brt]

@rjcarr : thanks for the quick response! I was able to get an OpenVPN client running inside the Multipass VM with openvpn --config client.ovpn --daemon, which worked for my use-case.

[rjcarr]

@stefan-novak-brt: Great, glad you got it working; I didn't even attempt a multipass vpn because the host was also running a vpn, but you're right, that likely would have solved it. Thanks for the info!

[jhchnc]

For my use case, just start openconnect inside container...

[ebeinges]

This advice is still relevant. Thanks to all for the information!

Just want to share a slight hack that made this work better for me. I have a VPN installed by IT that only routes the VPN traffic to the VPN interface (for me utun3), general internet traffic goes to en0.

# nat-anchor "com.apple/*"
nat on utun3 from bridge100:network to 192.168.11.22 -> (utun3)
nat on en0 from bridge100:network to any -> (en0)

worked, but would not survive a restart. The issue is that the bridge100:network is resolved when pf.conf is read so it is a race condition on startup. Since multipass is configured to use 192.168.64.0/24 as the local IP range, setting this in the rules makes things much more stable:

# nat-anchor "com.apple/*"
nat on utun3 from 192.168.64.0/24 to 192.168.11.22 -> (utun3)
nat on en0 from 192.168.64.0/24 to any -> (en0)

[townsend2010]

Duplicate of #1121