VPN routing broken

[cobrowserAlex]

trafficstars

I am unable to route traffic from the VM over my openVPN connection like it was possible in the past: https://github.com/TheNewNormal/kube-cluster-osx/issues/33

After not needing the project for a while I updated to the latest version and since them I am unable to route traffic as suggested in the kube-cluster issue.

$ /Applications/Kube-Solo.app/Contents/Resources/up.command; exit;

Starting VM ...

> booting k8solo-01 (1/1)
---> 'k8solo-01' started successfuly with address 192.168.64.3 and PID 9167
---> 'k8solo-01' boot logs can be found at '/Users/alexknol/.coreos/running/90394897-5956-4F9D-AFC4-E274C45DAB31/log'
---> 'k8solo-01' console can be found at '/Users/alexknol/.coreos/running/90394897-5956-4F9D-AFC4-E274C45DAB31/tty'

Checking internet availablity on VM...
There is no internet access from the VM !!!

Please check your Mac's firewall, network setup, stop dnsmasq (if you have installed such)
and try to fix the problem !!!

k8solo-01 VM is still running, so you can troubleshoot the network problem

When you done fixing it, do via menu 'Halt' and 'Up' and the installation will start again ...

Press [Enter] key to abort installation ...

I am running the script below after I connect to the VPN (private repository). This used to be enough, but now doesn't work anymore. Also a suggestion I found on the web: restart Corectld after vpn connection is made brings no avail.

sudo pfctl -a com.apple/utun9nat -F nat
RULE="nat on {utun0} proto {tcp, udp, icmp} from 192.168.64.0/24 to any -> {utun0}"
echo "$RULE"  | sudo pfctl -a com.apple/utun0nat -f -

As soon as I bring my vpn down all works, but I cannot reach my private repository :-(

[rimusz]

@AntonioMeireles ^^^

[AntonioMeireles]

@cobrowserAlex

long story short... at runtime corectld manipulates pf rules so that dns behaves bidirectionally (host to/from VMs)

that is achieved by running the embedded dns server (basically skydns) on port 15353 and then inserting in the pf chain a matching rule ...

>> sudo pfctl -s nat -a corectl-dns-forwarding 
rdr pass on bridge100 inet proto tcp from any to any port = 53 -> 192.168.64.1 port 15353
rdr pass on bridge100 inet proto udp from any to any port = 53 -> 192.168.64.1 port 15353

the other half is achieved by a macOS droppin in /etc/resolver/corectld along

#
# This file is automatically generated and managed by corectl
#
domain coreos.local
search coreos.local
nameserver 127.0.0.1
port 15353

So, can you paste ouput of sudo pfctl -s nat on your side plz and of sudo pfctl -s nat -a ${anchorName} for each anchor please ?

And many, many thanks in advance for your care and patience!

[cobrowserAlex]

I did this with VPN -> ON

$ sudo pfctl -s nat
No ALTQ support in kernel
ALTQ related functions disabled
nat-anchor "com.apple.internet-sharing" all
rdr-anchor "corectl-dns-forwarding" all
rdr-anchor "com.apple.internet-sharing" all

$ sudo pfctl -s nat -a com.apple.internet-sharing
No ALTQ support in kernel
ALTQ related functions disabled
nat-anchor "shared_v4" all
rdr-anchor "shared_v4" all

$ sudo pfctl -s nat -a corectl-dns-forwarding
No ALTQ support in kernel
ALTQ related functions disabled
rdr pass on bridge100 inet proto tcp from any to any port = 53 -> 192.168.64.1 port 15353
rdr pass on bridge100 inet proto udp from any to any port = 53 -> 192.168.64.1 port 15353

$ sudo pfctl -s nat -a com.apple.internet-sharing
No ALTQ support in kernel
ALTQ related functions disabled
nat-anchor "shared_v4" all
rdr-anchor "shared_v4" all

[AntonioMeireles]

@cobrowserAlex i don't see your rules above ... did you run (when capturing above)

sudo pfctl -a com.apple/utun9nat -F nat
RULE="nat on {utun0} proto {tcp, udp, icmp} from 192.168.64.0/24 to any -> {utun0}"
echo "$RULE"  | sudo pfctl -a com.apple/utun0nat -f -

?

[cobrowserAlex]

yes, I did:

$ RULE="nat on {utun0} proto {tcp, udp, icmp} from 192.168.64.0/24 to any -> {utun0}"
alexknol@Alexs-MacBook-Pro-2~ $ echo "$RULE"  | sudo pfctl -a com.apple/utun0nat -f -
pfctl: Use of -f option, could result in flushing of rules
present in the main ruleset added by the system at startup.
See /etc/pf.conf for further details.

No ALTQ support in kernel
ALTQ related functions disabled

$ sudo pfctl -s nat -a com.apple.internet-sharing
No ALTQ support in kernel
ALTQ related functions disabled
nat-anchor "shared_v4" all
rdr-anchor "shared_v4" all

$ sudo pfctl -s nat -a corectl-dns-forwarding
No ALTQ support in kernel
ALTQ related functions disabled
rdr pass on bridge100 inet proto tcp from any to any port = 53 -> 192.168.64.1 port 15353
rdr pass on bridge100 inet proto udp from any to any port = 53 -> 192.168.64.1 port 15353

maybe I have a syntax error in my part ?

[AntonioMeireles]

interesting. will try to reproduce in next 24hrs.

[cobrowserAlex]

great!

[cobrowserAlex]

@AntonioMeireles were you able to find time to check this out?

[rbellamy]

@AntonioMeireles I'm also anxiously awaiting word back from you about this...