corectl
corectl copied to clipboard
TheNewNormal
port redirects broken when corectld is running corectld
I have a custom anchor for my port redirects (port 80 to mehserve), but once corectld is running those rules disappear.
I have a script to print PF rules: https://gist.github.com/vitaly/cd0024d232a3a134b00e5496e4e779b8
here is the output w/o corectld: https://gist.github.com/vitaly/10b9f4cd467370c226daf78424f02816
here is the output with corectld: https://gist.github.com/vitaly/b40d230cc72b88fbcb7c1ade87a12163
The difference is this:
--- 1 2016-11-30 15:10:05.000000000 +0100
+++ 2 2016-11-30 15:10:27.000000000 +0100
@@ -1,9 +1,6 @@
- r scrub-anchor "com.apple/*" all fragment reassemble
- r anchor "com.apple/*" all
- n nat-anchor "com.apple/*" all
- n nat-anchor "custom" all
- n rdr-anchor "com.apple/*" all
- n rdr-anchor "custom" all
+ n nat-anchor "com.apple.internet-sharing" all
+ n rdr-anchor "corectl-dns-forwarding" all
+ n rdr-anchor "com.apple.internet-sharing" all
A com.apple
A com.apple.internet-sharing
A corectl-dns-forwarding
Note that not only my 'custom' anchors are skipped but the default apple ones as well.
I don't know go, so I'd rather not try to change the code (for now ;)). but I think I do know what needs to be done in terms of PF.
right now it does pfctl -F pfC_file which in turn loads pfR_file. pfC declares 2 anchors: corectl-dns-forwarding, and com.apple.internet-sharing, and then loads corectl-dns-forwarding from pfR. The problem is that it clears the default "root" config (e.g. all com.apple/* anchors).
but default config already uses rdr rules from com.apple/*, so instead we can just load pfR directly into this namespace:
pfctl -a com.apple/corectl-dns-forwarding -f pfR_file
and to remove it:
pfctl -a com.apple/corectl-dns-forwarding -Fa
It'a a bug, my fault - you're spot on. I' m on the road ATM, will fix it tomorrow. And many, many thanks!
@AntonioMeireles it looks like kube-solo also messes up with it. Not sure if directly or through corectl though.
@vitaly
Hi again, could you please fetch latest tip, built it, and confirm that it fixes this for you ? Thanks in advance!
@AntonioMeireles please see my comment on https://github.com/TheNewNormal/corectl/issues/106
couldn't build it, not a golang user.
OK, so I finally got it to compile using hints from /Applications/corectl.app/Contents/Resources/get_go_binaries/corectl_compile_from_source_corectl.sh
DNS port configuration seems to be working, but PF rules are still messed up. Actually it seems to be worse, as I can't even connect outside from a VM.
Note: I'm again using my pf rules dump script, as it produces a much shorter output and much easier to diff. You can see the script here: https://gist.github.com/vitaly/cd0024d232a3a134b00e5496e4e779b8
rules after firewall reset:
r scrub-anchor "com.apple/*" all fragment reassemble
r anchor "com.apple/*" all
n nat-anchor "com.apple/*" all
n rdr-anchor "com.apple/*" all
A com.apple
A com.apple.internet-sharing
com.apple r anchor "200.AirDrop/*" all
com.apple r anchor "250.ApplicationFirewall/*" all
com.apple A com.apple/200.AirDrop
com.apple A com.apple/250.ApplicationFirewall
com.apple A com.apple/mehserve
com.apple/200.AirDrop A com.apple/200.AirDrop/Bonjour
com.apple/200.AirDrop/Bonjour r pass in on p2p0 inet6 proto udp from any to any port = 5353 keep state
com.apple/200.AirDrop/Bonjour r pass out on p2p0 proto tcp all flags any keep state
com.apple/250.ApplicationFirewall r scrub in all fragment reassemble
com.apple/250.ApplicationFirewall r block drop in inet proto icmp all icmp-type echoreq
com.apple/250.ApplicationFirewall r block drop in inet proto icmp all icmp-type echoreq
com.apple/250.ApplicationFirewall r block drop in inet6 proto ipv6-icmp all icmp6-type echoreq
com.apple/mehserve n rdr pass inet proto tcp from any to 127.0.0.1 port = 80 -> 127.0.0.1 port 12439
rules after corectld started:
r scrub-anchor "com.apple/*" all fragment reassemble
r anchor "com.apple/*" all
n nat-anchor "com.apple/*" all
n rdr-anchor "com.apple/*" all
A com.apple
A com.apple.internet-sharing
com.apple r anchor "200.AirDrop/*" all
com.apple r anchor "250.ApplicationFirewall/*" all
com.apple A com.apple/200.AirDrop
com.apple A com.apple/250.ApplicationFirewall
com.apple A com.apple/corectl-dns-forwarding
com.apple A com.apple/mehserve
com.apple/200.AirDrop A com.apple/200.AirDrop/Bonjour
com.apple/200.AirDrop/Bonjour r pass in on p2p0 inet6 proto udp from any to any port = 5353 keep state
com.apple/200.AirDrop/Bonjour r pass out on p2p0 proto tcp all flags any keep state
com.apple/250.ApplicationFirewall r scrub in all fragment reassemble
com.apple/250.ApplicationFirewall r block drop in inet proto icmp all icmp-type echoreq
com.apple/250.ApplicationFirewall r block drop in inet proto icmp all icmp-type echoreq
com.apple/250.ApplicationFirewall r block drop in inet6 proto ipv6-icmp all icmp6-type echoreq
com.apple/corectl-dns-forwarding n nat-anchor "com.apple.internet-sharing" all
com.apple/corectl-dns-forwarding n rdr-anchor "corectl-dns-forwarding" all
com.apple/corectl-dns-forwarding n rdr-anchor "com.apple.internet-sharing" all
com.apple/corectl-dns-forwarding A com.apple/corectl-dns-forwarding/com.apple.internet-sharing
com.apple/corectl-dns-forwarding A com.apple/corectl-dns-forwarding/corectl-dns-forwarding
com.apple/mehserve n rdr pass inet proto tcp from any to 127.0.0.1 port = 80 -> 127.0.0.1 port 12439
here is the diff:
--- 1-after-reset 2016-12-02 03:49:00.000000000 +0100
+++ 2-after-corectld-started 2016-12-02 03:50:50.000000000 +0100
@@ -9,6 +9,7 @@
com.apple r anchor "250.ApplicationFirewall/*" all
com.apple A com.apple/200.AirDrop
com.apple A com.apple/250.ApplicationFirewall
+com.apple A com.apple/corectl-dns-forwarding
com.apple A com.apple/mehserve
com.apple/200.AirDrop A com.apple/200.AirDrop/Bonjour
com.apple/200.AirDrop/Bonjour r pass in on p2p0 inet6 proto udp from any to any port = 5353 keep state
@@ -17,4 +18,9 @@
com.apple/250.ApplicationFirewall r block drop in inet proto icmp all icmp-type echoreq
com.apple/250.ApplicationFirewall r block drop in inet proto icmp all icmp-type echoreq
com.apple/250.ApplicationFirewall r block drop in inet6 proto ipv6-icmp all icmp6-type echoreq
+com.apple/corectl-dns-forwarding n nat-anchor "com.apple.internet-sharing" all
+com.apple/corectl-dns-forwarding n rdr-anchor "corectl-dns-forwarding" all
+com.apple/corectl-dns-forwarding n rdr-anchor "com.apple.internet-sharing" all
+com.apple/corectl-dns-forwarding A com.apple/corectl-dns-forwarding/com.apple.internet-sharing
+com.apple/corectl-dns-forwarding A com.apple/corectl-dns-forwarding/corectl-dns-forwarding
com.apple/mehserve n rdr pass inet proto tcp from any to 127.0.0.1 port = 80 -> 127.0.0.1 port 12439
Note the weird anchors: com.apple/corectl-dns-forwarding/com.apple.internet-sharing, and com.apple/corectl-dns-forwarding/corectl-dns-forwarding.
Once you properly nest your rules inside com.apple/*, there is no need to declare the anchors, just the rules.
Since at this stage you don't yet add any commands, I think instead of adding nat-anchor and rdr-anchor you only need to reset them (Also note the renamed internet-sharing anchor to properly nest it inside com.apple/):
sudo pfctl -a com.apple/corectl-dns-forwarding -Fa
sudo pfctl -a com.apple/internet-sharing -Fa
But at least at this point my mehserve is still working.
Now, here is how it looks once I run a VM with it:
r scrub-anchor "com.apple/*" all fragment reassemble
r scrub-anchor "com.apple.internet-sharing" all fragment reassemble
r anchor "com.apple/*" all
r anchor "com.apple.internet-sharing" all
n nat-anchor "com.apple/*" all
n nat-anchor "com.apple.internet-sharing" all
n rdr-anchor "com.apple/*" all
n rdr-anchor "com.apple.internet-sharing" all
A com.apple
A com.apple.internet-sharing
com.apple r anchor "200.AirDrop/*" all
com.apple r anchor "250.ApplicationFirewall/*" all
com.apple A com.apple/200.AirDrop
com.apple A com.apple/250.ApplicationFirewall
com.apple A com.apple/corectl-dns-forwarding
com.apple A com.apple/mehserve
com.apple/200.AirDrop A com.apple/200.AirDrop/Bonjour
com.apple/200.AirDrop/Bonjour r pass in on p2p0 inet6 proto udp from any to any port = 5353 keep state
com.apple/200.AirDrop/Bonjour r pass out on p2p0 proto tcp all flags any keep state
com.apple/250.ApplicationFirewall r scrub in all fragment reassemble
com.apple/250.ApplicationFirewall r block drop in inet proto icmp all icmp-type echoreq
com.apple/250.ApplicationFirewall r block drop in inet proto icmp all icmp-type echoreq
com.apple/250.ApplicationFirewall r block drop in inet6 proto ipv6-icmp all icmp6-type echoreq
com.apple/corectl-dns-forwarding n nat-anchor "com.apple.internet-sharing" all
com.apple/corectl-dns-forwarding n rdr-anchor "corectl-dns-forwarding" all
com.apple/corectl-dns-forwarding n rdr-anchor "com.apple.internet-sharing" all
com.apple/corectl-dns-forwarding A com.apple/corectl-dns-forwarding/com.apple.internet-sharing
com.apple/corectl-dns-forwarding A com.apple/corectl-dns-forwarding/corectl-dns-forwarding
com.apple/mehserve n rdr pass inet proto tcp from any to 127.0.0.1 port = 80 -> 127.0.0.1 port 12439
com.apple.internet-sharing r scrub-anchor "shared_v4" all fragment reassemble
com.apple.internet-sharing r anchor "shared_v4" all
com.apple.internet-sharing n nat-anchor "shared_v4" all
com.apple.internet-sharing n rdr-anchor "shared_v4" all
com.apple.internet-sharing A com.apple.internet-sharing/shared_v4
com.apple.internet-sharing/shared_v4 r scrub on en0 all no-df fragment reassemble
com.apple.internet-sharing/shared_v4 r scrub on bridge100 all no-df max-mss 1460 fragment reassemble
com.apple.internet-sharing/shared_v4 r scrub on bridge100 proto esp all no-df fragment reassemble
com.apple.internet-sharing/shared_v4 r pass on en0 all flags any keep state
com.apple.internet-sharing/shared_v4 r pass on en0 proto esp all no state
com.apple.internet-sharing/shared_v4 n nat on en0 inet from 192.168.64.0/24 to any -> (en0:0) extfilter ei
com.apple.internet-sharing/shared_v4 n no nat on bridge100 inet from 192.168.64.1 to 192.168.64.0/24
here is the diff:
--- 2-after-corectld-started 2016-12-02 03:50:50.000000000 +0100
+++ 3-after-corectl-run 2016-12-02 03:53:11.000000000 +0100
@@ -1,8 +1,12 @@
./pfdump
r scrub-anchor "com.apple/*" all fragment reassemble
+ r scrub-anchor "com.apple.internet-sharing" all fragment reassemble
r anchor "com.apple/*" all
+ r anchor "com.apple.internet-sharing" all
n nat-anchor "com.apple/*" all
+ n nat-anchor "com.apple.internet-sharing" all
n rdr-anchor "com.apple/*" all
+ n rdr-anchor "com.apple.internet-sharing" all
A com.apple
A com.apple.internet-sharing
com.apple r anchor "200.AirDrop/*" all
@@ -24,3 +28,15 @@
com.apple/corectl-dns-forwarding A com.apple/corectl-dns-forwarding/com.apple.internet-sharing
com.apple/corectl-dns-forwarding A com.apple/corectl-dns-forwarding/corectl-dns-forwarding
com.apple/mehserve n rdr pass inet proto tcp from any to 127.0.0.1 port = 80 -> 127.0.0.1 port 12439
+com.apple.internet-sharing r scrub-anchor "shared_v4" all fragment reassemble
+com.apple.internet-sharing r anchor "shared_v4" all
+com.apple.internet-sharing n nat-anchor "shared_v4" all
+com.apple.internet-sharing n rdr-anchor "shared_v4" all
+com.apple.internet-sharing A com.apple.internet-sharing/shared_v4
+com.apple.internet-sharing/shared_v4 r scrub on en0 all no-df fragment reassemble
+com.apple.internet-sharing/shared_v4 r scrub on bridge100 all no-df max-mss 1460 fragment reassemble
+com.apple.internet-sharing/shared_v4 r scrub on bridge100 proto esp all no-df fragment reassemble
+com.apple.internet-sharing/shared_v4 r pass on en0 all flags any keep state
+com.apple.internet-sharing/shared_v4 r pass on en0 proto esp all no state
+com.apple.internet-sharing/shared_v4 n nat on en0 inet from 192.168.64.0/24 to any -> (en0:0) extfilter ei
+com.apple.internet-sharing/shared_v4 n no nat on bridge100 inet from 192.168.64.1 to 192.168.64.0/24
Seems like it is rewriting the global scope again, which is not needed if you just nest the rules inside com.apple/.
At this stage when I corectl ssh into the VM I can't even ping 8.8.8.8, where previously I could ping it, just couldn't resolve any hostnames (due to DNS port conflict, the forwarding was working)
Note that the DNS rules are missing now. I tried adding them, but it didn't help (din't expect it to, it has nothing to do with ping)
I tried to mess with the rules for a while, but couldn't get it to work, so I reverted to the old version for now.
@vitaly
sorry lag, can you try again against latest tip ? (and many thanks for your patience!)
just tried the latest tip.
result:
- DNS and ping work from inside VM
- my port redirection doesn't ;(
PF dumps: https://gist.github.com/vitaly/fc0cb6986a2a3d92fb86b2239c8cf1cf
Looking at the current corectl pf config, I see that it was done outside of 'com.apple' on purpose ;). and, to my untrained eye it actually looks OK vs breaking other rules. i.e. I can't see how it can break my redirect, but it does ;(. On the other hand I also don't understand how your corectld-forwarding-dns anchor works. it is not referenced from root or any other anchors (btw, can you please explain this? ;)). it clearly does work though, because I do have DNS inside the VM, but I really can't figure out how it manages to do that. which means I have no idea how this PF thing actually works ;)
I think I already mentioned this, but it might be worth pointing out again.
My redirect still works once corectld is started and stops working as soon as I run any VM
BUT
it still doesn't work even after I stop VM and corectd, even though PF dump looks exactly like before corectld was started
only once I run sudo pfctl -f /etc/pf.conf it starts working again, but I can not detect any difference in PF config before and after I do that. The only idea that I have is that its some kind of transient state that is getting cleared when I re–run the root scope config.
Hopefully this helps ... ;)
interesting... can you paste your exact redirect rules and how you are loading them please... (and once again MANY thanks for your patience!)
Pinging as I have a similar problem. corectl and sshuttle also don't get along so well.
@vitaly and @AntonioMeireles maybe pfctl needs to be called with -E, the token stored somewhere, then torn down with -X + that token? That looks like that might solve this particular problem.