Unable to proxy traffic from the local host via TPROXY?

[Orum]

trafficstars

I'm trying to proxy all traffic (other than that destined for a private IP) from the local machine through shadowsocks. That is to say, sslocal is running on the system that I want outgoing traffic to be proxied on, and not a separate system/router. Initially I was using shadowsocks like this: sslocal -b "127.0.0.1:1080" --protocol redir -s "192.168.x.y:8388" -m none --tcp-redir "redirect", with iptables rules taken from redsocks (and only slightly modified), i.e.:

*nat
:PREROUTING ACCEPT [0:0]
:INPUT ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
:POSTROUTING ACCEPT [0:0]
:REDSOCKS - [0:0]

-A OUTPUT -p tcp -j REDSOCKS
-A REDSOCKS -d 0.0.0.0/8 -j RETURN
-A REDSOCKS -d 10.0.0.0/8 -j RETURN
-A REDSOCKS -d 127.0.0.0/8 -j RETURN
-A REDSOCKS -d 169.254.0.0/16 -j RETURN
-A REDSOCKS -d 172.16.0.0/12 -j RETURN
-A REDSOCKS -d 192.168.0.0/16 -j RETURN
-A REDSOCKS -d 224.0.0.0/4 -j RETURN
-A REDSOCKS -d 240.0.0.0/4 -j RETURN
-A REDSOCKS -p tcp -j REDIRECT --to-ports 1080

COMMIT

This works, but obviously only proxies TCP traffic. As I'd like to also proxy UDP, I went looking first for what iptables rules should look like, and found this script, which I modified slightly as I'm not trying to bypass the GFW:

#!/bin/bash
iptables-save | grep -v shadowsocks- | iptables-restore

SHADOWSOCKS_REDIR_IP=127.0.0.1
SHADOWSOCKS_REDIR_PORT=1080

readonly IPV4_RESERVED_IPADDRS="\
0/8 \
10/8 \
127/8 \
172.16/12 \
169.254/16 \
192.168/16 \
224/4 \
240/4 \
255.255.255.255/32 \
"

## TCP+UDP
# Strategy Route
ip -4 rule del fwmark 0x1 table 803
ip -4 rule add fwmark 0x1 table 803
ip -4 route del local 0.0.0.0/0 dev lo table 803
ip -4 route add local 0.0.0.0/0 dev lo table 803

# TPROXY for LAN
iptables -t mangle -N shadowsocks-tproxy
# Skip LoopBack, Reserved
for addr in ${IPV4_RESERVED_IPADDRS}; do
	iptables -t mangle -A shadowsocks-tproxy -d "${addr}" -j RETURN
done

# Bypass LAN data
iptables -t mangle -A shadowsocks-tproxy -m addrtype --dst-type LOCAL -j RETURN
# Bypass sslocal's outbound data
iptables -t mangle -A shadowsocks-tproxy -m mark --mark 0xff/0xff -j RETURN
# UDP: TPROXY UDP to SS's port
iptables -t mangle -A shadowsocks-tproxy -p udp -j TPROXY --on-ip ${SHADOWSOCKS_REDIR_IP} --on-port ${SHADOWSOCKS_REDIR_PORT} --tproxy-mark 0x01/0x01
# TCP: TPROXY TCP to SS's port
iptables -t mangle -A shadowsocks-tproxy -p tcp -j TPROXY --on-ip ${SHADOWSOCKS_REDIR_IP} --on-port ${SHADOWSOCKS_REDIR_PORT} --tproxy-mark 0x01/0x01

# TPROXY for Local
iptables -t mangle -N shadowsocks-tproxy-mark
# Skip LoopBack, Reserved
for addr in ${IPV4_RESERVED_IPADDRS}; do
	iptables -t mangle -A shadowsocks-tproxy-mark -d "${addr}" -j RETURN
done

# TCP: conntrack
iptables -t mangle -A shadowsocks-tproxy-mark -p tcp -m conntrack --ctdir REPLY -j RETURN
# Bypass sslocal's outbound data
iptables -t mangle -A shadowsocks-tproxy-mark -m mark --mark 0xff/0xff -j RETURN
# UDP: Set MARK and reroute
iptables -t mangle -A shadowsocks-tproxy-mark -p udp -j MARK --set-xmark 0x01/0xffffffff
# TCP: Set MARK and reroute
iptables -t mangle -A shadowsocks-tproxy-mark -p tcp -j MARK --set-xmark 0x01/0xffffffff

# Apply TPROXY to LAN
iptables -t mangle -A PREROUTING -p udp -j shadowsocks-tproxy
iptables -t mangle -A PREROUTING -p tcp -j shadowsocks-tproxy
#iptables -t mangle -A PREROUTING -p udp -m addrtype ! --src-type LOCAL ! --dst-type LOCAL -j shadowsocks-tproxy
# Apply TPROXY for Local
iptables -t mangle -A OUTPUT -p udp -j shadowsocks-tproxy-mark
iptables -t mangle -A OUTPUT -p tcp -j shadowsocks-tproxy-mark
#iptables -t mangle -A OUTPUT -p udp -m addrtype --src-type LOCAL ! --dst-type LOCAL -j shadowsocks-tproxy-mark

As this uses TPROXY, I changed my shadowsocks command a bit too: sslocal -b "127.0.0.1:1080" --protocol redir -s "192.168.x.y:8388" -m none -U --tcp-redir "tproxy" --udp-redir "tproxy" --outbound-fwmark 255 (and yes, I remembered to add -U to the system running ssserver as well). I cleared the redsocks rules out of iptables, ran the script for the new rules, and restarted sslocal with the new command.

However, this does not work for proxying, either TCP or UDP. I also tried uncommenting those last two rules that are commented out (as I'm not sure if they need to be there or not), and rerunning the script, but it didn't seem to make any difference. So, does anyone know what I'm doing wrong?

[zonyitoo]

Here is an example script for configuring TPROXY: https://github.com/shadowsocks/shadowsocks-rust/blob/master/configs/iptables_tproxy.sh

Ah, it seems that you have found this example script.

I don't know what was really happening with your provided detail. My suggestion is to make a very simple rules that could match and bypass all traffics to sslocal and see if sslocal could receive any requests.

[Orum]

I'm not sure how I can tell if traffic actually arrives sslocal. It's possible the problem is with sslocal, or with iptables, but how do I tell?

After reading more about redirection in iptables though, I suspect the problem lies with it. From what I've read, to transparently proxy UDP, you need to have the rules in the mangle table of the PREROUTING chain. However, this only works for incoming traffic (e.g. on a router), and can't be used for hosts.

It looks like the script somehow tries to work around this limitation, but I don't see how? I'm no expert in iptables, though.

[zonyitoo]

Just run sslocal with -vvv, it will log out the packet when it receives from iptables.

[Orum]

So when using -vvv (and the rest of the sslocal for tproxy command I described in the original post), along with the rules from that bash script, I see no traffic reaching sslocal at all, whether it is TCP or UDP. So basically local traffic is not being transparently proxied over to shadowsocks, and I'm at a loss as to where to go from here.

Everything I've read makes it sound like such local redirection isn't possible with iptables, and so far that has been my experience as well. Does anyone have a working set of rules that proves otherwise?

[zonyitoo]

I can confirm https://github.com/shadowsocks/shadowsocks-rust/blob/master/configs/iptables_tproxy.sh works in my environment. Tested on Ubuntu, Debian, OpenWRT.

[Orum]

OpenWRT? Are you running the transparent redirection on your router?

[zonyitoo]

OpenWRT? Are you running the transparent redirection on your router?

Yes, indeed.

[Orum]

That makes sense then, as that isn't local redirection. I don't think local redirection via tproxy is possible with iptables.

[zonyitoo]

That makes sense then, as that isn't local redirection. I don't think local redirection via tproxy is possible with iptables.

Process on OpenWRT, like dnsmasq, their requests also processed by sslocal.

[ge9]

The script worked for me, in both TCP and UDP. I'm using only one machine and local packets can be processed with TPROXY. I think --outbound-fwmark 255 is also required for ssserver, to avoid infinite loop.

[zonyitoo]

ssserver do have --outbound-fwmark. But why would it need that?

[ge9]

Ah sorry I was running ssserver in the same machine. You don't need it otherwise, and your script seems perfect. I recommend troubleshoot with tcpdump and iptables' LOG function.

[Orum]

I recommend troubleshoot with tcpdump and iptables' LOG function.

What would tcpdump show me that I don't already know? iptables simply isn't redirecting traffic from localhost to sslocal. Why? I don't know. In any case, I've given up on trying to use iptables to get this working.

[ge9]

FYI, this is a packet capture of typical working TPROXY in iptables (not shadowsocks but normal SOCKS5).

10:13:38.897424 lo    In  IP 192.168.5.5.48202 > 3.132.228.249.3479: UDP, length 28
10:13:38.897566 lo    In  IP 127.0.0.1.60698 > 127.0.0.3.3130: Flags [S], seq 188947714, win 65495, options [mss 65495,sackOK,TS val 841486027 ecr 0,nop,wscale 7], length 0
10:13:38.897596 lo    In  IP 127.0.0.3.3130 > 127.0.0.1.60698: Flags [S.], seq 1277796748, ack 188947715, win 65483, options [mss 65495,sackOK,TS val 3344603128 ecr 841486027,nop,wscale 7], length 0
10:13:38.897623 lo    In  IP 127.0.0.1.60698 > 127.0.0.3.3130: Flags [.], ack 1, win 512, options [nop,nop,TS val 841486028 ecr 3344603128], length 0
10:13:38.897722 lo    In  IP 127.0.0.1.60698 > 127.0.0.3.3130: Flags [P.], seq 1:4, ack 1, win 512, options [nop,nop,TS val 841486028 ecr 3344603128], length 3
10:13:38.897739 lo    In  IP 127.0.0.3.3130 > 127.0.0.1.60698: Flags [.], ack 4, win 512, options [nop,nop,TS val 3344603128 ecr 841486028], length 0
10:13:38.897955 lo    In  IP 127.0.0.3.3130 > 127.0.0.1.60698: Flags [P.], seq 1:3, ack 4, win 512, options [nop,nop,TS val 3344603128 ecr 841486028], length 2
10:13:38.897977 lo    In  IP 127.0.0.1.60698 > 127.0.0.3.3130: Flags [.], ack 3, win 512, options [nop,nop,TS val 841486028 ecr 3344603128], length 0
10:13:38.898047 lo    In  IP 127.0.0.1.60698 > 127.0.0.3.3130: Flags [P.], seq 4:14, ack 3, win 512, options [nop,nop,TS val 841486028 ecr 3344603128], length 10
10:13:38.898129 lo    In  IP 127.0.0.3.3130 > 127.0.0.1.60698: Flags [P.], seq 3:13, ack 14, win 512, options [nop,nop,TS val 3344603128 ecr 841486028], length 10
10:13:38.898253 lo    In  IP 127.0.0.1.49384 > 127.0.0.3.53298: UDP, length 38
10:13:38.898317 enp1s0 Out IP 192.168.1.13.52394 > 3.132.228.249.3479: UDP, length 28
10:13:38.939622 lo    In  IP 127.0.0.1.60698 > 127.0.0.3.3130: Flags [.], ack 13, win 512, options [nop,nop,TS val 841486070 ecr 3344603128], length 0
10:13:39.061935 enp1s0 In  IP 3.132.228.249.3479 > 192.168.1.13.52394: UDP, length 68
10:13:39.062134 lo    In  IP 127.0.0.3.53298 > 127.0.0.1.49384: UDP, length 78
10:13:39.062243 lo    In  IP 3.132.228.249.3479 > 192.168.5.5.48202: UDP, length 68

The important thing is that transparent proxy port (in your setting, 127.0.0.1:1080) will not be shown in tcpdump. (So my advice may be a bit inappropriate, sorry)

[Orum]

I know what it should look like, but that doesn't change that iptables isn't redirecting it. I'm now just trying to get this to work on FreeBSD with pf instead, but it doesn't look like there is any documentation on what the firewall's rules should look like. I assume a simple rdr rule would be enough, but I can't find any examples in the project of the specifics.

[ge9]

My recommendation is creating another IP address for TPROXY.

sudo ip addr add 192.168.5.5 dev lo
sudo ip rule add from 192.168.5.5 lookup 803
sudo iptables -t mangle -A PREROUTING -p tcp -s 192.168.5.5 -j TPROXY --on-ip 127.0.0.1 --on-port 1080
sudo iptables -t mangle -A PREROUTING -p udp -s 192.168.5.5 -j TPROXY --on-ip 127.0.0.1 --on-port 1080