shadowsocks-rust
shadowsocks-rust copied to clipboard
Transparently proxying UDP traffic with pf?
This is a similar symptom to my earlier issue (#1473), but as the circumstances have drastically changed I thought I'd open a new issue. To summarize the changes, I am no longer running sslocal on the host (Linux/iptables) machine; it's instead being run on my FreeBSD router which uses pf.
Obviously, this requires different firewall rules to work, but there is little if any documentation I could find for transparently proxying via pf within the shadowsocks-rust project. What I've done here is largely guesswork, but this is the rule I came up with to transparently proxy both TCP & UDP to sslocal (and then on to ssserver):
rdr on $int_if inet proto { tcp, udp } to !<private> -> 127.0.0.1 port 1080
sslocal is being run on the router via the following command:
sslocal -b 127.0.0.1:1080 -U --protocol redir -s 192.168.x.y:8388 -m none --tcp-redir pf --udp-redir pf
...and finally, ssserver is run with -U -m none -b 192.168.x.y:8388
This "works" in the sense that both TCP & UDP traffic are being redirected to sslocal (unlike in #1473 where only TCP traffic was ever reaching sslocal), which is then sending the traffic on to sserver. However, once again, only TCP traffic is being fully proxied correctly.
UDP traffic does arrive at sslocal without any doubt, as it shows up when running sslocal with -vvv like so (timestamps removed for brevity, and the host system's address being substituted here with simply <host>):
TRACE tokio-runtime-worker ThreadId(05) shadowsocks::relay::udprelay::proxy_socket: crates/shadowsocks/src/relay/udprelay/proxy_socket.rs:235: UDP server client send to 127.0.0.1:1080, control: UdpSocketControlData { client_session_id: 7043831170210357480, server_session_id: 0, packet_id: 1, user: None }, payload length 96 bytes, packet length 103 bytes
TRACE tokio-runtime-worker ThreadId(05) shadowsocks_service::local::redir::udprelay: crates/shadowsocks-service/src/local/redir/udprelay/mod.rs:307: received UDP packet from <host>:44849, destination 127.0.0.1:1080, length 96 bytes
TRACE tokio-runtime-worker ThreadId(05) shadowsocks_service::local::net::udp::association: crates/shadowsocks-service/src/local/net/udp/association.rs:433: udp relay <host>:44849 -> 127.0.0.1:1080 (proxied) with 96 bytes
This traffic is then forwarded on to ssserver, which I can verify by watching outgoing traffic on the router via tcpdump. After they arrive at ssserver though, they fail to reach the internet, so clearly something isn't configured correctly.
My best guess as to what is happening here, based on the logs, is that sslocal thinks the destination of the UDP packet is 127.0.0.1 (i.e. the address that sslocal is running on and where incoming UDP traffic on the router is redirected to), and fails to retrieve the original destination address before pf's rdr rule took effect. This wouldn't surprise me as my rdr rule is complete guesswork, so I assume I'm missing something.
If that is indeed the problem, what should the rdr rule look like to transparently proxy UDP traffic with pf? Or am I barking up the wrong tree, and is the problem due to something else entirely?
I have never tested on FreeBSD, so this project supports FreeBSD is still "theoretically".
I have tested pf on macOS, which supports TCP perfectly just like you did on FreeBSD. But on macOS, UDP's destination must be retrieved from ioc_getstates. This API is far different from FreeBSD's definition, so I think FreeBSD doesn't use the same mechanism to get UDP packets' destination when redirected by pf.
In your last PR, you said on FreeBSD UDP's destination could be retrieved from msg_name returned by recvfrom. Is that correct? I couldn't see any document from FreeBSD talking about it.
The 127.0.0.1:1080 in your case was exactly the value of msg_name returned from recv_dest_from.
In your last PR, you said on FreeBSD UDP's destination could be retrieved from msg_name returned by recvfrom. Is that correct? I couldn't see any document from FreeBSD talking about it.
That wasn't me, so you must be thinking of someone else. While I know how to write rules for pf fairly well, I've never programmed anything to interface with it, so I'm unsure of how the API is supposed to be used.
The good news is that you can access the man pages online. There's also a forum, mailing lists, and a Discord server, all of which feature some form of development/programming discussion.
I'm interested in learning as well, but I'm still learning rust, so it may be some time until I am of use.
Glad to know that you are familiar with FreeBSD and pf. Could you find any doc / references about how to get the "original destination address" of UDP packets?
On Linux, it is quite straight forward: https://man7.org/linux/man-pages/man7/ip.7.html (IP_RECVORIGDSTADDR).
On macOS, I got it from another post: (?) Deleted.
A famous tools mitmproxy only supports TCP on FreeBSD: https://docs.mitmproxy.org/stable/howto-transparent/ .
So it is quite hard to find a reference about how FreeBSD handles UDP redirects and how to get the original destination address programmatically.
And you could also try the ipfw: https://forums.freebsd.org/threads/retrieving-the-destination-port-in-an-ipfwed-udp-packet.37545/
@Orum Do you still interest in making this Project working on FreeBSD? What's the current status?
I am still very interested in making it work. However, I think the best way to do so at this point is to just read through FreeBSD's source code. As such, I'm familiarizing myself with it, but it's going to take some time as I have other obligations and projects demanding my attention, so it's on the back burner for me right now. This is probably the best, or at least, surest way to figure out how to get the address (assuming it's even possible at present) from pf.
Additionally, I'd need to learn much more about rust before I contribute anything other than information to this project, as my knowledge of it is spotty at best. But, if I do figure out the FreeBSD side of the picture, perhaps someone here can handle that side of things once they have the information. I'm also not only willing, but happy to test things to see if they work once they're implemented.
https://github.com/semigodking/redsocks/issues/200
FYI, I succeeded in running transparent proxy in FreeBSD with redsocks+SOCKS5 (sorry they are completely unrelated to shadowsocks) and ipfw, but not with pf. When I tested with pf, the UDP destination address was obtained as the transparent proxy's listen address, just as @Orum mentioned.
As I mentioned in the link, in OpenBSD, where pf originates from, using divert-to rule of pf worked perfectly. divert rule seems not ported to FreeBSD's pf.
FreeBSD and OpenBSD diverged in their rules syntax quite a few years back when the latter made some changes that broke older rule sets, IIRC. I've stuck with FreeBSD, which might have an equivalent of divert-to (or not), but I don't really know what it does as I never bothered to keep up with OBSD's syntax as I don't use it.
I also don't really want to switch to ipfw, or any other firewall, as I really value simplicity and clarity of pf's rules' syntax. So many other firewalls are hard to decipher, which is undesirable to say the least when it comes to security.
In any case I will try and dedicate some time to reading through and understanding the pf/bpf code in the future, but that will be some time from now.
@ge9 Have you tried shadowsocks-rust with ipfw? What problem did you see?
I tested it and it worked, but there were a problem.
shadowsocks-rust uses IPv4-mapped IPv6 address to return message to IPv4 clients:
https://github.com/shadowsocks/shadowsocks-rust/blob/ac67a08f87afe54e844241fb75d8bfc018a0bd03/crates/shadowsocks-service/src/local/redir/udprelay/mod.rs#L199
However, FreeBSD disables IPv4-mapped IPv6 address by default (according to https://github.com/scala-native/scala-native/issues/3630). If I set sysctl net.inet6.ip6.v6only=0, then it worked.
Except for this, normal transparent proxy setting will work.
FYI, My settings are following:
- Initial setup (the machine already had 10.0.2.15/24)
#!/bin/sh
kldload ipfw
fwcmd=ipfw
ifconfig em0 alias 10.0.2.25 netmask 0xffffff00
$fwcmd add 100 allow all from any to any via lo0
$fwcmd add 500 fwd 127.0.0.1,22222 tcp from 10.0.2.25 to any
$fwcmd add 600 fwd 127.0.0.1,22222 udp from 10.0.2.25 to any
$fwcmd add 700 allow ip from any to any
sslocal -b 127.0.0.1:22222 -U --protocol redir -s 192.168.1.7:1080 --tcp-redir pf --udp-redir pf -m none- (specifying
pfseems fine)
- (specifying
ssserver -U -s 192.168.1.7:1080 -m none(on another PC)- note that specifying
0.0.0.0:1080here doesn't work
- note that specifying
There is a SUPPORT_IPV6_TRANSPARENT flag that indicates whether the current platform supports IPv6, if it is true, then shadowsocks-rust will always convert IPv4 to IPv4-mapped-IPv6 and uses only one socket for sending back.
This is always working for Linux, because dual-stack socket was enabled by default for most distribution of Linux.
Is there a way to test if dual-stack is enabled on FreeBSD? By reading sysctl net.inet6.ip6.v6only=0 should be one possible option.
Could you please help to run the test program on FreeBSD and see if any of these syscalls returned errors? https://stackoverflow.com/questions/30184377/how-to-detect-if-dual-stack-socket-is-supported
cc @madeye , FreeBSD doesn't support IPv4-mapped-IPv6 by default. So my previous (years ago) PR that uses IPv6 sockets for sending back UDP packets won't be able to run properly on FreeBSD.
This is the result of the python test program.
Traceback (most recent call last):
File "/root/test.py", line 4, in <module>
s.connect(('::ffff:169.254.1.1', 53))
OSError: [Errno 22] Invalid argument
If I set sysctl net.inet6.ip6.v6only=0 and commented out s.setsockopt(socket.IPPROTO_IPV6, socket.IPV6_V6ONLY, 1), it worked. Otherwise, it failed.
In the shadowsocks, this is the error.
2024-06-17T03:45:10.334750916+09:00 WARN udp failed to send back 68 bytes to client 10.0.2.25:44550, from target 3.132.228.249:3478 (proxied), error: Invalid argument (os error 22)
This originates from https://github.com/shadowsocks/shadowsocks-rust/blob/ac67a08f87afe54e844241fb75d8bfc018a0bd03/crates/shadowsocks-service/src/local/redir/udprelay/mod.rs#L203 .
Please test if it is Ok without settingnet.inet6.ip6.v6only=0 manually.
I just found a better solution. Working on it.
Tested on OpenWRT. Please help testing it if it is working correctly on FreeBSD. @ge9
5ba8b7d worked but 765c9e5 or cd25d25 (latest) doesn't (same error).
Please run again with -v and check these debug logs:
https://github.com/shadowsocks/shadowsocks-rust/blob/cd25d25b22a3a9b1fa5805e1482826e6214f797a/crates/shadowsocks/src/net/sys/mod.rs#L143-L181
Are they all true?
here is the log. seems all true.
2024-06-17T06:38:52.894052461+09:00 TRACE tokio-runtime-worker ThreadId(02) shadowsocks_service::local::redir::udprelay: crates/shadowsocks-service/src/local/redir/udprelay/mod.rs:285: received UDP packet from 10.0.2.25:65158, destination 3.132.228.249:3478, length 28 bytes
2024-06-17T06:38:52.894445849+09:00 DEBUG tokio-runtime-worker ThreadId(02) shadowsocks_service::local::net::udp::association: crates/shadowsocks-service/src/local/net/udp/association.rs:123: created udp association for 10.0.2.25:65158
2024-06-17T06:38:52.894817377+09:00 TRACE tokio-runtime-worker ThreadId(02) shadowsocks_service::local::net::udp::association: crates/shadowsocks-service/src/local/net/udp/association.rs:433: udp relay 10.0.2.25:65158 -> 3.132.228.249:3478 (proxied) with 28 bytes
2024-06-17T06:38:52.894896617+09:00 TRACE tokio-runtime-worker ThreadId(02) mio::poll: /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/mio-0.8.11/src/poll.rs:551: registering event source with poller: token=Token(79171555479936), interests=READABLE | WRITABLE
2024-06-17T06:38:52.895050148+09:00 TRACE tokio-runtime-worker ThreadId(02) shadowsocks::relay::udprelay::proxy_socket: crates/shadowsocks/src/relay/udprelay/proxy_socket.rs:103: connected udp remote 192.168.1.8:1080 (outbound: 192.168.1.8:1080) with ConnectOpts { user_cookie: None, bind_local_addr: None, bind_interface: None, tcp: TcpSocketOpts { send_buffer_size: None, recv_buffer_size: None, nodelay: false, fastopen: false, keepalive: Some(15s), mptcp: false }, udp: UdpSocketOpts { mtu: None } }
2024-06-17T06:38:52.895129216+09:00 TRACE tokio-runtime-worker ThreadId(02) shadowsocks::relay::udprelay::proxy_socket: crates/shadowsocks/src/relay/udprelay/proxy_socket.rs:235: UDP server client send to 3.132.228.249:3478, control: UdpSocketControlData { client_session_id: 786820869274048462, server_session_id: 0, packet_id: 1, user: None }, payload length 28 bytes, packet length 35 bytes
2024-06-17T06:38:53.135364801+09:00 TRACE tokio-runtime-worker ThreadId(02) shadowsocks::relay::udprelay::proxy_socket: crates/shadowsocks/src/relay/udprelay/proxy_socket.rs:445: UDP server client receive from 3.132.228.249:3478, control: None, packet length 75 bytes, payload length 68 bytes
2024-06-17T06:38:53.136228332+09:00 TRACE tokio-runtime-worker ThreadId(02) shadowsocks_service::local::net::udp::association: crates/shadowsocks-service/src/local/net/udp/association.rs:607: udp relay 10.0.2.25:65158 <- 3.132.228.249:3478 (proxied) received 68 bytes
2024-06-17T06:38:53.136421625+09:00 DEBUG tokio-runtime-worker ThreadId(02) shadowsocks::net::sys: crates/shadowsocks/src/net/sys/mod.rs:155: IpStackCapability support_ipv4=true
2024-06-17T06:38:53.136798744+09:00 DEBUG tokio-runtime-worker ThreadId(02) shadowsocks::net::sys: crates/shadowsocks/src/net/sys/mod.rs:164: IpStackCapability support_ipv6=true
2024-06-17T06:38:53.137187564+09:00 DEBUG tokio-runtime-worker ThreadId(02) shadowsocks::net::sys: crates/shadowsocks/src/net/sys/mod.rs:175: IpStackCapability support_ipv4_mapped_ipv6=true
2024-06-17T06:38:53.13739486+09:00 TRACE tokio-runtime-worker ThreadId(02) mio::poll: /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/mio-0.8.11/src/poll.rs:551: registering event source with poller: token=Token(79171555480192), interests=READABLE | WRITABLE
2024-06-17T06:38:53.137685127+09:00 WARN tokio-runtime-worker ThreadId(02) shadowsocks_service::local::net::udp::association: crates/shadowsocks-service/src/local/net/udp/association.rs:620: udp failed to send back 68 bytes to client 10.0.2.25:65158, from target 3.132.228.249:3478 (proxied), error: Invalid argument (os error 22)
Well, so FreeBSD allows bind()ing a IPv4-mapped-IPv6 address, but doesn’t allow sendmsg to it. Programs that are written in Go should also fails with the same reason.
Call connect() in this commit. Please help verify it again.
Hmm it seems still not working...
2024-06-16T23:01:19.392825484Z TRACE tokio-runtime-worker ThreadId(03) shadowsocks_service::local::redir::udprelay: crates/shadowsocks-service/src/local/redir/udprelay/mod.rs:285: received UDP packet from 10.0.2.25:13870, destination 3.132.228.249:3478, length 28 bytes
2024-06-16T23:01:19.393494564Z DEBUG tokio-runtime-worker ThreadId(03) shadowsocks_service::local::net::udp::association: crates/shadowsocks-service/src/local/net/udp/association.rs:123: created udp association for 10.0.2.25:13870
2024-06-16T23:01:19.393869192Z TRACE tokio-runtime-worker ThreadId(03) shadowsocks_service::local::net::udp::association: crates/shadowsocks-service/src/local/net/udp/association.rs:433: udp relay 10.0.2.25:13870 -> 3.132.228.249:3478 (proxied) with 28 bytes
2024-06-16T23:01:19.394037649Z TRACE tokio-runtime-worker ThreadId(03) mio::poll: /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/mio-0.8.11/src/poll.rs:551: registering event source with poller: token=Token(88188202799232), interests=READABLE | WRITABLE
2024-06-16T23:01:19.394211694Z TRACE tokio-runtime-worker ThreadId(03) shadowsocks::relay::udprelay::proxy_socket: crates/shadowsocks/src/relay/udprelay/proxy_socket.rs:103: connected udp remote 192.168.1.212:1080 (outbound: 192.168.1.212:1080) with ConnectOpts { user_cookie: None, bind_local_addr: None, bind_interface: None, tcp: TcpSocketOpts { send_buffer_size: None, recv_buffer_size: None, nodelay: false, fastopen: false, keepalive: Some(15s), mptcp: false }, udp: UdpSocketOpts { mtu: None } }
2024-06-16T23:01:19.394416748Z TRACE tokio-runtime-worker ThreadId(03) shadowsocks::relay::udprelay::proxy_socket: crates/shadowsocks/src/relay/udprelay/proxy_socket.rs:235: UDP server client send to 3.132.228.249:3478, control: UdpSocketControlData { client_session_id: 8860848107368566370, server_session_id: 0, packet_id: 1, user: None }, payload length 28 bytes, packet length 35 bytes
2024-06-16T23:01:19.566423525Z TRACE tokio-runtime-worker ThreadId(03) shadowsocks::relay::udprelay::proxy_socket: crates/shadowsocks/src/relay/udprelay/proxy_socket.rs:445: UDP server client receive from 3.132.228.249:3478, control: None, packet length 75 bytes, payload length 68 bytes
2024-06-16T23:01:19.567810852Z TRACE tokio-runtime-worker ThreadId(03) shadowsocks_service::local::net::udp::association: crates/shadowsocks-service/src/local/net/udp/association.rs:607: udp relay 10.0.2.25:13870 <- 3.132.228.249:3478 (proxied) received 68 bytes
2024-06-16T23:01:19.568463729Z DEBUG tokio-runtime-worker ThreadId(03) shadowsocks::net::sys: crates/shadowsocks/src/net/sys/mod.rs:156: IpStackCapability support_ipv4=true
2024-06-16T23:01:19.568883894Z DEBUG tokio-runtime-worker ThreadId(03) shadowsocks::net::sys: crates/shadowsocks/src/net/sys/mod.rs:165: IpStackCapability support_ipv6=true
2024-06-16T23:01:19.56980524Z DEBUG tokio-runtime-worker ThreadId(03) shadowsocks::net::sys: crates/shadowsocks/src/net/sys/mod.rs:173: IpStackCapability support_ipv4_mapped_ipv6=true
2024-06-16T23:01:19.569994091Z TRACE tokio-runtime-worker ThreadId(03) mio::poll: /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/mio-0.8.11/src/poll.rs:551: registering event source with poller: token=Token(88188202799488), interests=READABLE | WRITABLE
2024-06-16T23:01:19.578334536Z WARN tokio-runtime-worker ThreadId(03) shadowsocks_service::local::net::udp::association: crates/shadowsocks-service/src/local/net/udp/association.rs:620: udp failed to send back 68 bytes to client 10.0.2.25:13870, from target 3.132.228.249:3478 (proxied), error: Invalid argument (os error 22)
Replace it with the method just like the one on stackoverflow. Please try again. @ge9
Still not working...
2024-06-16T23:53:16.161455017Z TRACE tokio-runtime-worker ThreadId(02) shadowsocks_service::local::redir::udprelay: crates/shadowsocks-service/src/local/redir/udprelay/mod.rs:285: received UDP packet from 10.0.2.25:34707, destination 3.132.228.249:3478, length 28 bytes
2024-06-16T23:53:16.161934966Z DEBUG tokio-runtime-worker ThreadId(02) shadowsocks_service::local::net::udp::association: crates/shadowsocks-service/src/local/net/udp/association.rs:123: created udp association for 10.0.2.25:34707
2024-06-16T23:53:16.162531969Z TRACE tokio-runtime-worker ThreadId(02) shadowsocks_service::local::net::udp::association: crates/shadowsocks-service/src/local/net/udp/association.rs:433: udp relay 10.0.2.25:34707 -> 3.132.228.249:3478 (proxied) with 28 bytes
2024-06-16T23:53:16.162692883Z TRACE tokio-runtime-worker ThreadId(02) mio::poll: /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/mio-0.8.11/src/poll.rs:551: registering event source with poller: token=Token(51249433133440), interests=READABLE | WRITABLE
2024-06-16T23:53:16.162828655Z TRACE tokio-runtime-worker ThreadId(02) shadowsocks::relay::udprelay::proxy_socket: crates/shadowsocks/src/relay/udprelay/proxy_socket.rs:103: connected udp remote 192.168.1.212:1080 (outbound: 192.168.1.212:1080) with ConnectOpts { user_cookie: None, bind_local_addr: None, bind_interface: None, tcp: TcpSocketOpts { send_buffer_size: None, recv_buffer_size: None, nodelay: false, fastopen: false, keepalive: Some(15s), mptcp: false }, udp: UdpSocketOpts { mtu: None } }
2024-06-16T23:53:16.16302868Z TRACE tokio-runtime-worker ThreadId(02) shadowsocks::relay::udprelay::proxy_socket: crates/shadowsocks/src/relay/udprelay/proxy_socket.rs:235: UDP server client send to 3.132.228.249:3478, control: UdpSocketControlData { client_session_id: 17626901403037311660, server_session_id: 0, packet_id: 1, user: None }, payload length 28 bytes, packet length 35 bytes
2024-06-16T23:53:16.356517518Z TRACE tokio-runtime-worker ThreadId(02) shadowsocks::relay::udprelay::proxy_socket: crates/shadowsocks/src/relay/udprelay/proxy_socket.rs:445: UDP server client receive from 3.132.228.249:3478, control: None, packet length 75 bytes, payload length 68 bytes
2024-06-16T23:53:16.357477975Z TRACE tokio-runtime-worker ThreadId(02) shadowsocks_service::local::net::udp::association: crates/shadowsocks-service/src/local/net/udp/association.rs:607: udp relay 10.0.2.25:34707 <- 3.132.228.249:3478 (proxied) received 68 bytes
2024-06-16T23:53:16.358022457Z DEBUG tokio-runtime-worker ThreadId(02) shadowsocks::net::sys: crates/shadowsocks/src/net/sys/mod.rs:155: IpStackCapability support_ipv4=true
2024-06-16T23:53:16.358794623Z DEBUG tokio-runtime-worker ThreadId(02) shadowsocks::net::sys: crates/shadowsocks/src/net/sys/mod.rs:164: IpStackCapability support_ipv6=true
2024-06-16T23:53:16.359158356Z DEBUG tokio-runtime-worker ThreadId(02) shadowsocks::net::sys: crates/shadowsocks/src/net/sys/mod.rs:172: IpStackCapability support_ipv4_mapped_ipv6=true
2024-06-16T23:53:16.361218674Z TRACE tokio-runtime-worker ThreadId(02) mio::poll: /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/mio-0.8.11/src/poll.rs:551: registering event source with poller: token=Token(51249433133696), interests=READABLE | WRITABLE
2024-06-16T23:53:16.362913582Z WARN tokio-runtime-worker ThreadId(02) shadowsocks_service::local::net::udp::association: crates/shadowsocks-service/src/local/net/udp/association.rs:620: udp failed to send back 68 bytes to client 10.0.2.25:34707, from target 3.132.228.249:3478 (proxied), error: Invalid argument (os error 22)
Udp Socket bind([::ffff:3.132.228.249]:3478) was Ok, but then sendmsg([::ffff:10.0.2.25]:34707) failed?
Is that a bug in FreeBSD? :(
Further more, in the test code, sockets bind([::ffff:127.0.0.1]:0) and connect([::ffff:127.0.0.1]:53) were all working well.
For TCP socket, bind([::ffff:127.0.0.1]:0) and then listen() Ok. For client socket, bind([::ffff:127.0.0.1]:0) and then connect() to the server socket Ok.
Getting Headache.
(edited)
By the way, I forgot to test TCP. With ipfw, --udp-redir pf is fine, but --tcp-redir pf did not work. And TCP seems not affected by this IPv4-mapped address issue (it works fine with ipfw and --tcp-redir ipfw).
UDP works with pf but TCP doesn't? That wasn't expected.
It's about testing with ipfw, not pf.
Ok. Please also test the TCP with ipfw. Let's fix all of them together.