celzero
feature request: implement Endpoint-Independent Mapping for UDP
I know two methods:
- use unconnected UDP sockets
- bind multiple UDP sockets to one address
This mapping behavior makes NAT traversal between two rethink users possible.
Firestack have to give up its reliance on gonet.UDPConn (which handles outgoing unconnected UDP sockets just fine).
This is not going to be trivial though a few projects like SagerNet did once implement this (unsure if it worked as they don't seem to use it anymore).
https://github.com/SagerNet/LibSagerNetCore/blob/1fce969ea5/gvisor/udp.go (old) / https://github.com/SagerNet/sing-tun/blob/aecfc190f4/stack_gvisor_udp.go (new).
Welcome PR, for sure. :D
Also: https://github.com/xjasonlyu/tun2socks/issues/177
bind multiple UDP sockets to one address
Curious, how does this work?
Set two sockopt, SO_REUSEADDR and SO_REUSEPORT.
Firestack have to give up its reliance on gonet.UDPConn
Why? I am not familiar with gvisor, but its document claims gonet.DialUDP() can create unconnected UDPConn.
https://pkg.go.dev/gvisor.dev/[email protected]/pkg/tcpip/adapters/gonet#DialUDP
The problem isn't port forwarding (if I may call it that) using gonet.DialUDP, the problem is, the protocol handlers firestack sets up (for TCP and UDP) are not called for unconnected UDP (and listening TCP) sockets.
Firestack have to give up its reliance on gonet.UDPConn (which handles outgoing unconnected UDP sockets just fine).
Don't think even the SagerNet code I shared above would work. In that,
-
Either: Android does not forward packets from unconnected sockets to the VPN tunnel:
Testing this may be easy: Listening on
unspecifiedIP:port(over both UDP, TCP) using nc+Termux, say, would not generate any relevant logs in firestack (in Rethink, Configure -> Settings -> Log level must be set to Very Verbose). - Or: firestack must handle UDP on its own and not rely on netstack at all. That is, instead of delivering the UDP packet (injecting it in) to netstack: https://github.com/celzero/firestack/blob/7ea0539e57ba513065dcde893084f70cfedd74b9/intra/netstack/fdbased.go#L505 ... deal with it as-is.
Android does not forward packets from unconnected sockets to the VPN tunnel
This is incorrect. nc -u -l -p 12345 does not send UDP packets until netcat receives one UDP packet.
This is incorrect. nc -u -l -p 12345 does not send UDP packets until netcat receives one UDP packet.
Is there a way, then, for firestack to know if such a listener was started, so it can start an egress listener?
Also, I guess, I am at a loss as to what an "endpoint-independent mapping for UDP" would look like... Does it mean, instead of dialing to an (remote) endpoint, ^0 always announce UDP instead ^1?
What could we do for TCP [^2]? Port forwarding?
[^2]: Proxies (severs) currently have to be excluded for them to work: https://github.com/celzero/rethink-app/issues/1399
Is there a way, then, for firestack to know if such a listener was started, so it can start an egress listener?
On Linux you can use inotify API to monitor /proc/net/udp and /proc/net/tcp, without some permissions this method may not work on Android.
Also, I guess, I am at a loss as to what an "endpoint-independent mapping for UDP" would look like... Does it mean, instead of dialing to an (remote) endpoint, 1 always announce UDP instead 2?
Yes.
What could we do for TCP 3? Port forwarding?
Footnotes
1. https://github.com/celzero/firestack/blob/7ea0539e57ba513065dcde893084f70cfedd74b9/intra/udp.go#L352 [↩](#user-content-fnref-0-044726a13dad48d457abd69cf7035293) 2. https://github.com/celzero/firestack/blob/7ea0539e57ba513065dcde893084f70cfedd74b9/intra/udp.go#L345 [↩](#user-content-fnref-1-044726a13dad48d457abd69cf7035293) 3. Proxies (severs) currently _have_ to be excluded for them to work: https://github.com/celzero/rethink-app/issues/1399 [↩](#user-content-fnref-2-044726a13dad48d457abd69cf7035293)
Allow users to write some port forwarding rules? ( e.g. TCP 0.0.0.0:12346 -> 127.0.0.1:12345 )
Yes.
Guess, Proxy.Announce would need to be paired with gonet.DialUDP instead of gonet.NewUDPConn?
https://github.com/celzero/firestack/blob/7ea0539e57ba513065dcde893084f70cfedd74b9/intra/netstack/udp.go#L149
Linux ... inotify ... procfs
I guess Netlink cmds might work too (would be surprised if they worked on Android but it might).
Allow users to write some port forwarding rules? (TCP 0.0.0.0:12346 -> 127.0.0.1:12345)
SagerNet seems to be doing some stuff with this, but I am not sure exactly what it does:
- (multicast filter) https://github.com/SagerNet/sing-tun/commit/150b1162316cb63bbbf5c3c01f3a2b494a56e72d
- (broadcast filter) https://github.com/SagerNet/sing-tun/commit/da350ecc726daa502b4256a7a70d70a4c6cde6d4
Guess,
Proxy.Announcewould need to be paired withgonet.DialUDPinstead ofgonet.NewUDPConn?https://github.com/celzero/firestack/blob/7ea0539e57ba513065dcde893084f70cfedd74b9/intra/netstack/udp.go#L149
gonet.DialUDP() calls gonet.NewUDPConn() internally.
See https://github.com/google/gvisor/blob/4542eb5ba62b/pkg/tcpip/adapters/gonet/gonet.go#L588
I guess Netlink cmds might work too (would be surprised if they worked on Android but it might).
Ah, I think starting an "egress listener" automatically is not that useful, because the setsockopt trick I mentioned doesn't work if two processes don't have the same effective UID.
Allow users to write some port forwarding rules? (TCP 0.0.0.0:12346 -> 127.0.0.1:12345)
SagerNet seems to be doing some stuff with this, but I am not sure exactly what it does:
* (multicast filter) [SagerNet/sing-tun@150b116](https://github.com/SagerNet/sing-tun/commit/150b1162316cb63bbbf5c3c01f3a2b494a56e72d) * (broadcast filter) [SagerNet/sing-tun@da350ec](https://github.com/SagerNet/sing-tun/commit/da350ecc726daa502b4256a7a70d70a4c6cde6d4)
I guess these two commits allow packets having multicast or broadcast destination address to bypass the userspace netstack.
A off topic question: Why does firestack send SYN+ACK packets immediately after receiving SYN packets?
gonet.DialUDP() calls gonet.NewUDPConn() internally. See: https://github.com/google/gvisor/blob/4542eb5ba62b/pkg/tcpip/adapters/gonet/gonet.go#L588
I think, firestack needs to create a new gonet.UDPConn without using udp.ForwarderRequest.CreateEndpoint + gonet.NewUDPConn (as that makes a connected UDP socket, per docs):
https://github.com/celzero/firestack/blob/7ea0539e57ba513065dcde893084f70cfedd74b9/intra/netstack/udp.go#L143
think starting an "egress listener" automatically is not that useful, because the setsockopt trick I mentioned doesn't work if two processes
Believe apps part of the VPN tunnel in Android are in a different network namespace and as such must not be able to stomp on the VPN app's sockets?
Using the other option (unconnected UDP over sockopt+reuse) the VPN app NATs the traffic from its listening port (bound to actual underlying network) onto the same port bound to the TUN device, if any. Which is the original proposal, that would work, would it not?
Why does firestack send SYN+ACK packets immediately after receiving SYN packets
netstack's gonet API requires for its internal book-keeping (that is, routing subsequent packets from the same TCP flow to an already established gonet.TCPConn).
https://github.com/celzero/firestack/blob/7ea0539e57ba513065dcde893084f70cfedd74b9/intra/netstack/tcp.go#L116-L117
Does this break anything apart from confusing apps into thinking they've connected to remote while they may infact be blocked later due to a firewall rule?
think starting an "egress listener" automatically is not that useful, because the setsockopt trick I mentioned doesn't work if two processes
Believe apps part of the VPN tunnel in Android are in a different network namespace and as such must not be able to stomp on the VPN app's sockets?
Using the other option (unconnected UDP over sockopt+reuse) the VPN app NATs the traffic from its listening port (bound to actual underlying network) onto the same port bound to the TUN device, if any. Which is the original proposal, that would work, would it not?
I don't know if apps on android are in different namespaces, 😂 but on android every app has its own user, you can check with adb shell sh -c "ps -d|cat -" . The reason why I mentioned the setsockopt trick a few days ago is just I thought you might not like unconnected sockets.
Please read https://man.voidlinux.org/man7/socket.7#SO_REUSEPORT
Thanks. Now, I recall reading a Cloudflare blog post about so_reuseport (mirror).
Possibly the biggest downside, though, is that a service listening on the wildcard
INADDR_ANYaddress claims the port number exclusively for itself. Binding over the wildcard-listening socket with a specific IP and port fails miserably due to the address already being taken (EADDRINUSE).bind(3, {sa_family=AF_INET, sin_port=htons(12345), sin_addr=inet_addr("0.0.0.0")}, 16) = 0 bind(4, {sa_family=AF_INET, sin_port=htons(12345), sin_addr=inet_addr("127.0.0.1")}, 16) = -1 EADDRINUSE (Address already in use)Unless your service is UDP-only, setting the
SO_REUSEADDRsocket option, will not help you overcome this restriction. The only way out is to turn toSO_REUSEPORT, normally used to construct a load-balancing socket group. And that is only if you are lucky enough to run the port-conflicting services as the same user (UID). That is a story for another post.
The reason why I mentioned the setsockopt trick a few days ago is just I thought you might not like unconnected sockets.
I don't like (or understand much of) networking in general, but here we are ;)
(but no, I don't hold any special grudge on unconnected sockets)
I don't know if apps on android are in different namespaces
I am not sure, either (though this must be easy to find out...). Even so, the apps (by default) can't really bind to underlying interfaces (wifi, mobile, usb, etc) and so them listening on a wildcard addr:port (say tcp/udp [::]:8080) means listening on (tcp/udp) packets sent to 8080 to the TUN device only? I take what you're saying is, despite that, the Kernel wouldn't let both (the VPN app and the installed app) bind to the same port (8080, in this case) regardless of routing rules (as the effective UIDs are different), yeah?.
Btw, checking if you working on this change (so we both don't end up impl it)?
I am not sure, either (though this must be easy to find out...). Even so, the apps (by default) can't really bind to underlying interfaces (wifi, mobile, usb, etc) and so them listening on a wildcard addr:port (say tcp/udp
[::]:8080) means listening on (tcp/udp) packets sent to8080to the TUN device only?
Not only the TUN device, they can also receive packets sent to [::1]:8080 and [ the Link-Local Address of dummy0 ]:8080.
I take what you're saying is, despite that, the Kernel wouldn't let both (the VPN app and the installed app) bind to the same port (
8080, in this case) regardless of routing rules (as the effective UIDs are different), yeah?.
Try this:
package main
import (
"flag"
"fmt"
"syscall"
"time"
)
const SO_REUSEPORT = 15
func p(e error) {
if e != nil {
panic(e)
}
}
func main() {
var localAddr syscall.SockaddrInet6
flag.IntVar(&localAddr.Port, "listen-port", 65535, "")
flag.Parse()
tcpSock, err := syscall.Socket(syscall.AF_INET6, syscall.SOCK_STREAM, syscall.IPPROTO_TCP)
p(err)
defer syscall.Close(tcpSock)
err = syscall.SetsockoptInt(tcpSock, syscall.SOL_SOCKET, syscall.SO_REUSEADDR, 1)
p(err)
err = syscall.SetsockoptInt(tcpSock, syscall.SOL_SOCKET, SO_REUSEPORT, 1)
p(err)
err = syscall.Bind(tcpSock, &localAddr)
p(err)
err = syscall.Listen(tcpSock, 8)
p(err)
nfd, from, err := syscall.Accept(tcpSock)
p(err)
defer syscall.Close(nfd)
fmt.Printf("accept: nfd: %v ; from: %v\n", nfd, from)
fmt.Println("sleeping...")
time.Sleep(600 * time.Second)
}
Btw, checking if you working on this change (so we both don't end up impl it)?
I am not working on this change.
Not only the TUN device, they can also receive packets sent to [::1]:8080 and [ the Link-Local Address of dummy0 ]:8080.
On Android, unsure if link-local gets sent to TUN (lo isn't, Wifi Calling isn't, Tethering isn't).
f78f98d522be5fc4 is demuxed to 1.1.1.1, 8.8.8.8, 9.9.9.9, and 8.8.4.4 endpoints from :37288.
2024-08-30 21:32:07.422 28538-2688 GoLog com.celzero.bravedns I common.go:412: D onFlow: udp noalg? false or hasips? false
2024-08-30 21:32:07.422 28538-2688 GoLog com.celzero.bravedns I common.go:416: D onFlow: udp no realips() or domains( + ), for src=10.111.222.1:40413 dst=8.8.4.4:53
2024-08-30 21:32:07.423 28538-2697 GoLog com.celzero.bravedns I common.go:412: D onFlow: udp noalg? false or hasips? false
2024-08-30 21:32:07.423 28538-2697 GoLog com.celzero.bravedns I common.go:416: D onFlow: udp no realips() or domains( + ), for src=10.111.222.1:40413 dst=8.8.8.8:53
2024-08-30 21:32:07.423 28538-2697 GoLog com.celzero.bravedns I common.go:412: D onFlow: udp noalg? false or hasips? false
2024-08-30 21:32:07.423 28538-2697 GoLog com.celzero.bravedns I common.go:416: D onFlow: udp no realips() or domains( + ), for src=10.111.222.1:40413 dst=1.1.1.1:53
2024-08-30 21:32:07.426 28538-2697 GoLog com.celzero.bravedns I common.go:412: D onFlow: udp noalg? false or hasips? false
2024-08-30 21:32:07.426 28538-2692 GoLog com.celzero.bravedns I common.go:416: D onFlow: udp no realips() or domains( + ), for src=10.111.222.1:40413 dst=9.9.9.9:53
2024-08-30 21:32:07.447 28538-2692 GoLog com.celzero.bravedns I protect.go:67: D control: netbinder: Exit: udp4(0.0.0.0:0); err? <nil>
2024-08-30 21:32:07.450 28538-2692 GoLog com.celzero.bravedns I udpmux.go:497: I udp: mux: f78f98d522be5fc4 new assoc for 10.111.222.1:40413
2024-08-30 21:32:07.450 28538-2697 GoLog com.celzero.bravedns I udpmux.go:265: I udp: mux: f78f98d522be5fc4 route: new for 8.8.8.8:53; stats: &{0 {13955697707408456447 1307458265908 532290359584} {{} 1} {{} 0} {{} 0}}
2024-08-30 21:32:07.450 28538-2697 GoLog com.celzero.bravedns I udpmux.go:125: D udp: mux: f78f98d522be5fc4 awaiter: watching 0.0.0.0:37288 => 8.8.8.8:53
2024-08-30 21:32:07.451 28538-3152 GoLog com.celzero.bravedns E udp.go:190: E ns: udp: dial: endpoint for 10.111.222.1:40413 => 9.9.9.9:53; err(connect udp 10.111.222.1:40413: port is in use)
2024-08-30 21:32:07.451 28538-3152 GoLog com.celzero.bravedns E udp.go:147: E ns: udp: demuxer: dial: connect udp 10.111.222.1:40413: port is in use; src(10.111.222.1:40413) dst(9.9.9.9:53)
2024-08-30 21:32:07.451 28538-3152 GoLog com.celzero.bravedns E udpmux.go:261: E udp: mux: f78f98d522be5fc4 route: vend failure 9.9.9.9:53; err connect udp 10.111.222.1:40413: port is in use
2024-08-30 21:32:07.451 28538-3152 GoLog com.celzero.bravedns E udp.go:190: E ns: udp: dial: endpoint for 10.111.222.1:40413 => 1.1.1.1:53; err(connect udp 10.111.222.1:40413: port is in use)
2024-08-30 21:32:07.451 28538-3152 GoLog com.celzero.bravedns E udp.go:147: E ns: udp: demuxer: dial: connect udp 10.111.222.1:40413: port is in use; src(10.111.222.1:40413) dst(1.1.1.1:53)
2024-08-30 21:32:07.452 28538-2695 GoLog com.celzero.bravedns E udp.go:140: W udp: proxy: 10.111.222.1:40413 -> 9.9.9.9:53; err connect udp 10.111.222.1:40413: port is in use
2024-08-30 21:32:07.452 28538-2695 GoLog com.celzero.bravedns E udpmux.go:261: E udp: mux: f78f98d522be5fc4 route: vend failure 1.1.1.1:53; err connect udp 10.111.222.1:40413: port is in use
2024-08-30 21:32:07.452 28538-2692 GoLog com.celzero.bravedns E udp.go:190: E ns: udp: dial: endpoint for 10.111.222.1:40413 => 8.8.4.4:53; err(connect udp 10.111.222.1:40413: port is in use)
2024-08-30 21:32:07.452 28538-2692 GoLog com.celzero.bravedns E udp.go:147: E ns: udp: demuxer: dial: connect udp 10.111.222.1:40413: port is in use; src(10.111.222.1:40413) dst(8.8.4.4:53)
2024-08-30 21:32:07.452 28538-2692 GoLog com.celzero.bravedns E udp.go:140: W udp: proxy: 10.111.222.1:40413 -> 8.8.4.4:53; err connect udp 10.111.222.1:40413: port is in use
2024-08-30 21:32:07.452 28538-2692 GoLog com.celzero.bravedns E udpmux.go:261: E udp: mux: f78f98d522be5fc4 route: vend failure 8.8.4.4:53; err connect udp 10.111.222.1:40413: port is in use
2024-08-30 21:32:07.453 28538-3095 GoLog com.celzero.bravedns E udp.go:140: W udp: proxy: 10.111.222.1:40413 -> 1.1.1.1:53; err connect udp 10.111.222.1:40413: port is in use
2024-08-30 21:32:07.453 28538-2696 GoLog com.celzero.bravedns I udp.go:315: I udp: connect: f78f98d522be5fc4 (proxy? [email protected]:1337) 0.0.0.0:37288 -> 8.8.8.8:53/8.8.8.8:53; mux? true, uid 10671
2024-08-30 21:32:07.453 28538-2696 GoLog com.celzero.bravedns I udpmux.go:265: I udp: mux: f78f98d522be5fc4 route: new for 9.9.9.9:53; stats: &{0 {13955697707408456447 1307458265908 532290359584} {{} 2} {{} 0} {{} 0}}
2024-08-30 21:32:07.453 28538-2696 GoLog com.celzero.bravedns I udp.go:315: I udp: connect: 20790a938f64d070 (proxy? [email protected]:1337) 0.0.0.0:37288 -> 9.9.9.9:53/9.9.9.9:53; mux? true, uid 10671
2024-08-30 21:32:07.454 28538-2696 GoLog com.celzero.bravedns I udpmux.go:125: D udp: mux: f78f98d522be5fc4 awaiter: watching 0.0.0.0:37288 => 9.9.9.9:53
2024-08-30 21:32:07.454 28538-2696 GoLog com.celzero.bravedns I udpmux.go:265: I udp: mux: f78f98d522be5fc4 route: new for 1.1.1.1:53; stats: &{0 {13955697707408456447 1307458265908 532290359584} {{} 3} {{} 0} {{} 61}}
2024-08-30 21:32:07.454 28538-2696 GoLog com.celzero.bravedns I udp.go:315: I udp: connect: 5c44094c5361f93f (proxy? [email protected]:1337) 0.0.0.0:37288 -> 1.1.1.1:53/1.1.1.1:53; mux? true, uid 10671
2024-08-30 21:32:07.454 28538-2696 GoLog com.celzero.bravedns I udpmux.go:374: D udp: mux: f78f98d522be5fc4 demux 0.0.0.0:37288 => 9.9.9.9:53 close, in: 0, over: 0
2024-08-30 21:32:07.454 28538-2696 GoLog com.celzero.bravedns I udpmux.go:125: D udp: mux: f78f98d522be5fc4 awaiter: watching 0.0.0.0:37288 => 1.1.1.1:53
2024-08-30 21:32:07.454 28538-2696 GoLog com.celzero.bravedns I udpmux.go:387: I udp: mux: f78f98d522be5fc4 demux from 0.0.0.0:37288 => 9.9.9.9:53 closed
2024-08-30 21:32:07.454 28538-2696 GoLog com.celzero.bravedns I udpmux.go:374: D udp: mux: f78f98d522be5fc4 demux 0.0.0.0:37288 => 9.9.9.9:53 close, in: 0, over: 0
2024-08-30 21:32:07.454 28538-2696 GoLog com.celzero.bravedns I udpmux.go:374: D udp: mux: f78f98d522be5fc4 demux 0.0.0.0:37288 => 1.1.1.1:53 close, in: 0, over: 0
2024-08-30 21:32:07.454 28538-2696 GoLog com.celzero.bravedns I udpmux.go:387: I udp: mux: f78f98d522be5fc4 demux from 0.0.0.0:37288 => 1.1.1.1:53 closed
2024-08-30 21:32:07.454 28538-2696 GoLog com.celzero.bravedns I udpmux.go:265: I udp: mux: f78f98d522be5fc4 route: new for 8.8.4.4:53; stats: &{0 {13955697707408456447 1307458265908 532290359584} {{} 4} {{} 0} {{} 183}}
2024-08-30 21:32:07.454 28538-2696 GoLog com.celzero.bravedns I udpmux.go:374: D udp: mux: f78f98d522be5fc4 demux 0.0.0.0:37288 => 9.9.9.9:53 close, in: 0, over: 0
2024-08-30 21:32:07.454 28538-2696 GoLog com.celzero.bravedns I udp.go:315: I udp: connect: acbd3729b62afe4f (proxy? [email protected]:1337) 0.0.0.0:37288 -> 8.8.4.4:53/8.8.4.4:53; mux? true, uid 10671
2024-08-30 21:32:07.454 28538-2696 GoLog com.celzero.bravedns I udpmux.go:276: I udp: mux: f78f98d522be5fc4 unrouting... 0.0.0.0:37288 => 1.1.1.1:53
2024-08-30 21:32:07.454 28538-2696 GoLog com.celzero.bravedns I common.go:412: D onFlow: udp noalg? false or hasips? false
2024-08-30 21:32:07.454 28538-2696 GoLog com.celzero.bravedns I udpmux.go:125: D udp: mux: f78f98d522be5fc4 awaiter: watching 0.0.0.0:37288 => 8.8.4.4:53
2024-08-30 21:32:07.454 28538-2696 GoLog com.celzero.bravedns I udpmux.go:374: D udp: mux: f78f98d522be5fc4 demux 0.0.0.0:37288 => 1.1.1.1:53 close, in: 0, over: 0
2024-08-30 21:32:07.454 28538-2696 GoLog com.celzero.bravedns I udpmux.go:374: D udp: mux: f78f98d522be5fc4 demux 0.0.0.0:37288 => 8.8.4.4:53 close, in: 0, over: 0
2024-08-30 21:32:07.454 28538-2696 GoLog com.celzero.bravedns I udpmux.go:374: D udp: mux: f78f98d522be5fc4 demux 0.0.0.0:37288 => 9.9.9.9:53 close, in: 0, over: 0
2024-08-30 21:32:07.454 28538-2696 GoLog com.celzero.bravedns I udpmux.go:387: I udp: mux: f78f98d522be5fc4 demux from 0.0.0.0:37288 => 8.8.4.4:53 closed
2024-08-30 21:32:07.454 28538-2696 GoLog com.celzero.bravedns I udpmux.go:276: I udp: mux: f78f98d522be5fc4 unrouting... 0.0.0.0:37288 => 8.8.4.4:53
2024-08-30 21:32:07.454 28538-2696 GoLog com.celzero.bravedns I common.go:412: D onFlow: udp noalg? false or hasips? false
2024-08-30 21:32:07.454 28538-2696 GoLog com.celzero.bravedns I common.go:416: D onFlow: udp no realips() or domains( + ), for src=10.111.222.1:40413 dst=8.8.4.4:53
2024-08-30 21:32:07.454 28538-2696 GoLog com.celzero.bravedns I udpmux.go:374: D udp: mux: f78f98d522be5fc4 demux 0.0.0.0:37288 => 1.1.1.1:53 close, in: 0, over: 0
2024-08-30 21:32:07.454 28538-2696 GoLog com.celzero.bravedns I udpmux.go:374: D udp: mux: f78f98d522be5fc4 demux 0.0.0.0:37288 => 1.1.1.1:53 close, in: 0, over: 0
2024-08-30 21:32:07.454 28538-2696 GoLog com.celzero.bravedns I common.go:416: D onFlow: udp no realips() or domains( + ), for src=10.111.222.1:40413 dst=9.9.9.9:53
2024-08-30 21:32:07.454 28538-2696 GoLog com.celzero.bravedns I udpmux.go:374: D udp: mux: f78f98d522be5fc4 demux 0.0.0.0:37288 => 8.8.4.4:53 close, in: 0, over: 0
2024-08-30 21:32:07.454 28538-2696 GoLog com.celzero.bravedns I udpmux.go:276: I udp: mux: f78f98d522be5fc4 unrouting... 0.0.0.0:37288 => 9.9.9.9:53
2024-08-30 21:32:07.454 28538-2696 GoLog com.celzero.bravedns I udpmux.go:374: D udp: mux: f78f98d522be5fc4 demux 0.0.0.0:37288 => 8.8.4.4:53 close, in: 0, over: 0
2024-08-30 21:32:07.454 28538-2696 GoLog com.celzero.bravedns I udpmux.go:374: D udp: mux: f78f98d522be5fc4 demux 0.0.0.0:37288 => 8.8.4.4:53 close, in: 0, over: 0
2024-08-30 21:32:07.454 28538-2696 GoLog com.celzero.bravedns I common.go:412: D onFlow: udp noalg? false or hasips? false
2024-08-30 21:32:07.454 28538-2696 GoLog com.celzero.bravedns I common.go:416: D onFlow: udp no realips() or domains( + ), for src=10.111.222.1:40413 dst=1.1.1.1:53
2024-08-30 21:32:07.506 28538-2692 GoLog com.celzero.bravedns I udpmux.go:455: D udp: mux: f78f98d522be5fc4 demux: read: done(sz: 132)
2024-08-30 21:32:07.507 28538-3095 GoLog com.celzero.bravedns I udpmux.go:265: I udp: mux: f78f98d522be5fc4 route: new for 1.1.1.1:53; stats: &{0 {13955697707408456447 1307458265908 532290359584} {{} 5} {{} 264} {{} 183}}
2024-08-30 21:32:07.507 28538-3095 GoLog com.celzero.bravedns I udpmux.go:125: D udp: mux: f78f98d522be5fc4 awaiter: watching 0.0.0.0:37288 => 1.1.1.1:53
2024-08-30 21:32:07.508 28538-3095 GoLog com.celzero.bravedns I common.go:412: D onFlow: udp noalg? false or hasips? false
2024-08-30 21:32:07.508 28538-3095 GoLog com.celzero.bravedns I common.go:416: D onFlow: udp no realips() or domains( + ), for src=10.111.222.1:40413 dst=1.1.1.1:53
2024-08-30 21:32:07.511 28538-2692 GoLog com.celzero.bravedns I udpmux.go:265: I udp: mux: f78f98d522be5fc4 route: new for 9.9.9.9:53; stats: &{0 {13955697707408456447 1307458265908 532290359584} {{} 6} {{} 396} {{} 183}}
2024-08-30 21:32:07.511 28538-2692 GoLog com.celzero.bravedns I common.go:412: D onFlow: udp noalg? false or hasips? false
2024-08-30 21:32:07.511 28538-2692 GoLog com.celzero.bravedns I common.go:416: D onFlow: udp no realips() or domains( + ), for src=10.111.222.1:40413 dst=9.9.9.9:53
2024-08-30 21:32:07.512 28538-2697 GoLog com.celzero.bravedns I udpmux.go:125: D udp: mux: f78f98d522be5fc4 awaiter: watching 0.0.0.0:37288 => 9.9.9.9:53
2024-08-30 21:32:07.525 28538-3095 GoLog com.celzero.bravedns I protect.go:67: D control: netbinder: Exit: udp4(9.9.9.9:53); err? <nil>
2024-08-30 21:32:07.527 28538-2692 GoLog com.celzero.bravedns I protect.go:67: D control: netbinder: Exit: udp4(1.1.1.1:53); err? <nil>
2024-08-30 21:32:07.529 28538-2692 GoLog com.celzero.bravedns I exit.go:53: I proxy: exit: dial(udp) to 1.1.1.1:53; err? <nil>
2024-08-30 21:32:07.529 28538-2692 GoLog com.celzero.bravedns I udp.go:315: I udp: connect: 0c296b22bbd49481 (proxy? [email protected]:1337) 10.1.204.164:42507 -> 1.1.1.1:53/1.1.1.1:53; mux? false, uid 10671
2024-08-30 21:32:07.529 28538-2692 GoLog com.celzero.bravedns I exit.go:53: I proxy: exit: dial(udp) to 9.9.9.9:53; err? <nil>
2024-08-30 21:32:07.530 28538-2695 GoLog com.celzero.bravedns I udp.go:315: I udp: connect: 1f41c4b4d08808b3 (proxy? [email protected]:1337) 10.1.204.164:45931 -> 9.9.9.9:53/9.9.9.9:53; mux? false, uid 10671
2024-08-30 21:32:08.522 28538-2697 GoLog com.celzero.bravedns I common.go:412: D onFlow: udp noalg? false or hasips? false
2024-08-30 21:32:08.522 28538-2697 GoLog com.celzero.bravedns I common.go:416: D onFlow: udp no realips() or domains( + ), for src=10.111.222.1:40413 dst=8.8.4.4:53
2024-08-30 21:32:08.545 28538-2695 GoLog com.celzero.bravedns I udpmux.go:265: I udp: mux: f78f98d522be5fc4 route: new for 8.8.4.4:53; stats: &{0 {13955697707408456447 1307458265908 532290359584} {{} 7} {{} 396} {{} 244}}
2024-08-30 21:32:08.545 28538-2695 GoLog com.celzero.bravedns I udpmux.go:125: D udp: mux: f78f98d522be5fc4 awaiter: watching 0.0.0.0:37288 => 8.8.4.4:53
2024-08-30 21:32:08.545 28538-2695 GoLog com.celzero.bravedns I udp.go:315: I udp: connect: 0b2a4b8786a59117 (proxy? [email protected]:1337) 0.0.0.0:37288 -> 8.8.4.4:53/8.8.4.4:53; mux? true, uid 10671
2024-08-30 21:32:08.545 28538-2695 GoLog com.celzero.bravedns E udp.go:190: E ns: udp: dial: endpoint for 10.111.222.1:40413 => 8.8.4.4:53; err(connect udp 10.111.222.1:40413: port is in use)
2024-08-30 21:32:08.546 28538-2695 GoLog com.celzero.bravedns E udp.go:147: E ns: udp: demuxer: dial: connect udp 10.111.222.1:40413: port is in use; src(10.111.222.1:40413) dst(8.8.4.4:53)
2024-08-30 21:32:08.546 28538-2695 GoLog com.celzero.bravedns I udpmux.go:374: D udp: mux: f78f98d522be5fc4 demux 0.0.0.0:37288 => 8.8.4.4:53 close, in: 0, over: 0
2024-08-30 21:32:08.546 28538-2695 GoLog com.celzero.bravedns I udpmux.go:387: I udp: mux: f78f98d522be5fc4 demux from 0.0.0.0:37288 => 8.8.4.4:53 closed
2024-08-30 21:32:08.546 28538-2695 GoLog com.celzero.bravedns E udpmux.go:261: E udp: mux: f78f98d522be5fc4 route: vend failure 8.8.4.4:53; err connect udp 10.111.222.1:40413: port is in use
2024-08-30 21:32:08.546 28538-2695 GoLog com.celzero.bravedns E udp.go:140: W udp: proxy: 10.111.222.1:40413 -> 8.8.4.4:53; err connect udp 10.111.222.1:40413: port is in use
2024-08-30 21:32:08.546 28538-2695 GoLog com.celzero.bravedns I udpmux.go:276: I udp: mux: f78f98d522be5fc4 unrouting... 0.0.0.0:37288 => 8.8.4.4:53
2024-08-30 21:32:08.546 28538-2695 GoLog com.celzero.bravedns I common.go:412: D onFlow: udp noalg? false or hasips? false
2024-08-30 21:32:08.546 28538-2695 GoLog com.celzero.bravedns I common.go:416: D onFlow: udp no realips() or domains( + ), for src=10.111.222.1:40413 dst=8.8.4.4:53
2024-08-30 21:32:08.546 28538-2688 GoLog com.celzero.bravedns I udpmux.go:374: D udp: mux: f78f98d522be5fc4 demux 0.0.0.0:37288 => 8.8.4.4:53 close, in: 0, over: 0
2024-08-30 21:32:08.546 28538-2688 GoLog com.celzero.bravedns I udpmux.go:374: D udp: mux: f78f98d522be5fc4 demux 0.0.0.0:37288 => 8.8.4.4:53 close, in: 0, over: 0
2024-08-30 21:32:08.546 28538-3107 GoLog com.celzero.bravedns I udpmux.go:374: D udp: mux: f78f98d522be5fc4 demux 0.0.0.0:37288 => 8.8.4.4:53 close, in: 0, over: 0
2024-08-30 21:32:08.574 28538-2695 GoLog com.celzero.bravedns I udpmux.go:455: D udp: mux: f78f98d522be5fc4 demux: read: done(sz: 132)
2024-08-30 21:32:08.589 28538-2697 GoLog com.celzero.bravedns I udpmux.go:265: I udp: mux: f78f98d522be5fc4 route: new for 8.8.4.4:53; stats: &{0 {13955697707408456447 1307458265908 532290359584} {{} 8} {{} 660} {{} 305}}
2024-08-30 21:32:08.589 28538-2692 GoLog com.celzero.bravedns I udpmux.go:125: D udp: mux: f78f98d522be5fc4 awaiter: watching 0.0.0.0:37288 => 8.8.4.4:53
2024-08-30 21:32:08.590 28538-2692 GoLog com.celzero.bravedns I common.go:412: D onFlow: udp noalg? false or hasips? false
2024-08-30 21:32:08.590 28538-2692 GoLog com.celzero.bravedns I common.go:416: D onFlow: udp no realips() or domains( + ), for src=10.111.222.1:40413 dst=8.8.4.4:53
2024-08-30 21:32:08.609 28538-2692 GoLog com.celzero.bravedns I protect.go:67: D control: netbinder: Exit: udp4(8.8.4.4:53); err? <nil>
2024-08-30 21:32:08.614 28538-2697 GoLog com.celzero.bravedns I exit.go:53: I proxy: exit: dial(udp) to 8.8.4.4:53; err? <nil>
2024-08-30 21:32:08.614 28538-2697 GoLog com.celzero.bravedns I udp.go:315: I udp: connect: 5049098b0031bfbe (proxy? [email protected]:1337) 10.1.204.164:42248 -> 8.8.4.4:53/8.8.4.4:53; mux? false, uid 10671
2024-08-30 21:32:08.843 28538-3107 GoLog com.celzero.bravedns I common.go:50: D intra: 8fec78d73c5a1f36 download(792) done(read udp 10.1.204.164:41362->8.8.8.8:53: i/o timeout) b/w a(10.111.222.1:35376->8.8.8.8:53) => b(10.1.204.164:41362<-8.8.8.8:53)
2024-08-30 21:32:09.620 28538-3107 GoLog com.celzero.bravedns I udpmux.go:455: D udp: mux: f78f98d522be5fc4 demux: read: done(sz: 132)
2024-08-30 21:32:10.616 28538-3107 GoLog com.celzero.bravedns I udpmux.go:455: D udp: mux: f78f98d522be5fc4 demux: read: done(sz: 132)
2024-08-30 21:32:11.227 28538-2692 GoLog com.celzero.bravedns I udpmux.go:198: I udp: mux: 78140498e6950c35 read timeout(1): read udp4 0.0.0.0:45297: i/o timeout
2024-08-30 21:32:11.227 28538-2692 GoLog com.celzero.bravedns I udpmux.go:198: I udp: mux: 78140498e6950c35 read timeout(2): read udp4 0.0.0.0:45297: i/o timeout
2024-08-30 21:32:11.227 28538-2692 GoLog com.celzero.bravedns I udpmux.go:203: I udp: mux: 78140498e6950c35 read done n(0): read udp4 0.0.0.0:45297: i/o timeout
2024-08-30 21:32:11.227 28538-2692 GoLog com.celzero.bravedns I udpmux.go:374: D udp: mux: 78140498e6950c35 demux 0.0.0.0:45297 => 100.75.223.185:443 close, in: 0, over: 0
2024-08-30 21:32:11.227 28538-2692 GoLog com.celzero.bravedns I udpmux.go:387: I udp: mux: 78140498e6950c35 demux from 0.0.0.0:45297 => 100.75.223.185:443 closed
2024-08-30 21:32:11.228 28538-2692 GoLog com.celzero.bravedns I udpmux.go:276: I udp: mux: 78140498e6950c35 unrouting... 0.0.0.0:45297 => 100.75.223.185:443
2024-08-30 21:32:11.228 28538-2692 GoLog com.celzero.bravedns I udpmux.go:142: D udp: mux: 78140498e6950c35 stop
2024-08-30 21:32:11.228 28538-2692 GoLog com.celzero.bravedns I udpmux.go:133: I udp: mux: 78140498e6950c35 awaiter: done
2024-08-30 21:32:11.228 28538-2692 GoLog com.celzero.bravedns I udpmux.go:164: I udp: mux: 78140498e6950c35 drain: closing 0 demuxed conns
2024-08-30 21:32:11.228 28538-2692 GoLog com.celzero.bravedns I udpmux.go:153: I udp: mux: 78140498e6950c35 stopped; stats: tx: 0, rx: 6250, conns: 1, dur: 123s
2024-08-30 21:32:11.228 28538-2692 GoLog com.celzero.bravedns I udpmux.go:374: D udp: mux: 78140498e6950c35 demux 0.0.0.0:45297 => 100.75.223.185:443 close, in: 0, over: 0
2024-08-30 21:32:11.228 28538-2692 GoLog com.celzero.bravedns I udpmux.go:374: D udp: mux: 78140498e6950c35 demux 0.0.0.0:45297 => 100.75.223.185:443 close, in: 0, over: 0
2024-08-30 21:32:11.228 28538-2692 GoLog com.celzero.bravedns I udpmux.go:510: I udp: mux: 78140498e6950c35 (Exit) dissoc for 10.111.222.1:36198
2024-08-30 21:32:11.629 28538-2697 GoLog com.celzero.bravedns I udpmux.go:455: D udp: mux: f78f98d522be5fc4 demux: read: done(sz: 132)
2024-08-30 21:32:12.642 28538-3107 GoLog com.celzero.bravedns I udpmux.go:455: D udp: mux: f78f98d522be5fc4 demux: read: done(sz: 132)
2024-08-30 21:32:13.656 28538-3107 GoLog com.celzero.bravedns I udpmux.go:455: D udp: mux: f78f98d522be5fc4 demux: read: done(sz: 132)
netstack's gonet API requires for its internal book-keeping (that is, routing subsequent packets from the same TCP flow to an already established gonet.TCPConn).
@ignoramous Can't we call CreateEndpoint() and Complete() after Dial() return? Let me see what will happen. 🧐
Can't we call CreateEndpoint() and Complete() after Dial() return? Let me see what will happen.
May work if in "Single Threaded" mode.
https://github.com/celzero/firestack/blob/f3a671e5ff01ad3d4836fc0f5722b5a516bb5955/intra/settings/config.go#L82-L84
May work if in "Single Threaded" mode.
I don't know if gvisor is in "Single Threaded" mode by default, but it works.
https://github.com/Lanius-collaris/gvisor-playground
(I use built-in fdbased.New() and built-in stack.New())
https://github.com/Lanius-collaris/gvisor-playground/blob/ba027236b8a490a6c7c426338d4fa965a1f49708/app/libs/aegis/stack/tcp.go#L75 think that wait might block all other processing in netstack.
It looks to me that by default netstack's fdbased runs a "single processor". The client may have to opt-in (via ProcessorsPerChannel) to run multiple of those. firestack runs 6.
https://github.com/Lanius-collaris/gvisor-playground/blob/ba027236b8a490a6c7c426338d4fa965a1f49708/app/libs/aegis/stack/tcp.go#L75 think that wait might block all other processing in netstack.
@ignoramous
Wait() is called by another goroutine, can you see go func(){ ... }() ? I guess that you reached maxInFlight ( It is 64 in demo_cli ). Try setting a smaller timeout in DialTCPFn and increasing maxInFlight. firestack called Complete() too early, so it is unlikely that firestack will reach maxInFlight .
https://github.com/google/gvisor/blob/17edc7260e94/pkg/tcpip/transport/tcp/forwarder.go#L135
It looks to me that by default netstack's fdbased runs a "single processor".
default ProcessorsPerChannel >= 1 https://github.com/google/gvisor/blob/17edc7260e94/pkg/tcpip/link/fdbased/endpoint.go#L333-L335
if opts.ProcessorsPerChannel == 0 {
opts.ProcessorsPerChannel = max(1, runtime.GOMAXPROCS(0)/len(opts.FDs))
}
Gotcha! Thanks.
Attempt: https://github.com/celzero/firestack/commit/b1397718b1a6f406820de348484a7a68471db60c
@ignoramous
Can't we call CreateEndpoint() and Complete() after Dial() return? Let me see what will happen. …
Gotcha! Thanks.
Attempt: https://github.com/celzero/firestack/commit/b1397718b1a6f406820de348484a7a68471db60c
https://github.com/celzero/firestack/blob/18e9d907b35f1a31d544f849685df7799b2858c0/intra/netstack/tcp.go#L121-L134
This is not enough, you need to adjust intra/tcp.go as well:
diff --git a/intra/tcp.go b/intra/tcp.go
index 4d8d702..738518e 100644
--- a/intra/tcp.go
+++ b/intra/tcp.go
@@ -186,14 +186,6 @@ func (h *tcpHandler) Proxy(gconn *netstack.GTCPConn, src, target netip.AddrPort)
return deny
}
- // handshake; since we assume a duplex-stream from here on
- if open, err = gconn.Establish(); !open {
- log.E("tcp: %s connect err %v; %s => %s for %s", cid, err, src, target, uid)
- clos(gconn)
- h.queueSummary(smm.done(err))
- return deny // == !open
- }
-
if isAnyBasePid(pids) { // see udp.go:Connect
if h.dnsOverride(gconn, target, uid) {
// SocketSummary not sent; x.DNSSummary supercedes it
@@ -209,6 +201,13 @@ func (h *tcpHandler) Proxy(gconn *netstack.GTCPConn, src, target netip.AddrPort)
}
if err = h.handle(px, gconn, boundSrc, dstipp, smm); err == nil {
+ // handshake; since we assume a duplex-stream from here on
+ if open, err = gconn.Establish(); !open {
+ log.E("tcp: %s connect err %v; %s => %s for %s", cid, err, src, target, uid)
+ clos(gconn)
+ h.queueSummary(smm.done(err))
+ return deny // == !open
+ }
// smm instead queued by handle() => forward()
return allow
} // else try the next realip
https://github.com/google/gvisor/blob/60ec4e7a009d/pkg/tcpip/transport/tcp/forwarder.go#L101 gvisor's built-in TCP Forwarder always runs the handler in a new goroutine, so a "blocking" handler can't block the netstack "processor".🤣
And would you like to introduce a CLI "frontend" to test some functions on Linux? The Go mobile is too slow. My patch:
diff --git a/linux-tool/glue/glue.go b/linux-tool/glue/glue.go
new file mode 100644
index 0000000..3261fbd
--- /dev/null
+++ b/linux-tool/glue/glue.go
@@ -0,0 +1,118 @@
+package glue
+
+import (
+ "encoding/binary"
+ "log"
+ "syscall"
+ "unsafe"
+
+ "github.com/celzero/firestack/intra"
+ "github.com/celzero/firestack/intra/backend"
+)
+
+type MyBridge struct{}
+
+// implement intra.log.Console
+func (bridge *MyBridge) Log(level int32, msg string) {
+ log.Printf("log level %d, %s", level, msg)
+}
+
+// implement intra.backend.Controller
+func (bridge *MyBridge) Bind4(who, addrport string, fd int) {}
+func (bridge *MyBridge) Bind6(who, addrport string, fd int) {}
+func (bridge *MyBridge) Protect(who string, fd int) {}
+
+// implement intra.SocketListener
+func (bridge *MyBridge) Preflow(protocol, uid int32, src, dst, domains string) *intra.PreMark {
+ return &intra.PreMark{
+ UID: "-1",
+ IsUidSelf: false,
+ }
+}
+func (bridge *MyBridge) Flow(protocol, uid int32, src, dst, origdsts, domains, probableDomains, blocklists string) *intra.Mark {
+ return &intra.Mark{
+ PIDCSV: "Base",
+ CID: "0",
+ UID: "-1",
+ }
+}
+func (bridge *MyBridge) Inflow(protocol, uid int32, src, dst string) *intra.Mark {
+ return &intra.Mark{
+ PIDCSV: "Base",
+ CID: "0",
+ UID: "-1",
+ }
+}
+func (bridge *MyBridge) OnSocketClosed(*intra.SocketSummary) {}
+
+// implement intra.backend.ResolverListener
+func (bridge *MyBridge) OnDNSAdded(id string) {}
+func (bridge *MyBridge) OnDNSRemoved(id string) {}
+func (bridge *MyBridge) OnDNSStopped() {}
+
+// implement intra.backend.DNSListener
+func (bridge *MyBridge) OnQuery(uid, domain string, qtyp int) *backend.DNSOpts {
+ return &backend.DNSOpts{
+ PIDCSV: "base",
+ IPCSV: "",
+ TIDCSV: "Preferred",
+ TIDSECCSV: "",
+ NOBLOCK: false,
+ }
+}
+func (bridge *MyBridge) OnResponse(*backend.DNSSummary) {}
+
+// implement intra.backend.ServerListener
+func (bridge *MyBridge) SvcRoute(sid, pid, network, sipport, dipport string) *backend.Tab {
+ return &backend.Tab{
+ CID: "0",
+ Block: false,
+ }
+}
+func (bridge *MyBridge) OnSvcComplete(*backend.ServerSummary) {}
+
+// implement intra.backend.ProxyListener
+func (bridge *MyBridge) OnProxyAdded(id string) {}
+func (bridge *MyBridge) OnProxyRemoved(id string) {}
+func (bridge *MyBridge) OnProxyStopped(id string) {}
+func (bridge *MyBridge) OnProxiesStopped() {}
+
+func Ioctl(fd int, req uint64, data []byte) (int, error) {
+ p := unsafe.Pointer(&data[0])
+ r1, _, errno := syscall.Syscall(syscall.SYS_IOCTL, uintptr(fd), uintptr(req), uintptr(p))
+ if errno == 0 {
+ return int(r1), nil
+ } else {
+ return int(r1), errno
+ }
+}
+func OpenTUN(name string) (int, error) {
+ fd, err := syscall.Open("/dev/net/tun", syscall.O_RDWR, 600)
+ if err != nil {
+ return fd, err
+ }
+ var req [64]byte
+ copy(req[:15], []byte(name))
+ flags := uint16(syscall.IFF_TUN | syscall.IFF_NO_PI)
+ binary.NativeEndian.PutUint16(req[16:], flags)
+ _, err = Ioctl(fd, syscall.TUNSETIFF, req[:])
+ if err != nil {
+ defer syscall.Close(fd)
+ return -1, err
+ }
+ return fd, nil
+}
+func GenFDCmsg(fds []uint32) []byte {
+ cmsgLen := 16 + len(fds)*4
+ cmsg := make([]byte, cmsgLen)
+ binary.NativeEndian.PutUint64(cmsg[:], uint64(cmsgLen))
+ binary.NativeEndian.PutUint32(cmsg[8:], syscall.SOL_SOCKET)
+ binary.NativeEndian.PutUint32(cmsg[12:], syscall.SCM_RIGHTS)
+
+ offset := 16
+ for _, v := range fds {
+ binary.NativeEndian.PutUint32(cmsg[offset:], v)
+ offset += 4
+ }
+ return cmsg
+}
diff --git a/linux-tool/toy/main.go b/linux-tool/toy/main.go
new file mode 100644
index 0000000..a9d59a3
--- /dev/null
+++ b/linux-tool/toy/main.go
@@ -0,0 +1,100 @@
+package main
+
+import (
+ "encoding/binary"
+ "flag"
+ "fmt"
+ "net"
+ "os"
+ "os/exec"
+ "os/signal"
+ "syscall"
+
+ "github.com/celzero/firestack/intra"
+ "github.com/celzero/firestack/intra/settings"
+ "github.com/celzero/firestack/linux-tool/glue"
+)
+
+func p(e error) {
+ if e != nil {
+ panic(e)
+ }
+}
+func sendTUN(name string) {
+ sock := int(3)
+ defer syscall.Close(sock)
+ syscall.SetNonblock(sock, true)
+ tun, err := glue.OpenTUN(name)
+ p(err)
+ defer syscall.Close(tun)
+
+ ifInfo, err := net.InterfaceByName(name)
+ p(err)
+ var msg [4]byte
+ binary.NativeEndian.PutUint32(msg[:], uint32(ifInfo.MTU))
+ cmsg := glue.GenFDCmsg([]uint32{uint32(tun)})
+ err = syscall.Sendmsg(sock, msg[:], cmsg, nil, 0)
+ p(err)
+}
+func demo(targetPid int, tunName string, fakeDNS string) {
+ pair, err := syscall.Socketpair(syscall.AF_UNIX, syscall.SOCK_SEQPACKET, 0)
+ p(err)
+ defer func() {
+ syscall.Close(pair[0])
+ syscall.Close(pair[1])
+ }()
+ syscall.SetNonblock(pair[0], true)
+
+ elfPath, err := os.Executable()
+ p(err)
+ c1 := exec.Command("/usr/bin/nsenter", "--target", fmt.Sprintf("%d", targetPid), "--user", "--net",
+ "--preserve-credentials", "--keep-caps",
+ elfPath, "-mode", "sendfd", "-tun", tunName)
+ c1.ExtraFiles = []*os.File{os.NewFile(uintptr(pair[1]), "")}
+ fmt.Println("starting child...")
+ output, err := c1.CombinedOutput()
+ if len(output) > 0 {
+ fmt.Printf("child's output:\n%s\n", string(output))
+ }
+ p(err)
+
+ var buf [4]byte
+ var cmsgBuf [20]byte
+ n, cmsgN, msgFlag, _, err := syscall.Recvmsg(pair[0], buf[:], cmsgBuf[:], 0)
+ p(err)
+ fmt.Printf("msg: %v\ncmsg: %v\nflag: %d\n", buf[:n], cmsgBuf[:cmsgN], msgFlag)
+ tun := int(binary.NativeEndian.Uint32(cmsgBuf[16:]))
+ defer syscall.Close(tun)
+ mtu := binary.NativeEndian.Uint32(buf[:])
+
+ settings.SetDialerOpts(settings.SplitDesync, settings.RetryNever, 10, true)
+ var bridge glue.MyBridge
+ tunnel, err := intra.NewTunnel(tun, int(mtu), fakeDNS, nil, &bridge)
+ p(err)
+ defer tunnel.Disconnect()
+
+ ch := make(chan os.Signal, 1)
+ signal.Notify(ch, syscall.SIGINT, syscall.SIGTERM)
+ wait1 := <-ch
+ fmt.Printf("received %v signal, exiting\n", wait1)
+}
+func main() {
+ var mode string
+ flag.StringVar(&mode, "mode", "main", "main or sendfd")
+ var tunName string
+ flag.StringVar(&tunName, "tun", "tun0", "")
+ var pid int
+ flag.IntVar(&pid, "target", -1, "target process to get namespaces from")
+ var fakeDNS string
+ flag.StringVar(&fakeDNS, "dns", "10.0.2.3:53", "DNS passed to intra.NewTunnel()")
+ flag.Parse()
+
+ switch mode {
+ case "main":
+ demo(pid, tunName, fakeDNS)
+ case "sendfd":
+ sendTUN(tunName)
+ default:
+ fmt.Printf("unknown mode: %s\n", mode)
+ }
+}
Usage: In the first terminal
unshare -U -r -n
$ ip link set lo up
$ ip tuntap add mode tun name tun0
$ ip link set tun0 mtu 1500
$ ip link set tun0 up
$ ip addr add 10.0.2.100/24 dev tun0
$ ip addr add fd00::100/64 dev tun0
$ ip route add 0.0.0.0/0 dev tun0
$ ip route add ::/0 dev tun0
$ echo $$
3000
In the second terminal
$ go build -trimpath ./linux-tool/toy
$ ./toy -mode main -tun tun0 -target 3000
And would you like to introduce a CLI "frontend" to test some functions on Linux
Nice. I'll merge this if you'll PR it or I can cherry-pick relevant commits from your branch (if it is public).
gvisor's built-in TCP Forwarder always runs the handler in a new goroutine, so a "blocking" handler can't block the netstack "processor.
iirc, in our testing (must be mid ~2023), handling TCP + ICMP in quick succession (or some such) stun netstack into submission & introduction of goroutines "fixed" it. It could very well have been that our handling of ICMP was problematic & TCP probably had nothing to do with what we observed.
And would you like to introduce a CLI "frontend" to test some functions on Linux
Nice. I'll merge this if you'll PR it or I can cherry-pick relevant commits from your branch (if it is public).
I uploaded the patch to the quick-test1 branch, it's not ready for opening a pull request.