server sending TCP RST / reset by peer

[seek2222]

HI all,

I have a problem that the server is sending TCP RST toward the client. I have gone through the logs and I couldn't find the root cause of sending the RST.

tshark/tcpdump from server: 36 31.420796365 SERVER_IP → CLIENT_IP TCP 54 443 → 53520 [RST, ACK] Seq=1 Ack=1 Win=65535 Len=0 37 31.663715425 CLIENT_IP → SERVER_IP TCP 78 53530 → 443 [SYN] Seq=0 Win=42340 Len=0 MSS=1460 SACK_PERM TSval=717831679 TSecr=0 WS=2048 TFO=R

server: RUST_LOG=info /root/phantun/target/release/server --local 443 --remote 127.0.0.1:51822 &> /var/log/phantun_server.log &

~# tail -f /var/log/phantun_server.log INFO server > Remote address is: 127.0.0.1:51822 INFO server > 1 cores available INFO server > Created TUN device tun0 INFO server > Listening on 443

client: RUST_LOG=info /root/phantun/target/release/client --local 127.0.0.1:1234 --remote SERVER_IP:443 &> /var/log/phantun_client.log &

INFO client > New UDP client from 127.0.0.1:46180 INFO fake_tcp > Sent SYN to server INFO fake_tcp > Connection to SERVER_IP:443 established INFO fake_tcp > Connection (Fake TCP connection from 192.168.200.2:27979 to SERVER_IP:443) reset by peer INFO fake_tcp > Fake TCP connection to (Fake TCP connection from 192.168.200.2:27979 to SERVER_IP:443) closed

server iptables: iptables -t filter -nvL Chain INPUT (policy ACCEPT 530K packets, 197M bytes) pkts bytes target prot opt in out source destination
Chain FORWARD (policy ACCEPT 0 packets, 0 bytes) pkts bytes target prot opt in out source destination
562 22480 ACCEPT 0 -- tun0 * 0.0.0.0/0 0.0.0.0/0
562 35940 ACCEPT 0 -- * tun0 0.0.0.0/0 0.0.0.0/0

Chain OUTPUT (policy ACCEPT 0 packets, 0 bytes) pkts bytes target prot opt in out source destination

iptables -t nat -nvL Chain PREROUTING (policy ACCEPT 365K packets, 28M bytes) pkts bytes target prot opt in out source destination
1083 68296 DNAT 6 -- eth0 * 0.0.0.0/0 0.0.0.0/0 tcp dpt:443 to:192.168.201.2

any input is much appreciated. P.s, the 443 TCP/udp port is free from the netstat side as well. I know already that the phantun is not registering the listening port on Kernel.

[smith752]

Check whether system kernel forwarding is turned on, the server side and client side the value of net.ipv4.ip_forward should be equal 1 # sysctl -a | grep "net.ipv4.ip_forward" net.ipv4.ip_forward = 1

you can execute this command to turn on kernel forwarding echo "net.ipv4.ip_forward = 1">>/etc/sysctl.conf;sysctl -p

[seek2222]

@smith752 first off, thanks for replying. secondly, the ipv4 forwarding is enabled on both phantun client and server.

[seek2222]

my server is running Ubuntu 22.04 and on another instance it is running on 24.04 with latest kernel installed.

[iopq]

Annoyingly, you may need to also disable the rp_filter

sudo sysctl -a | grep rp_filter
sudo sysctl -w net.ipv4.conf.all.rp_filter=0

check if it's on and turn it off

[shigella520]

I’m having the same problem. Client output log: reset by peer

Server output log: RUST_LOG=trace /usr/local/bin/phantun_server --local 61820 --remote 127.0.0.1:51888 INFO server > Remote address is: 127.0.0.1:51888 INFO server > 1 cores available DEBUG neli::socket > Message sent: Nlmsghdr { nl_len: 64, nl_type: Newaddr, nl_flags: NlmFFlags(FlagBuffer(1, PhantomDataneli::consts::nl::NlmF)), nl_seq: 0, nl_pid: 0, nl_payload: Payload(Ifaddrmsg { ifa_family: Inet6, ifa_prefixlen: 128, ifa_flags: IfaFFlags(FlagBuffer(0, PhantomDataneli::consts::rtnl::IfaF)), ifa_scope: 0, ifa_index: 30, rtattrs: RtBuffer([Rtattr { rta_len: 20, rta_type: Local, rta_payload: Buffer }, Rtattr { rta_len: 20, rta_type: Address, rta_payload: Buffer }]) }) } INFO server > Created TUN device tun0 INFO server > Listening on 61820

but netstat -tuln | grep 61820 show nothing

[dndx]

Without tcpdump it will be hard to figure out what happened, could you run tcpdump on your public facing interface and see what caused the RST?

[paranoidrookie]

I have tried for many days but still can't solve this problem, could you please help me?

root@racknerd-df5dc34:~/phantun# iptables -L
Chain INPUT (policy ACCEPT)
target     prot opt source               destination         

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination         

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination         
root@racknerd-df5dc34:~/phantun# iptables -L -t nat
Chain PREROUTING (policy ACCEPT)
target     prot opt source               destination         
DNAT       tcp  --  anywhere             anywhere             tcp dpt:4567 to:192.168.201.2

Chain INPUT (policy ACCEPT)
target     prot opt source               destination         

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination         

Chain POSTROUTING (policy ACCEPT)
target     prot opt source               destination         
MASQUERADE  all  --  anywhere             anywhere        

root@racknerd-df5dc34:~/phantun# tcpdump -i tun0 -vv
tcpdump: listening on tun0, link-type RAW (Raw IP), snapshot length 262144 bytes
12:07:27.391549 IP (tos 0x28, ttl 49, id 16171, offset 0, flags [DF], proto TCP (6), length 60)
    82.25.47.xx.38114 > 192.168.201.2.4567: Flags [S], cksum 0xfeb0 (correct), seq 17510862, win 64240, options [mss 1400,sackOK,TS val 1256398592 ecr 0,nop,wscale 9], length 0
12:07:27.391874 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 40)
    192.168.201.2.4567 > 82.25.47.xx.38114: Flags [R.], cksum 0xcb1a (correct), seq 0, ack 17510863, win 65535, length 0
^C
2 packets captured
2 packets received by filter
0 packets dropped by kernel
root@racknerd-df5dc34:~/phantun# iptables -L -t nat -v
Chain PREROUTING (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         
    5   300 DNAT       tcp  --  eth0   any     anywhere             anywhere             tcp dpt:4567 to:192.168.201.2

Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         

Chain OUTPUT (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         

Chain POSTROUTING (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         
   10   696 MASQUERADE  all  --  any    eth0    anywhere             anywhere            
root@racknerd-df5dc34:~/phantun# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether 00:16:3c:8f:d5:1e brd ff:ff:ff:ff:ff:ff
    altname enp0s3
    altname ens3
    inet xxx.xxx.xxxxxx/24 brd 23.95.247.255 scope global eth0
       valid_lft forever preferred_lft forever
6: tun0: <POINTOPOINT,MULTICAST,NOARP,UP,LOWER_UP> mtu 1500 qdisc mq state UNKNOWN group default qlen 500
    link/none 
    inet 192.168.201.1 peer 192.168.201.2/32 scope global tun0
       valid_lft forever preferred_lft forever

 ./phantun_server --local 4567 --remote xxx.xxx.xxx.xx:51820
 INFO  server > Remote address is: xxx.xxx.xxx.xx:51820
 INFO  server > 3 cores available
 TRACE neli::socket::synchronous > Message sent:
Nlmsghdr { nl_len: 64, nl_type: Newaddr, nl_flags: NlmF(1), nl_seq: 0, nl_pid: 0, nl_payload: Payload(Ifaddrmsg { ifa_family: Inet6, ifa_prefixlen: 128, ifa_flags: IfaF(0), ifa_scope: Universe, ifa_index: 6, rtattrs: RtBuffer([Rtattr { rta_len: 20, rta_type: Local, rta_payload: Buffer }, Rtattr { rta_len: 20, rta_type: Address, rta_payload: Buffer }]) }) }
 TRACE neli::socket::synchronous > Buffer sent: [64, 0, 0, 0, 20, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 10, 128, 0, 0, 6, 0, 0, 0, 20, 0, 2, 0, 252, 201, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 20, 0, 1, 0, 252, 201, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 2]
 INFO  server                    > Created TUN device tun0
 INFO  server                    > Listening on 4567
 TRACE fake_tcp                  > Cache miss, checking the shared tuples table for connection
 TRACE fake_tcp                  > Bad TCP SYN packet from xx.xx.xx.xx:38114, sending RST

[dndx]

@paranoidrookie

12:07:27.391549 IP (tos 0x28, ttl 49, id 16171, offset 0, flags [DF], proto TCP (6), length 60)
    82.25.47.xx.38114 > 192.168.201.2.4567: Flags [S], cksum 0xfeb0 (correct), seq 17510862, win 64240, options [mss 1400,sackOK,TS val 1256398592 ecr 0,nop,wscale 9], length 0

This line indicates the client is not Phantun (it has the wrong window size and a couple other TCP params suggesting this is not Phantun's TCP client), did you ran any intermediate TCP proxy between the client and server? Phantun can not work with TCP proxy, only TCP port forwarding and NAT will work.

[paranoidrookie]

@paranoidrookie

12:07:27.391549 IP (tos 0x28, ttl 49, id 16171, offset 0, flags [DF], proto TCP (6), length 60)
    82.25.47.xx.38114 > 192.168.201.2.4567: Flags [S], cksum 0xfeb0 (correct), seq 17510862, win 64240, options [mss 1400,sackOK,TS val 1256398592 ecr 0,nop,wscale 9], length 0

This line indicates the client is not Phantun (it has the wrong window size and a couple other TCP params suggesting this is not Phantun's TCP client), did you ran any intermediate TCP proxy between the client and server? Phantun can not work with TCP proxy, only TCP port forwarding and NAT will work.

I added a hysteria2 proxy layer in the middle, maybe that's the issue. I also encountered this issue when my VMware VM was in NAT mode; after switching to bridged mode and disabling the Hysteria2 proxy, Phantun worked correctly.

[dndx]

Not sure how VMWare's NAT mode works, but yes Phantun can only work without an intermediate TCP stack running in the middle. Glad it is sorted out.