TCP max size get stuck

[SkyperTHC]

The bug was introduced after 0.3.1 and all future version suffer from the same: https://github.com/sandialabs/wiretap/compare/v0.3.1...v0.4.0

WIRETAP_INTERFACE_PRIVATEKEY='XXX=' \
WIRETAP_PEER_PUBLICKEY='YYY=' \
WIRETAP_PEER_ENDPOINT='1.2.3.4:44444' \
WIRETAP_RELAY_INTERFACE_PRIVATEKEY='XXX=' \
WIRETAP_RELAY_PEER_PUBLICKEY='YYY=' \
WIRETAP_RELAY_PEER_ENDPOINT='1.2.3.4:44444' \
WIRETAP_SIMPLE='true' \
WIRETAP_RELAY_PEER_ALLOWED='172.16.0.0/16,fd:16::0/104' \
WIRETAP_RELAY_INTERFACE_LOCALHOSTIP='172.16.0.1' \
./wiretap serve -d

Any TCP download stops at exactly 8192 bytes. I've not looked much into it. Let me know if you like me to investigate this further.

Origin uses WireGuard. EXIT uses wiretap (MTU=1420). The same works fine if WireGuard is used on both, Origin and EXIT (MTU=1420).

I played around with the MTU=1000 but to no avail. I played around with TCPMSS but to no avail either.

[Aptimex]

To clarify, are you saying that any TCP connections made through Wiretap are only able to transfer 8192 bytes with those settings? Do they hang, close the connection unexpectedly, or something else?

[SkyperTHC]

A TCP connection that uses small packet chunks (interactive netcat or ssh) are not effected. The bug instead appears when requesting (on ORIGIN) curl -fL http://SERVER through the wiretap tunnel and when these conditions are met:

1. WireTap EXIT is within <10ms of SERVER
2. SERVER sends large amount of data (e.g. burst).

(There might be other scenarios but in these scenarios I can reproduce it 100% of the time).

In this situation, where the SERVER sends lots of information and very fast, then the ORIGIN only sees the first 8192 bytes of it and then the connection "hangs" (it does not disconnect).

I can dig deeper into it (tcpdump etc) if you like (let me know).

[fragtion]

I've noticed a similar issue, where pings to the wiretap node's localhost IP (or peer address) often won't go through unless there is an active TCP connection in progress. As soon as there's a TCP connection, the host responds to pings. Not sure if it's related to this or needs its own issue but thought I should mention it here. It would be nice if we could make ping "availability" more reliable/robust than that

[Aptimex]

Thanks for clarifying. If you can do a little more digging that would definitely be appreciated. Do you know what a reliable threshold is for "lots of information and very fast" is? Is just requesting a single very large file over a gigabit connection enough to satisfy that condition?

Fragtion, please report that in a different issue since that seems to be a different problem that will probably need to be assessed separately. If you have reason to believe it's related to this feel free to reference this issue in the new issue.

[Aptimex]

After some initial testing I wasn't able to reproduce this.

I started the demo docker environment. CLIENT/ORIGIN is at 10.1.0.2, SERVER/EXIT is directly reachable at 10.1.0.3, and a target machine is at 10.2.0.4 which the CLIENT/ORIGIN can only reach by going through the SERVER/EXIT. The virtual Ethernet interfaces in the docker containers run at 10000Mb/s (as reported by ethtool), and ping consistently shows a round-trip time of <3ms between virtual hosts (including when going through Wiretap).

I configured Wiretap with this command:

./wiretap configure --endpoint 10.1.0.2:51820 --routes 10.2.0.0/16,fd:2::/64 --simple

This generates the following serve command to use, which is pretty close to what are using:

WIRETAP_RELAY_INTERFACE_PRIVATEKEY=XXX \ 
WIRETAP_RELAY_INTERFACE_PORT=51820 \
WIRETAP_RELAY_PEER_PUBLICKEY=YYY \
WIRETAP_RELAY_PEER_ENDPOINT=10.1.0.2:51820 \
WIRETAP_SIMPLE=true \
./wiretap serve

I ran that command on the SERVER/EXIT, and successfully connected to it using this command on the CLIENT/ORIGIN: wg-quick up ./wiretap.conf.

I generated a 1GB random file on the target host (openssl rand -out sample.txt -base64 $(( 2**30 * 3/4 ))) and started a file server to serve it: python3 -m http.server 8080.

I'm able to download the entire file, through wiretap, without issue on the CLIENT/ORIGIN host: curl -fL 10.2.0.4:8080/sample.txt -o out.txt. The Wiretap tunnel seems to limit download speeds to ~32MB/s (~300mbps?) based on the time it takes to transfer that file using curl, not sure if that matches what you're seeing on your end.

---

If you can reproduce this issue inside the demo docker environment let me know how your setup differs from what I tested. If you can't reproduce it there we'll have to track down what environment factors are contributing to this issue on your end so I can reproduce it.

[SkyperTHC]

I'm away for 1 week but will help when I am back. The data can be very little (like 512k) that the SERVER sends.

I can not reproduce it in docker either. This is always true for my setup and I am sorry for not having mentioned this earlier (when I can reproduce it):

1. There is a >100ms latency between ORIGIN and EXIT (real Internet.)
2. There is a <10ms latency between EXIT and SERVER.

I feel bad that I can not help right now but will jump onto this next week. Feel free to wait.

I wonder if it's an IP-fragmentation problem but than WireGuard should fail as well...hmm.. I wish I could help today :> I like tricky problems.