[Bug]: Very slow connection to a relayed node behind a CGNAT

[tustunkok]

Contact Details

[email protected]

What happened?

Hi,

I have used Netmaker quite sometime now, and generally I am very happy with it. It is a great tool. Currently, I am running a Raspberry Pi in my home as a client. The Netmaker server is in a Linode, I have one more client which is a Digitalocean droplet, and several external clients. My home network is behind a NAT in my router and behind a CGNAT because of the ISP (Superonline). I enable UDP hole punching in my server config. I also relayed my Raspberry Pi server with the Netmaker server client.

The problem is that I have connection to the Raspberry Pi from my other nodes but is very slow. The bandwith is about 55 - 60 Kbits/sec. Once in a while the bandwidth increases to about 8 Mbits/sec while I am playing with the settings and making tests. After some time without heavy communication between nodes, the bandwidth decreases again to 50 - 60 KBits/sec.

When I tried without relay and check for the latest handshake, for example it shows 38 minutes, 20 seconds ago.

I am sending the network graph and iperf tests below.

Network Graph with relay

Network Graph without relay

iperf Results:

Client connecting to 10.190.190.5, TCP port 5001 with pid 103816 (1 flows)
Write buffer size:  128 KByte
TCP window size: 45.0 KByte (default)
------------------------------------------------------------
[  3] local 10.190.190.1%nm-home port 36688 connected with 10.190.190.5 port 5001 (MSS=1228) (ct=344.93 ms)
client await server close failed: Resource temporarily unavailable
[ ID] Interval            Transfer    Bandwidth       Write/Err  Rtry     Cwnd/RTT        NetPwr
[  3] 0.0000-10.2534 sec  56.4 KBytes  45.1 Kbits/sec  4/0         13        3K/1651785 us  0.00

Without relay, with UDP hole punching after some time

interface: nm-home
  public key: 6et6ojHxs1wKj/nWo9lVPUto+unIPBI36ZhP2DIWcTw=
  private key: (hidden)
  listening port: 40999

peer: zOAHFoqHPUf1TtgCxNtbdRFCGKFvqekT6suR4cTfc1c=
  endpoint: 172.104.131.43:51822
  allowed ips: 10.190.190.254/32, 10.190.190.2/32, 10.190.190.3/32, 10.190.190.4/32
  latest handshake: 1 minute, 34 seconds ago
  transfer: 204.62 KiB received, 30.41 KiB sent
  persistent keepalive: every 20 seconds

peer: GrYiBnyigRAsEcZgjnm8L6r4VOyUiwa225A9CmeVZFA=
  endpoint: 176.234.231.243:11323
  allowed ips: 10.190.190.1/32, 192.168.1.0/24
  latest handshake: 56 minutes, 17 seconds ago
  transfer: 14.09 MiB received, 712.49 KiB sent
  persistent keepalive: every 20 seconds

I tried different MTU settings (1024, 1280, 1420) none of them made a difference. I tried to debug it by myself, search the net. However, no luck until today. I will closely monitor the issue to provide useful feedback if needed. Any help is greatly appreciated. Thank you.

Version

v0.17.1

What OS are you using?

Linux

Relevant log output

No response

Contributing guidelines

• [X] Yes, I did.

[tustunkok]

I notice that if there is no network activity, the handshakes stop working.

[mattkasun]

Have you tried using the Do droplet as the relay instead of netmaker server? Takes docker networking out of the equation

[tustunkok]

Thanks for the reply. Yes, I tried it. It cannot even handshake with the relay (DO droplet).

interface: nm-home
  public key: Va2+kPDO4Emgd8Z38QBFL+NOm+SnKeC5ALgMAkb7XEc=
  private key: (hidden)
  listening port: 35603

peer: B/OIWiVEeJLaKJloC83pNqEkWVwqloAcGzE8aluPNns=
  endpoint: 206.81.19.58:36800
  allowed ips: 10.190.190.1/32, 10.190.190.254/32
  transfer: 0 B received, 2.89 KiB sent
  persistent keepalive: every 20 seconds

[tustunkok]

Thanks for the reply. Yes, I tried it. It cannot even handshake with the relay (DO droplet).

interface: nm-home
  public key: Va2+kPDO4Emgd8Z38QBFL+NOm+SnKeC5ALgMAkb7XEc=
  private key: (hidden)
  listening port: 35603

peer: B/OIWiVEeJLaKJloC83pNqEkWVwqloAcGzE8aluPNns=
  endpoint: 206.81.19.58:36800
  allowed ips: 10.190.190.1/32, 10.190.190.254/32
  transfer: 0 B received, 2.89 KiB sent
  persistent keepalive: every 20 seconds

I disabled the firewall on the droplet. Now, the handshake is successful. I will wait for some time to see if the connection gets worse.

[tustunkok]

The issue continues on DO droplet as well. The latest hanshake is about 10 hours ago.

interface: nm-home
  public key: Va2+kPDO4Emgd8Z38QBFL+NOm+SnKeC5ALgMAkb7XEc=
  private key: (hidden)
  listening port: 46438

peer: B/OIWiVEeJLaKJloC83pNqEkWVwqloAcGzE8aluPNns=
  endpoint: 206.81.19.58:36800
  allowed ips: 10.190.190.1/32, 10.190.190.254/32
  latest handshake: 10 hours, 5 minutes, 56 seconds ago
  transfer: 222.22 KiB received, 4.52 MiB sent
  persistent keepalive: every 20 seconds

[abhishek9686]

@tustunkok please try out the latest version, where we have Turn support to establish connection with hard-to-reach devices, also we have improved relay implementation which is now a EE feature