netmaker icon indicating copy to clipboard operation
netmaker copied to clipboard

[Bug]: High CPU load crashing server with internet gateway on AWS

Open bpmarine opened this issue 1 year ago • 5 comments

Contact Details

[email protected]

What happened?

Testing two different instances of AWS Lightsail servers, setting up as default CE installs with custom domains using the latest v 0.21 quick-install script.

Single host installed by default during the installation. Added egress gateway to this host with internet gateway 0.0.0.0 for exit and no other networks. Single external client config added.

Connecting to the server from a Pixel 6 and the wireguard app. Observed no DNS resolution so I set default DNS as 9.9.9.9 to test. Resolved properly with this set.

While the external client is connected I'm seeing 100% CPU utilisation on the server. Available burst time drops to 0%. After several hours the server stops responding until rebooted from the Lightsail console.

Tested two different Lightsail instances, same results. No other configuration tested.

Version

v0.21.0

What OS are you using?

Linux

Relevant log output

No response

Contributing guidelines

  • [X] Yes, I did.

bpmarine avatar Sep 25 '23 10:09 bpmarine

Actually, I now see that the CPU usage while I'm connected is within 0-20%, very low. Spikes are occurring at other times and crashing the server

bpmarine avatar Sep 25 '23 15:09 bpmarine

I just experienced the same thing on a Lightsail server running the netmaker server. Huge spike in cpu load, stopped responding to the network. I had to force stop it and then restart the server. It is a bare Ubuntu server with just Netmaker installed so there is unlikely any other causes.

harry-hart avatar Oct 05 '23 08:10 harry-hart

@bpmarine @harry-hart what's the memory and cpu configuration of your server?

abhishek9686 avatar Oct 05 '23 10:10 abhishek9686

1 GB RAM, 2 vCPUs on a lightsail server. In normal usage it hovers around 1% usage, but was sustained bursting to 80% when the "bug" occured.

Server was version: v0.20.6

harry-hart avatar Oct 05 '23 11:10 harry-hart

image

This is the graph of cpu usage. The drop was when I force shutdown.

harry-hart avatar Oct 05 '23 11:10 harry-hart