smartnode icon indicating copy to clipboard operation
smartnode copied to clipboard

Monitoring is causing CPU spikes every 5 minutes

Open zer777 opened this issue 2 years ago • 3 comments

I experienced CPU spikes on testnet in July which were then fixed in Smartnode v1.9.8. I am now running Smartnode v1.10.2 on mainnet and experiencing almost the same CPU spikes every 5 minutes. When I disable monitoring, I do not have any CPU spikes anymore. Could we have a similar bug like the one that was fixed in 1.9.8? The CPU spikes seem to correlate with eth1 RPC calls.

Another user reported CPU spikes every 5 minutes, but thought to be normal for block validation. I do not think that's correct, because disabling monitoring is resolving this. So I do not seem to be the only one experiencing this.

zer777 avatar Oct 01 '23 14:10 zer777

I was the one thinking it was validator duties. Because I never saw it differently. If this behaviour was "fixed" in 1.9.8, it never did anything for my Lighthouse + Besu combination having monitoring and Beaconcha.in monitoring on.

Screenshot_20230930-223658566 (1)

I tested without monitoring and the graph is a bit less noisy. However, I don't feel like it makes that much of a difference but it's hard to tell with s-tui command line graphs.

That's why I still believe this is normal. If it's not, then maybe it has to do with my low base frequency of 1.1 GHz. Basically, whenever there's just a little work, it's a stress which is why there's a spike roughly whenever I need to validate.

Darkmessage avatar Oct 01 '23 20:10 Darkmessage

As per Fu Manshu on Discord:

Fyi i'm running on Besu and also noticed these CPU spikes every 5 minutes. You can solve the problem with "docker stop rocketpool_node" (apparently there is a bug in the current version of Besu RPC) and keep the monitoring on. See https://discord.com/channels/405159462932971535/405163713063288832/1125261530515132489 for reference.

Can confirm it's that for me, so I would suggest to close this issue.

Darkmessage avatar Oct 11 '23 04:10 Darkmessage

Seeing the same thing with Besu/Nimbus.

Stopped the spikes by following the below steps as suggested back in July.

  • nano ~/.rocketpool/user-settings.yml
  • Within root section change enableMetrics to "false"
  • Then do docker start rocketpool_node

Above changes can bee seen having been implemented at the 1930 mark. Between 1920 to 1930 the rocketpool_node docker container had been stopped. image

theslipp avatar Nov 06 '23 09:11 theslipp