High CPU usage

[soullivaneuh]

I have a rtorrent instance with currently ~2800 active torrents.

The node server is regularly at nearly 100% CPU usage:

And this even if I don't have any opened browser tab on it.

Any idea of why?

[jfurrow]

This is unfortunate, I suspect it's struggling to process 2800 torrents each time Flood gets the updated torrent list from rTorrent. What CPU are you running this on?

You can adjust how frequently Flood polls rTorrent by editing torrentClientPollInterval in config.js. This defaults to 2000 milliseconds (2 seconds). You can try to change this to 5 or 10 seconds, which would reduce the continuous load on your CPU, but won't solve the core problem.

To understand the core problem, let's look at what Flood does each time it requests data from rTorrent:

1. Ask rTorrent for list of torrents with all of the details needed by Flood (here are all of the details requested).
2. Transform rTorrent's response into an object of torrents, here. rTorrent's response is a two-dimensional array, each nested array being an array of values. This means each time Flood gets the list from rTorrent, it iterates over each torrent (2800 torrents in your case) and also iterates over each requested value, transforming the values as necessary (torrentListPropMap contains the functions which transform the values — 36 values in total, right now).
3. Some values need to be determined by one or more of the 36 values returned by rTorrent (percent complete and ETA, for example). So we loop over any external reducers (here]. Currently, all external reducers are added here.
4. After this object of torrent data is created, Flood iterates over each torrent again, comparing the new set of torrent data with the old, constructing a diff with which the client is updated.

That's a lot of work for the server to be doing — at the very least 218,400 iterations every 2 seconds, if my math is correct (2800 torrents * 36 values + 2800 torrents * 3 external reducers + 2800 torrents * 39 total values for diff comparison).

To optimize, we can do a number of things...

1. Flood uses the reduce method on the list of torrents and the list of values, then the foreach method on the list of external reducers. In web browsers, these methods are considerably slower than for loops, and that probably holds true for Node as well. So clientRequestService#processTorrentListResponse should be refactored to use for loops.
2. Each value transformation function should be double- and triple-checked for optimization opportunities, since they will be invoked so frequently. I suspect the external reducer for a torrent's status is particularly inefficient.
3. Construct the diff during the torrent value transformation, rather than a separate iteration afterwards.
4. ??? I'm sure there are more opportunities.

[tympanix]

I have experienced high load as well. In fact, flood is one of the busiest processes on my machine when everything is running in idle. Constantly polling the server seems unnecessary to me. Would one be able to only poll the rTorrent instance for updates, when there are in fact active clients to serve?

[soullivaneuh]

4 Dedicated ARM Cores

This is the only information I have, I delete this virtual machine since this issue... :D

[randomnonsense]

Works fine for me. Perhaps your VM was heavily underpowered. with default polling I get some 1.0-2.5% on Xeon E3, with around 4000 torrents. Dedicated server.

[hogklint]

218,400 iterations every 2 seconds

Doing all that work even when the system is completely idle seems excessive. A config option to only poll rtorrent for info when serving a webclient would be splendid. I assume it currently always polls to keep some kind of history, something I have no use for.

[noraj]

@nomme see https://github.com/jfurrow/flood/blob/master/config.template.js

[hogklint]

@nomme see https://github.com/jfurrow/flood/blob/master/config.template.js

I must be missing something... You can adjust how often to poll and how many records to keep. Neither of which changes the behavior if no client is connected, correct? The intention with my previous comment was to adress the CPU usage. If the CPU usage increases only when the information is displayed it would improve the situation.

[damolp]

First of all, great work with Flood. It's pretty awesome and the reason why I'm attempting to come back to rTorrent.

Anyway, I migrated my torrent boxes across from Deluge to rTorrent this weekend. I found that as I added more and more torrents from deluge to rTorrent the CPU on the box the node web service starts to hit 100% on one core. I tried adjusting torrentClientPollInterval like was mentioned before to 15 but this just delayed the problem. It got to the point that it was unusable and I had to script some Python with XMLRPC to do my bidding.

I tried the node profiler (disclaimer: I've never touched node.js before in my life) and saw that sax is taking most of the cycles (or ticks? in node world?) [1]. I had a look around for some more efficient libraries and found this one [2] which claims a 6x speed increase over sax.

Anyway, I've managed to monkey patch in saxen via [3], [4] and [5].

Well, the result from the node profiler [6] looks a lot better than before. And my CPU usage is now around 40% with torrentClientPollInterval set to 1 second! 😃 Note that this is just the deserialisation that I changed. We may expect a few more cycles back if we change serialisation as well.

I'll post some metrics tomorrow showing the difference. I've just turned on 1 minute metrics on these servers, they previously only had 1 hour metrics.

If I get some time this week I'll try and put this effort into a proper pull request.

[1] sax-profile.txt [2] https://github.com/nikku/js-sax-parser-tests [3] make-flood-use-xmlrpc-with-saxon.patch.txt [4] https://github.com/nikku/saxen/commit/398f09b56b23961f9de77406dd8d857b32f440fd [5] https://github.com/damolp/node-xmlrpc/commit/f97c5db3cced31f75767ef9958ef89341d66b167 [6] saxen-profile.txt

[damolp]

So thanks to @nikku for merging my commit in saxen without having to send him a pull request 😄

The change to saxen has made it upstream in version 8.1.0

Anyone experiencing high CPU usage with flood should be able to pull this commit to use the modified version of node-xmlrpc with saxen.

A bit more work needs to be done before I create a pull request though 😞

@tympanix would you be able to give it a test? I'd be interested to see if this helps reduce the CPU on your instance.

[nikku]

@damolp You're welcome. :cake:

[tympanix]

@damolp great research and work on everything. I have my own fork of node-xmlrpc which could need a little speed up with saxen using your work. Sadly I removed flood off my system a long time ago. The high CPU usage was simply not justified. Your additions is a huge step towards that - but in my opinion the only way to really combat this issue is to completely remove them problem: serve clients when there are clients to serve. Not doing that is simply polling for information and throwing it away (to my knowledge flood does not keep history?). That said, great work! 🥇 I may get around to testing your work. Busy times though.

[tehnatural1]

Using the commits from @damolp I can confirm that my CPU usage has been cut by more than in half. I still have an odd spikes but they are far less substantial.

[jfurrow]

Thanks @damolp for the XMLRPC deserializing improvement PR!

I think I'll keep this issue open for now, as there have been some other ideas discussed as well... what do you think @noraj?

[noraj]

I'm not very informed about sax but the claims of PR #797 sounds promising. I just wish the deserializer doesnt not introduce vulnerabilities.