umbrel
umbrel copied to clipboard
getumbrel
Tor high CPU usage
After installing Umbrel 0.5 I noticed that every container has it's own tor server, and a main tor server for the dashboard / auth.
This main tor server is extremely busy on my machine, it uses more CPU than any other process, even more than Bitcoin and Lightning combined.
This is really strange for several reasons:
- On previous releases this tor server never used so much CPU. It was always Bitcoin, Electrs and LND that were the top 3, now suddenly it is tor.
- This server is only used for the dashboard and for auth. But I don't even use the dashboard frequently, and when I do, it's never over tor.
- It uses MUCH more than even the bitcoin-tor-server, while you would expect Bitcoin to generate much more traffic than the dashboard.
So is there any logical explanation for this?
When I look in the logfiles it is continously trying to connect to an onion service that does not exist, failing every time. It's address is [scrubbed] so I have no idea which one it is, but I think this causes the high CPU usage.
@kroese Thanks for reporting this, I am now seeing similar behaviour and will need to investigate why. I see roughly 10% CPU utilisation for that tor container while all the other are at 0%.
@kroese I've taken a closer look and the behaviour I'm seeing is expected. In v0.5 we changed the architecture so that each app would it gets own Tor container for the purpose of serving apps as Hidden Services (HS) because pre v0.5 we had a lot of issues running all the apps (all the Hidden Services) inside just 1 Tor container.
The main tor container however can still experience higher CPU utilisation because it also serves as a SOCKS proxy for the system and for the apps. If you're doing for example doing a Initial Block Download (IBD) atm with the Bitcoin app, then you'll see some utilisation of this main tor container.
I un-installed all apps, and now that container is idling. We could run a SOCKS proxy in each tor container for each app, but we wanted to minimise usage of resources needed to run Tor.
Yes, I know in previous releases there was a limit of 10 services per tor instance as a workaround, and instances were spun up dynamicly based on the number of apps.
If there were also multiple proxies in the past, that are now considilated into one single instance, that would explain the difference. But as far as I remember that was not the case? But maybe Im mistaken.
Before we didn't have multiple proxies. We had the main tor container serving a SOCKS proxy and the Hidden Service for the dashboard. And then another tor container where as you say we batched up Hidden Services and then ran them using tor processes. Ie we only ever had 1 SOCKS proxy.
We are now using a newer version of Tor across the board, so potentially that is another reason.
It must be something related to this new version of Tor then, because for the past month I've monitored Umbrel's cpu usage every day, and I never saw Tor in the top 5 processes before.
I think it's the only variable left. What kind of utilisation are you seeing? On my Pi4, I am seeing ~5-10% utilisation and I'm currently doing a Initial Block Download (IBD) with Bitcoin.
I am not doing any IBD or generating large amounts of traffic otherwise. So the situation should be almost exactly the same is before upgrading to 0.5.
I'm not on a Pi4 but on a Intel Celeron N5105 which is a few times faster, so it will be difficult to compare CPU utilisations.
But I always look at the "CPU Time" in top. You can sort on this value by pressing SHIFT+T after starting top. This is an accumulation of all the CPU time combined since the start of the system. So this is a better metric than the current utilisation (which could be temporarily high but low again later).
So based on total CPU time in the last 24 hours I see that Tor is the most demanding process.
There are also more reports of this issue, I quote "My 8gb pi4b runs hotter than before" on https://community.getumbrel.com/t/0-5-0-multiple-tor-containers-running/7983