fwupd
High memory usage?
I have fwupd installed on Ubuntu 23.10 via apt.
top is currently reporting fwupd has a resident memory usage of 137 MB on a system with 947 MB available, or 14.1% of the total.
This seems quite a lot. What is the typical memory usage of this process?
ubuntu@server:~$ fwupdmgr --version
runtime org.freedesktop.fwupd 1.9.5
compile com.hughsie.libxmlb 0.3.10
runtime org.freedesktop.fwupd-efi 1.4
compile org.freedesktop.gusb 0.4.5
runtime com.dell.libsmbios 2.4
compile com.hughsie.libjcat 0.1.9
runtime org.kernel 6.5.0-26-generic
compile org.freedesktop.fwupd 1.9.5
runtime org.freedesktop.gusb 0.4.5
This seems quite a lot.
That seems huge to me.
What is the typical memory usage of this process?
On my system it's about 5MB of heap. Can you attach the output of
sudo cat /proc/`pidof fwupd`/smaps
....on your system please.
USS and RSS are higher, but not huge:
sudo smem
PID User Command Swap USS PSS RSS
82155 root /usr/libexec/fwupd/fwupd 0 22396 24202 41420
Wowsa, that's 102MB of heap -- way too much. Can you do sudo valgrind --tool=massif /usr/libexec/fwupd/fwupd --timed-exit (which will take a few minutes maybe) and then attach the massif.out.PIDHERE file please.
Hmm, that showed a 2.1MB peak -- I wonder if there's a memory leak somewhere that means it grows over time. Does that sound plausible?
ubuntu@server:~$ sudo valgrind --tool=massif /usr/libexec/fwupd/fwupd --timed-exit
==3472== Massif, a heap profiler
==3472== Copyright (C) 2003-2017, and GNU GPL'd, by Nicholas Nethercote
==3472== Using Valgrind-3.21.0 and LibVEX; rerun with -h for copyright info
==3472== Command: /usr/libexec/fwupd/fwupd --timed-exit
==3472==
12:31:48.537 FuEngine failed to add device /sys/devices/virtual/msr/msr0: failed to add device using on msr: could not read IA32_DEBUG_INTERFACE: failed to read from port 0x0c80: Input/output error
12:31:48.931 FuMain Daemon ready for requests (locale C.UTF-8)
==3472==
No idea TBH. For reference it's an amd64 VM on Oracle cloud with an imported cloud image downloaded from Canonical.
Shouldn't be too exotic.
Was that command meant to kill fwupd by the way? It's not running now.
Was that command meant to kill fwupd by the way? It's not running now.
Yup, the new "fwupd" takes over from the old one. Could you maybe run sudo valgrind --leak-check=full --show-leak-kinds=definite /usr/libexec/fwupd/fwupd as a background task for a few hours (using the VM like you normally do) and then ctrl+c that process and give us the full output? I'll end something like:
==86906== LEAK SUMMARY:
==86906== definitely lost: 271 bytes in 3 blocks
==86906== indirectly lost: 43 bytes in 1 blocks
Before we go too far down this rabbit hole, as it's a VM I think it's better to upgrade to 1.9.15 first.
1.9.5 was released 6 months ago and I know we've had various memory leak issues fixed since then...
valgrind results, ran for a couple of hours.
I also have an Ubuntu 24.04 VM running 1.9.14. Resident memory usage for fwupd on that machine is currently 124 MB.
Hmm. 484 bytes isn't anything much to worry about. Can you repeat without valgrind, and then do:
sudo cat /proc/`pidof fwupd`/smaps
...again after a few hours -- thanks. Something weird is going on.
The 23.10 instance is showing lower RES since stopping valgrind. Around 35 MB:
The 24.04 instance is showing 124 MB, it's only been up a few hours, here is the output from that:
Okay, then I don't understand :) 65MB of heap but no leaks -- maybe we can do sudo valgrind --tool=massif /usr/libexec/fwupd/fwupd again, but this time let it run until the RSS is huge rather than quitting after just a few seconds.
This time valgrind created a lot of massif.out.* files.
The initial one was: massif.out.6179.txt
stdout:
ubuntu@server:~$ sudo valgrind --tool=massif /usr/libexec/fwupd/fwupd
==6179== Massif, a heap profiler
==6179== Copyright (C) 2003-2017, and GNU GPL'd, by Nicholas Nethercote
==6179== Using Valgrind-3.21.0 and LibVEX; rerun with -h for copyright info
==6179== Command: /usr/libexec/fwupd/fwupd
==6179==
09:25:50.025 FuEngine failed to add device /sys/devices/virtual/msr/msr0: failed to add device using on msr: could not read IA32_DEBUG_INTERFACE: failed to read from port 0x0c80: Input/output error
09:25:50.417 FuMain Daemon ready for requests (locale C.UTF-8)
09:31:00.341 FuEngine HSI attribute org.fwupd.hsi.Mei.Version (from pci_mei) had unknown result
==6263==
==6265==
==6267==
==6269==
==6271==
==6273==
==6275==
==6277==
==6279==
==6281==
==6283==
==6285==
^C==6179==
==6179== Process terminating with default action of signal 2 (SIGINT)
==6179== at 0x518420F: poll (poll.c:29)
==6179== by 0x4FAF3CE: ??? (in /usr/lib/x86_64-linux-gnu/libglib-2.0.so.0.7800.0)
==6179== by 0x4F5446E: g_main_loop_run (in /usr/lib/x86_64-linux-gnu/libglib-2.0.so.0.7800.0)
==6179== by 0x10EFE0: main (in /usr/libexec/fwupd/fwupd)
==6179==
So, weirdly I can't reproduce this when refreshing the metadata and then quitting the daemon. I tried with main and also 1.9.5...
Oracle Cloud has a free tier if you fancy trying out my VM: https://www.oracle.com/uk/cloud/free/
UK South region, availability domain 2, VM.Standard.E2.1.Micro shape.
Only 22.04 is native, but you can upload a newer image to a bucket and import it as a custom image: https://cloud-images.ubuntu.com/
I don't think I can do that I'm afraid. Could you try updating to something newer and then check if the issue still happens? 1.9.15 would be perfect, .deb or snap.
I've purged the apt package and installed the snap. What do I run now to debug?
Perhaps the easiest way of running the latest fwupd is by running debian unstable.
High memory usage is consistent with the other versions I tested, currently 135 MB.
debian@fwupd:~$ ps -h -o rss `pidof fwupd`
137992
debian@fwupd:~$ fwupdmgr --version
compile org.freedesktop.fwupd 1.9.15
compile com.hughsie.libxmlb 0.3.15
compile com.hughsie.libjcat 0.2.0
runtime org.freedesktop.fwupd-efi 1.4
compile org.freedesktop.gusb 0.4.8
runtime org.freedesktop.gusb 0.4.8
runtime com.hughsie.libjcat 0.2.0
runtime org.freedesktop.fwupd 1.9.15
runtime org.kernel 6.7.9-amd64
@triatic can you do one more thing please:
sudo killall fwupd -9
sudo rm /var/cache/fwupd/metadata.xmlb
sudo /usr/libexec/fwupd/fwupd -vv
and then attach the entire log please. I've tried to reproduce quite a few different ways and can't get any kind of leak, let alone one that big.
Log: fwupd.txt
Memory usage is less (53 MB) when run in this fashion.
debian@fwupd:~$ ps -h -o rss `pidof fwupd`
54372
Does fwupd behave any differently when started automatically via the Debian systemd timer script?
@triatic I think this is really a measurement problem, more than a leak. Can you try https://github.com/fwupd/fwupd/pull/7039 and then wait 10 seconds after starting fwupd and/or refreshing the metadata?
I'm a debian person, so no idea I'm afraid. I'd suggest compiling by source -- ./contrib/setup should do a lot of the heavy lifting.
I'm not really up for compiling, so I'll watch Debian unstable for new packages and test when it releases.