Freezing and high CPU and memory usage when scrolling over /dev, likely memory leak

[SaphiraKai]

Just what the title says, when attempting to scroll over the /dev directory the file manager usually almost instantly freezes up. Occasionally it doesn't and works fine, but most of the time this ends up completely freezing the process and requiring a killall.

In order to get some more information, I opened up htop to look at what happens to the process's resource usage when this occurs:

When the bug occurs, joshuto immediately begins consuming around 150% CPU and starts consuming large amounts of memory, by the looks of things a potentially infinite amount.

I suspect the problem is caused by attempting to read files in the selected directory until the end, regardless of how large they are. As /dev has infinitely large files (e.g. /dev/random and /dev/zero), this would mean the process will consume memory and CPU forever.

A side note to this problem; when closing the window, the process continues running and consuming memory and CPU. I watched it accumulate 14GB of garbage data in memory while supposedly "closed." killall joshuto does immediately kill it properly however.

I don't know that joshuto intentionally reads all files listed into memory, if it does then I should point out that that's a rather dangerous thing to do for this reason and others, including security. If this is the case, I strongly recommend not reading the exact contents of all files in the selected directory, and instead either only reading the selected file itself with a hard cap on the size before it stops reading, or preferably not reading them automatically at all.

[DLFW]

Hi! Unfortunately, I cannot reproduce this behavior. I scrolled /dev up and down, waited at some files (also at /dev/random) in case some bug needs some time to wake up, scrolled through various /dev sub-dirs, but Joshuto kept calm, only used up to 4% of my CPU when scrolling fast over a directory whose entries had not been cached before. That's the same as for any other dir. Ah, and it did not freeze at all. :upside_down_face:

Are you using latest master? Does the bug happen with the default configuration? Does it happen all the time? Any other information which could help to reproduce?

I don't know that joshuto intentionally reads all files listed into memory,

AFAIK, it does not; Joshuto does not read the content of the files. Only the preview-script might do. But as you have used htop, you would have recognized if the preview-script caused the problem, I guess?

[SaphiraKai]

Hm, that's weird. I tested with the AUR joshuto package and then tested joshuto-git after i found the bug.

I came across it within about 15 seconds of starting the program, I haven't even touched the configuration yet. I started scrolling rapidly through the root directory to compare speed with ranger (obviously it was extremely favorable), however once I hit /dev it froze up. So far this has consistently happened almost every time I've selected it.

[SaphiraKai]

I've confirmed my first thought was incorrect, using lsof I can clearly see that only /dev itself is being opened.

I believe I have found the real cause! Just now, I noticed the ipfs daemon I'm running was using 200% CPU. Killed it, and lo and behold the joshuto bug disappears.

It seems to be a bad interaction with ipfs mounts, perhaps FUSE itself.

[kamiyaa]

Very interesting! I'm not very familiar with ipfs, so not exactly sure what the cause is

[SaphiraKai]

It doesn't seem to be FUSE itself, I tested mounting an ISO using fuseiso but there were no issues with that. So... Interestingly, it definitely has something to do with ipfs mounts.

You should be able to reproduce this by installing kubo and setting up an empty mount, iirc you should be able to set that up with:

sudo mkdir /ipfs /ipns # i know i know, adding things to the root is annoying but it makes you do it
sudo chown <username> /ipfs /ipns 
ipfs init
ipfs daemon & ipfs mount

[SaphiraKai]

Since the issue doesn't appear to affect ranger, maybe you can compare the differences in behavior to ranger to narrow down what it could be?

[asim215]

I have the same problem. After starting joshuto and spawning "j" key down for example in my home directory. Joshuto freezed and memory leak happend. It occupy all my RAM (12 Gb) and SWAP (10 Gb) memory. And system killed joshuto and my memory was restored to 1 Gb RAM usage. It happend with me at least 3 times in last 1-2 months. I am using joshuto-git, linux EndeavorOS.

[kamiyaa]

I have the same problem. After starting joshuto and spawning "j" key down for example in my home directory. Joshuto freezed and memory leak happend. It occupy all my RAM (12 Gb) and SWAP (10 Gb) memory. And system killed joshuto and my memory was restored to 1 Gb RAM usage. It happend with me at least 3 times in last 1-2 months. I am using joshuto-git, linux EndeavorOS.

Do you also have an IPFS setup?

[asim215]

Do you also have an IPFS setup?

No. I use: f2fs on nvme root; ext4 on hdd1; btrfs on hdd2. Freezes were on ex4 HDD, and on SSD (ext4 or f2fs). Recently I doesn't have them for at least 2 months. Not sure were was the problem? Maybe in dependencies, so it solved after update? If I have that problem again I will tell here.