Improve performance for tens of thousands of files

[mherrmann]

In #276, fman 1.0.0 was made ten times faster to make it possible to work with folders containing a few thousand files. That improvement is a start, but fman needs to become faster still. The goal of this issue is to make it so it's possible to work with folders containing tens of thousands of files without significant performance issues.

[raguay]

It is working great for me. Thanks.

The only bad issue I'm having is external programs adding/changing files in the directory that fman currently has open lags quite a bit. It is especially noticeable if there are a large number of files in the same directory.

Should this be a different issue? Or is it similar enough to add to here?

[oskretc]

Kudos on 1.0

I definitely see the improvement. Keep it up.

Off topic: made a new plugin. maybe a bit redundant, but I like how it works

https://github.com/oskretc/fman-sublimehelper

[mherrmann]

@raguay:

Should this be a different issue? Or is it similar enough to add to here?

Can you create a new issue please?

@oskretc:

I definitely see the improvement. Keep it up.

Happy to hear it! I will :-) Congrats on the plugin. It improves on fman's base implementation. Cool! :)

[tmssngr]

When opening a directory with only 66 files (.exe and .zip) on Windows using fman 1.0.9 it takes a few seconds the first time - later it becomes faster.

[sirspot]

Hi, I just started using fman and love it! A large part of the sluggish display is very likely because of how Qt gathers icons from Windows. For a super fast file manager, I don't need pretty icons. I suggest adding an option to either hide icons completely or just use a single set of generic icons for "file", "folder", "symlink", etc. I expect that would make a significant speed improvement.

[mherrmann]

Hi, happy to hear you like it :) fman only loads the icons you can see. That is, if you're in a folder with 10,000 files, of which 50 are visible on your screen, then fman only loads 50 icons. So I'm afraid icons aren't the bottleneck. What's more, Windows Explorer for instance is perfectly capable of showing you the correct icons quickly even if there are many files. So while I keep optimising fman's performance, there's still a way to go.

[kalons7]

I think it could be the icons. The first time I open a folder containing 10+ exes with distinct icons, it takes several seconds to load. Can it initially use a generic icon and then refresh the icons after loading the list?

[mherrmann]

I still don't think it's the icons but could be proved wrong by the following: @kalons7 is it also slow if you take those 10+ exes (but no more than visible on one screen) and copy them to a separate folder?

[kalons7]

A folder containing only 10 exes and nothing else takes several seconds to load after a cold start, if that's what you mean.

[mherrmann]

Yes, that's what I meant. Thanks. You're on Windows right? If yes, can I ask you to:

1. Start fman.
2. Open the directory with the 10 exes in both panes.
3. Close fman.
4. Open a command prompt.
5. Type the following command: %LOCALAPPDATA%\fman\Versions\1.3.2\fman.exe --profile. This starts fman.
6. Close fman.
7. Attach to this issue the file fman.profile that was created in the current directory.

[kalons7]

Yes, Windows 10.

fman.zip

[mherrmann]

Thanks. Your fman spent 2s loading plugins and 3s (way too much) rendering files. But hardly any time loading icons. So if it is to do with icons (which I doubt), then it should be to do with rendering, not loading them.

[kalons7]

Ok, maybe it just seemed that way, since other folders with a lot more files, but often having the same extension and icon, load more quickly the first time.

[ar-jan]

I noticed exactly the same problem, fman being particularly slow in opening a directory with exe files. I have a folder with about 70 files and a dozen subfolders which I guesstimate usually opens in about 15-20 seconds, then about 1-3 seconds on subsequent navigation, and also 1-3 seconds after closing and restarting fman. So instead I rebooted and opened fman via %LOCALAPPDATA%\fman\Versions\1.3.5\fman.exe --profile, and I think this time it took at least 45 sec to open (can I read the .profile file myself?), so hopefully the attached log is useful.

A few times, after I a long load time for this dir, a few seconds after the files had already been displayed, a spinner icon appeared, and clicking the mouse on the other pane (which had another dir open) resulted in a Windows "not responsive" message, and the fman window becoming greyed out for a few seconds. This seems to be much rarer though, but perhaps related.

fman.zip

[Raikiri]

I'm on windows 10 and this delay problem is unfortunately a dealbreaker for now. I'm pretty sure this has to have a solution (as in, it's not a principal limitation of chosen frameworks) and I really hope this gets addressed soon.

[dansteen]

I have this issue as well. not sure if it makes a difference or not, but the folder that I am trying to load is mounted into my WSL instances as well.

[x61x62x6d]

When entering a larger directory (777 folders and files) it takes about 13 seconds to show them. When i try to sort items by size or modification date it does nothing.