explorerplusplus
explorerplusplus copied to clipboard
derceg
Progressive file list load
Thanks for the app. My use case is tabs so I am really trying to use the same features as Windows 10 File Explorer.
At the moment I am using folders with many thousands of files. Opening any of these folders causes the app to not respond.
It would be good if the file list progressively loaded, like Windows File Explorer, instead of freezing. This would also allow the user to navigate away if they opened the wrong folder.
In some cases, I have had to crash the app and go into the registry to remove the open tab as each time I reopened the app, the same freeze would occur.
Maybe a million. Often more than 30,000. Yes, this is not a good thing to do but it is the constrains of the application we use, it all needs to be in one folder.
Wow, a million is a tall order!
I see a lag of about 8 seconds in directories with ~43000 files after my changes in PR #263 are applied, so if you build from source and sync them, you might see some improvement on the lower-side of your use cases, but I suspect a million won't see much improvement.
Thanks @sean-e, it might help. Does it fix the "Not Responding" too? Will it allow cancelling by navigating away?
My c++ environment and skills are rusty as I use c# most of the time now. I'll have to wait for the merge to master.
The changes in #263 do not change behavior re: progressive loading or cancelling the operation -- they just improve time to populate the file list. It's been awhile since I used a version without my changes; I don't recall having gotten "not responding" myself.
@craigneasbey, I ran some benchmarks with different versions of file loops in exp++ In ideal conditions (Detail view instead of List view, pure FindFirstFile+FindNextFile Loop, local files on SSD, Spectre mitigations turned off) I could manage to open a folder of 1 million items in 43,8 s with a 3.7 GHz Intel CPU. It would be faster than that when using either Windos 7 SP1 or with a newer version of Win10 (I used v1809) and a newer CPU than is less affected by the changes made to the OS following Meltdown/Spectre mitigations.
Exact results were: FindFirstFile+FindNextFile loop (Item Width check disabled) 42,1 seconds adding + 1,7 s sorting FindFirstFile+FindNextFile loop 53,6 seconds adding + 1,7 s sorting EnumObjects+SHGetDataFromIDList loop 70 seconds adding + 1,7 s sorting
You're still looking at a long wait even in the best scenario. How exactly do you work with these files? Do you need all of them in the listview at all times for size or date sorting purposes, or do you only need a certain subset with specific name patterns?
Hi @KrokusPokus and @sean-e, thanks for your efforts.
In essences this is a feature request, to mimic features in Windows 10 File Explorer, progressive load and cancel load.
So, the files I have are on a SSD network drive. I can't change that. I actually accidentally clicked one of these folders just then and I am still waiting (Not Responding). It can be more than half an hour or never, I am not willing to wait that long if I accidentally click it. I even had to go into the registry and remove the folder from the tabs in Explorer++ because if I crash it, it will just reappear (I like to restore tabs, the whole point I use this software).
So, I am not looking at performance improvement, even though that could help a little bit, I need the features. Explorer++ is only using 10% CPU and 350MB RAM. That is clearly not the issue. I need to be able to cancel the operation at very least.
thanks for your help.
To be able to cancel the operation, the enumeration would have to be done on a background thread. Which is something I've thought about doing for years, but have been hesitant to do so, given the potential complexity of the change and the possibility of introducing new bugs to the core of the application.
Moving the enumeration itself to a background thread might not be too difficult, but it would break the assumption that a tab is always open within a specific folder and showing the items within that folder. The second point may not be too important, given that a tab doesn't necessarily show all the items within a folder right now. For example, when a file or folder is created, it doesn't appear immediately. So there's already support for adding/changing/removing items after a navigation has completed.
I don't know exactly how difficult it might be to perform the enumeration on a background thread, given that I've never seriously tried to do it. What I might do is try implementing it on a new branch and see how it goes.
If that then works ok, it might be best to implement this behind a command line switch on master initially, then make it the default once it appears to work well. But I'm not sure how feasible it would be to support both the current behavior and new behavior at the same time. So I'll probably have to see exactly what changes need to be made first.
I've moved the enumeration onto a background thread in a new branch. There's a build of this branch available on AppVeyor that you can download and try out.
I've tested the changes on Windows 7, 8 and 10, with a variety of different folders and they seem to work ok. I've also tested some of the other functionality to make sure it still works when a folder is loading and that does appear to be the case. One current issue that I'm aware of is that if a file/folder is created during the enumeration, that item will be shown twice. That's because the item will be inserted both by the code handling the enumeration and the code handling directory changes. I'm not sure how much of a problem that will wind up being in practice, but it's likely something that would need to be fixed.
There are notes within the main commit that outline some of the implementation details, so I won't repeat those here. One item not explicitly mentioned in the commit message is that you can't directly cancel a navigation (e.g. by pressing ESC). You can, however, navigate to a different folder, close the tab, or continue to use the application while a folder is being loaded.
It would be good to get some feedback on the changes from other people. For example, whether the application crashes or hangs, whether there's any functionality that doesn't appear to work or whether there are any windows in the application (e.g. the status bar) that aren't updated correctly when you navigate into a folder.
Also, it's easiest to change the implementation now, so if you think there's something that could be improved, it's best to make suggestions now.
One idea that I initially had would be to try and provide some further feedback when loading a folder. For example, the count of items loaded so far. However, I don't think that would be very useful. For small folders, the count would only briefly flash and be unreadable. Large folders should be enumerated much more quickly because of this change, so it may not be very useful for those folders either. That does mean, though, that there's nothing that currently indicates the progress of the enumeration. It is still possible to determine that an enumeration is in progress, however, since the cursor will be set when in the listview and the status bar text will also indicate a load is taking place.
Hi @derceg, Thanks for all your time and effort. I have downloaded, seems to fix the issue. Works great so far. Tested with 80k documents on network share, it only took seconds now anyway. Back navigation cancels.
Sorry I don't have much time at the moment.
Actually, one issue, I can't change to another tab while this is happening. Well, I can but I don't see the contents. Hmmm...this sometimes works....and sometimes it goes not responding, if you click too many other tabs. I think it works most of the time.
Select all and then select one again might cause not responding too.
Is it possible to add progressive load? As in, adding the files to the file list as they are enumerated? Or in batches? You may have answered this above.
thanks heaps, Craig
@derceg I have been using this background_thread_enumeration build daily now since it first compiled on AppVeyor, trying to gauge performance and stability.
I have not experienced any issues at all. I think that it would be great if this could be either merged with master or, alternatively, add this to master but kept as an option to toggle on your new Advanced tab. That way users can toggle to try it out and provide some feedback.
Keep up the great work. Cheers!
