Directory listing stops and times out for large file counts

[Fluffkin]

FTP Server (F-Droid version) installed. Linux clients (various : gFTP, lftp, gvfs all fail)

Android folders with a large number of files fail to complete a simple directory listing. Instead, the listing output stops after partial retrieval, no more data is sent, then eventually the connection times out.

Test folders with 499 and 450 items on the Android both fail in the same way. One of those folders is just Android generated hashed filenames, so absolutely no weird encoding / languages involved.

Client software works flawlessly with two different test FTP servers hosted on the same LAN (using a raspberry pi as the server). So it's not a LAN / Router, or client problem.

EDIT: Example, an lftp mirror command fails at 14,480 bytes every time (if that helps)

[Xavron]

This may very well be the LIST issue I've just fixed. Was seeing something similar. 188 files and LIST time was causing the whole backup to reach 7 minutes while using a different server application is about 1 minute 44 seconds.

The Arrays.sort() comparator is causing massive performance loss. Not sure what the original reason was there but it doesn't appear to actually be necessary. Testing just Arrays.sort(entries) drops the overall time down to expected time and still successful.

https://github.com/ppareit/swiftp/blob/6bb7a94739e24b7d0ae1d115cb27c84002d8c5b3/app/src/main/java/be/ppareit/swiftp/server/CmdAbstractListing.java#L64

[ppareit]

Great! Please separate pull request and I'll apply. In commit, mention Issue #136 so I'll close.

[Xavron]

Will push the optimizations over what's looking like sometime next week. Still testing and not sure if Arrays.sort() was the whole picture there. Going to do more testing there.

In either case, have further LIST optimization. Should be at least faster by 300-400% (possibly even more) with a ultra quick guess based on what I'm seeing.

Woo hoo :)

Though, it would be nice to have a log entry to go with this. I'm not sure if its just from time or not. FileZilla I know was timing out for me at 20 seconds with the changes for Android 12 (that's now vastly improved and able). Other ftp clients may act differently. Of course some high number of files may still cause the issue. I'm not sure what the limit is now.

The biggest issue is that it has to build the file info response per file. That's heavy.

Edit: Further cut the time in half if its working as well as I'm seeing. Considering everything has multiple CPU cores, I might be able to cut it down really far. Will see what the end result is coming up. Will fix anything else I may run into with LIST.

Edit: Slightly delayed

[Xavron]

Oh. To add a note here. The client should have a setting to adjust the timeout. For instance, FileZilla has a setting in its settings to adjust its timeout from 20 seconds and you could make that 600 seconds if you really wanted. That would allow far more files to be listed.

However, added a pull request that a test shows of 9 seconds for 5k files to now under 1 second for same file listing. So, if its from the amount of files and the client timing out on it, then with that it should be far better, with that pull request.

So previously, that would have been a timeout in FileZilla's default timeout at about 10k files. Now it should be at least 90k (18 seconds * 5k files per second) files using the pull request code lol.

(This should be improved and about as good as it will get now. Currently only github project is updated and not fdroid or Play Store.)