kiwix-desktop
kiwix-desktop copied to clipboard
kiwix
Kiwix windows 64 bit freezes when trying to download file more than 20 mb
@eddisch Could you please test with latest nightly? http://download.kiwix.org/nightly/2024-04-25/kiwix-desktop_windows_x64_2024-04-25.zip
oh it unfroze. so it only freezes when i start a new download, then it unfreezes (the indicator)
Well it doesn't really freeze. It just stops downloading for like 1 minute and then starts downloading.
okay so when the download status freezes and you try to stop the download the whole gui hangs
@eddisch Could we have please a detailed step by step procedure to reproduce the problem?
I was able to reproduce the issue with the progress freezing and making kiwix-desktop unresponsive when an attempt to pause or cancel the download is made during that state. In my case it is caused by the download speed exceeding the disk/storage write speed (I tested by downloading to a USB thumb drive). Then aria2c (the downloader internally used by kiwix-desktop) becomes unresponsive when
- it preallocates the file (this can be countered with the
--file-allocation=noneand--stream-piece-selector=inorderoptions) - it tries to flush received data to the file system (which in the current implementation happens at least once ~per minute~ every 10 seconds).
On a system with a lot of RAM, fast connection and slow storage when aria2c decides to flush all of the accumulated data to the disk it has to wait a long time for that operation to complete. During that time kiwix-desktop's requests for the download progress status are blocked.
Attached are the log files from my experiments of running aria2c manually with various options.
@veloman-yunkan The problem with no pre-allocation is that might run out of space later. My feeling is that we should better handling the pre-allocation period (which can be long) than remove it.
@kelson42 It turns out that --file-allocation=none is (and has always been) the hardcoded mode of how aria2c is started by libkiwix.
@kelson42 It turns out that
--file-allocation=noneis (and has always been) the hardcoded mode of howaria2cis started bylibkiwix.
Though, since the --stream-piece-selector=inorder strategy was not used most of the disk space eventually assigned to the file is effectively "preallocated" well before the download approaches completion.
About --file-allocation=none we have found some information here : https://github.com/kiwix/kiwix-desktop/issues/87#issuecomment-1404069264
Mostly even with --file-allocation=none, aria2 does it anyway (at least if we download in parallel the file)
So it turns out that aria2c is a single-threaded application that processes RPC requests in the same thread in which it performs various download-related operations (establishing connections, reading data from the network sockets & saving it to disk). That's why the RPC interface is so outrageously non-responsive when it's frequently blocked due to slow storage.
@veloman-yunkan OK, but can we stop to make this RPC requests in our main thread? Actually I though this already has been implemented.
@kelson42 My previous comment was about aria2c, not kiwix-desktop. In kiwix-desktop RPC requests to aria2c for download status updates (which are the overwhelming part of RPC requests) are made from a dedicated thread. RPC requests for starting/pausing/resuming a download are still made from the main thread of kiwix-desktop. The problem is that aria2c doesn't handle those RPC requests in a timely manner when it is blocked. And the reason for that is that aria2c is a single threaded application.