LANraragi
LANraragi copied to clipboard
Difegue
[Bug] Shinobu permanently omitting files after it stops half way
This issue comes from #274 my big list of LRR bugs and suggestion. Please refer to that to see details of my setup and usage.
LRR Version and OS
LANraragi for Windows 0.6.9 (MSI installer) on Microsoft Windows 10 version 10.0.18363.720. Archives are located on a NAS and accessed via a mount point in the content folder.
Bug Details
When the Shinobu background worker imports a lot of galleries at once, it can stop running half way through without throwing anything. Then the background worker will need to be restarted to continue the addition. This can happen fairly frequently and Shinobu can stop twice while importing 1,000 archives. When Shinobu stops, its status is still "OK!" in the admin settings and no error is given even in debug mode.
Shinobu will eventually say "all done!" after necessary restarts, but it is possible that a few archives are not added at all and the total counts would differ between LANraragi and my file system. These archives are permanently omitted because moving them around and restarting stuff cannot trigger Shinobu to add them. I think this problem can always happen and is not affected by auto-tag and auto-plugin.
This is a serious bug, but it happens randomly and so far I can only report three findings; after Shinobu stops:
- The GUI can lag a bit.
- Untagged archives will not be automatically ticked on the batch tagging page.
- Batch tagging will start slowly, so after the "Started Batch Tagging operation..." message. it can take quite a while before it tags the first archive.
Difegue's comment from #274
I've had a few reports of a similar bug, but so far I have no idea what causes it. I thought the recent multiprocessing changes would fix it, but if you're running 0.6.9 I guess it doesn't do enough. I'm guessing it's a load problem where the server gets overwhelmed and fucks up the Shinobu process, but since it spits no errors I haven't been able to give it a proper diagnosis. I'd like a separate issue open for this as well so it can be properly tracked.
Due to #350 I've made the filewatcher a bit more resilient, with all the work that's gone into Minion as well I think overloading issues shouldn't be happening anymore as well.
Feel free to reopen if there are still any issues!
I have tested this issue again on 0.7.5. Now Shinobu will no longer break and stop half way, so that part is fixed. However, Shinobu can still permanently omit files when I add like more than a thousand archives, which means the omitted files cannot be detected even after I restart the file watcher.
I noticed that when I added a certain set of archives and some were permanently omitted, the ones omitted always seem to be the same archives. For example, when I add the same 5,258 zip files, I always get 5,220 archives in the library after Shinobu says all done. Therefore, this bug can be deterministically reproduced.
However, when I tried to figure out which archives were omitted by adding archives in smaller batches, this problem did not happen and the archives added this way added up to the correct total. It seems Shinobu only has this problem when there are too many archives to add (if the list of new archives is prepared in advance) or it keeps running for too long (if in arrear).
I have tested this issue again on 0.7.5. Now Shinobu will no longer break and stop half way, so that part is fixed. However, Shinobu can still permanently omit files when I add like more than a thousand archives, which means the omitted files cannot be detected even after I restart the file watcher.
I noticed that when I added a certain set of archives and some were permanently omitted, the ones omitted always seem to be the same archives. For example, when I add the same 5,258 zip files, I always get 5,220 archives in the library after Shinobu says all done. Therefore, this bug can be deterministically reproduced.
However, when I tried to figure out which archives were omitted by adding archives in smaller batches, this problem did not happen and the archives added this way added up to the correct total. It seems Shinobu only has this problem when there are too many archives to add (if the list of new archives is prepared in advance) or it keeps running for too long (if in arrear).
I can confirm, I'm experiencing the exact same issue on 0.7.5 (WIndows user). It was already happening on previous versions.
I did a clean installation and a complete database reset to experiment on this. I have around 24 000 files, and once the process ends, it's always off by ~100 files. If I restart from zero after yet another database reset, and decide to only process a set of 300 files and add them, say, 20 by 20 until I reach the 300 count, nothing is omitted. If I add those exact same 300 files in one go, then I'll end up with a few files being omitted.
I've rewritten the filemap in https://github.com/Difegue/LANraragi/commit/0214e5976329431a89774dde33e5faa7f82e85bd to entirely depend on Redis instead of the built-in data structures, which might alleviate the issue with large archive sets being halfway omitted.
If the issue still happens, restarting the worker is now slightly smarter as well (state is kept and it tries to only scan the diff instead of rescanning everything), so at worst it should be faster to get everything up to sync. (Thinking of my boys with the 75k sets)
I have the same problem, Shinobu scans about 2000 archives and then stops adding new ones, even if I manually restart LLR and click rescan, LLR only outputs Scanning content folder for changes... and doesn't add anything. I have about 9000 archives in my library and can only add 2000 eventually. This is a serious problem. Unlike other issues I use rclone to mount network drives
