How to debug hangs on rescan?

[alexhk]

I am usually processing larger folders. Sometimes it happens that VHA hangs indefinitely, for instance just now at 88% and it won't move. No restart, no reboot, nothing will help, it will always happen again.

I once read there are developer tools built in (Window) but I couldn't figure out how to start them. Is there a log file?

Any other way to figure out which video file causes the program to hang so I can remove it from the folder?

Thanks!

[whyboris]

I'm sorry this is happening - I'm unsure what it could be. There is a chance that FFmpeg crashes when it runs out of memory (it's the library I use to extract the screenshots). I have just updated to a newer version of the library today 🎉 ... could you try it out and see if it makes a difference?

All you have to do is use the main branch (the default one) and run npm install again. Though you should be using Node 14 or newer and it's better to use npm 7 or newer 🤝

[whyboris]

The extraction process is written so that if any one file takes more than some amount of time (depends on extraction settings like number of screenshots and resolution) the app should stop trying to get that screenshot and move on to the next video.

At least in theory it shouldn't ever hang - but since it does, it's some bug. There is no logging by default during the process, but you can add console.log(element) inside the thumbQueueRunner method (for example) to see which file it's processing.

I'm curious -- when it crashes, are you able to quit the app and then come back to where it was, or is the whole hub reset to empty? 😓

Also -- is this happening during adding files (1st stage - adding black rectangle placeholders) or adding screenshots (2nd stage) of the hub-generating process?

[alexhk]

Hi Boris, thanks for the quick reply and no worries.

I assume one of the video files is where it hangs, but no idea which one. It happens at the 2nd stage. It happened when I initially created the hub (I realized after a few hours it's stuck) and I then restarted VHA. Now when I rescan it always hangs at 88%. It doesn't really crash. It just stays at 88%. I can exit the app and start it again, the hub is not empty all files are there and most of them have images. When I manually rescan, it hangs at 88% again...

I actually tried to replace the ffmpeg and ffplay binaries with my v4.4 but after VHA wouldn't create thumbnails at all.

I did not build VHA so far, but I actually meant to install the dev environment so that I could make small modifications in the future. However, since I am new to Angular, Electron etc I just didn't get around it. However, I just pulled an allnighter at the office so it will take a day or two until I am rested.

-- // --

Don't understand the following wrong, I don't want you to change anything. This is just thinking out load for my own use or possibly for an updated version with a nnew (portable) data format in the future.

Since I am using Windows and Linux I wanted to see if the data file could be freed of the path (make it relative to it's location for instance) and take care of / (*nix) or \ (Windows) etc. - compatibility between the different platforms would be great. This way I could run VHA on a secondary machine instead of the notebook, but then at the end open the file from my notebook to find a particular video. Now I am using a second Windows (VM) to create the hubs folder by folder.

Also I wanted to see if I could change the drag and drop action from copy to move, or provide a small selection in the settings. This way if a particular file is found via VHA it can be moved away from it to a different target folder.

And then something completely different, I wanted to understand the code a bit better. Without having looked too far, my suggestion would be to create a oshash for each file when it is first seen. Then when a file is moved within the folders or deleted and possibly later added again, VHA understands it is a known file and not a new one, it then just has to update location(s) for the video but doesn't have to create images/previews again (file could have been moved or it may in fact be a second/duplicate file). This way, the hub becomes somewhat dynamic, since files often get reorganized etc.

Off to get some sleep now.

[whyboris]

Thank you for more details. On re-scan it should only try to add thumbnails that are missing, so if it was a video file that was causing trouble, it would on re-scan be the first in line to be scanned 🤔

Are you pressing the rescan button, or going to Edit folders and then Add missing thumbnails? Sorry for the not-obvious UI (user interface).

Either way, I'd love to replicate the problem you're having so I could fix it. Could it be that you have a massive number of files? Like over 100,000? The app seems to do fine with 20,000+ but I wonder if you're hitting some higher limit I'm not aware of.

Replacing ffmpeg.exe with the newer version might have not worked because of the timeouts I've added (to prevent the app from hanging indefinitely while scanning). I just ran into this problem and updating them just now: #674 😅 - code is on main branch ✅ though -- I just tried replacing the old FFmpeg with a new one by dragging it into the Program Files folder and it worked 🤷 -- might depend on extraction settings (since my code could make the timeout too short, and the video never gets extracted) ... this reminds me to test on a slow HD instead of my SSD - before releasing the new version of the app 😅

If you get the dev environment working on your machine, you'll be able to try out the new version of ffmpeg I put in and see if the bug was related to it or something else 🚀 would be really great to know 🤝 thank you 🙇

Getting the dev environment should be pretty simple: install node and git and use the terminal (git-bash that comes with git if you're on Windows) and then just follow the 3 steps on the README file (front page) on this repository.

I like your ideas with the software -- I'll be happy to help you out with the code 🤝 -- let me know when you have questions (whether it's about coding in general, or my app specifically).

ps - VHA already has hashing of files (by reading just the 1st, middle, and last parts of the file) and the screenshots it generates for the file persist even if you delete / restore / move the file. It doesn't keep the tags in memory after deletion though.

[alexhk]

I must have completely overlooked the Add missing thumbnails option. Using that instead of just rescan it completed all tasks without error. However, when I use rescan later it again stops at 88% again. It's the same with the new version (3.2), I installed it on two different machines for testing. It's just around 4000 video files. It's a mounted (local) network drive (drive letter), but not sure if that would cause a hang.

This is what I get in the console.log and in the bash shell:

It does actually see as if the process is done, just the display is not updated and stays at 88%. Now it's sometimes 94%, 88% or sometimes it's all good when I repeatedly hit rescan. Also, when it's stuck at 88% or 96% etc there is no ffmpeg task running as far as I can tell. Maybe just a display issue?

I added console.log('DEBUG: [' + element.fileName + ']'); at line 136 in main-extract-async.ts but I don't get anything in the console. I also added new files to the folder to have it generate new screenshots. I may be too tired to figure this out.

Also, on Windows I had to use npm install -f (force) otherwise the following happens:

npm ERR! notsup Unsupported platform for [email protected]: wanted {"os":"darwin","arch":"any"} (current: {"os":"win32","arch":"x64"})

[neilgrz]

I have the same problem. It just hangs on a scan forever and never continues. It doesn't crash. Does this app have any logging?

[whyboris]

Related issue: #725