Face detection

[whyboris]

It seems like it may be relatively simple to perform face detection (placing a bounding box across any face found in an image). This process may be super fast: apparently faster than realtime for video (which means stupid-fast for a few screenshots).

It may be simple enough to simply detect any faces within a filmstrip and extract them into a new faces array which can be viewed separately somehow (would need UI work -- unsure what it would look like yet).

~Possibly the best solution: https://github.com/tehnokv/picojs~

~Another alternative is: https://github.com/eduardolundgren/tracking.js but it hasn't been updated since 2018~ 😕

~Another alternative is:~ https://github.com/justadudewhohacks/face-api.js/ (see Tiny Face Detector), but it requires TensorFlow and I don't want some heavy dependencies for this minor feature 😅

Someday in far future, we could get very advanced with face recognition too: https://github.com/justadudewhohacks/face-api.js/

🤔 🤷‍♂ 😁

[whyboris]

Update:

• Tried picojs and it's not a good fit: it has many false-positives (e.g. finds faces in elbows)
• Tried face-api and I'm really liking it. Assuming including TensorFlow and other sub-dependencies isn't going to make the app too heavy, I just might include it. We'll see 🤷‍♂

I have a script that seems to generate an image strip with faces found in all the extracted screenshots. I'm thinking create a new view that looks like the filmstrip view, but it will only show faces 🤔

I'm open for ideas about how to display the found faces and what to do with them. 👂

[cal2195]

Well this sounds very exciting! 😄

Two thoughts about what you could do with the data:

• If you can find the same face in multiple videos, this would be a very cool improvement to the similar video recommendations (Right click -> Show more videos with this actor)

• Secondly, I see you can also detect expressions - this would be a very cool tag to add, if the user enabled it. (eg. tag videos with happy, surprised etc.)

I like this direction a lot! 🎉

[whyboris]

I have a repository for the internal functionality: https://github.com/whyboris/extract-faces-node

You can take any thumbnail strip from your library and run extract faces.

One caveat is you have to set the number of screenshots there are in the filmstrip inside the script: https://github.com/whyboris/extract-faces-node/blob/master/src/pipeline.ts#L8

🚀 I'll get a branch of VHA ready to see how well it plays with Electron 😅

[whyboris]

Currently running into trouble when trying to install canvas 😓 -- so far no-go ... Electron post-install script errors out not being able to find a pre-built binary(?)

[whyboris]

Problem solved 🎉 😂 https://github.com/whyboris/extract-faces-node/pull/1 -- was able to remove canvas 😁 Attempting again with VHA 🤞

[whyboris]

Seems to work well 😎 #342 App compiles to 125MB 👌 Finds and extracts faces while in dev but not yet from the built app -- I'll investigate 🔍 😁 🔎

[frez777]

Pretty cool idea!

Well, even the idea of adding an actor section itself is very useful and nice feature.

[whyboris]

@frez777 -- btw -- have you clicked on the "Auto Tags"? It will automatically find all the 2-word strings that appear in your files (generally actor names if they appear in the file names).

I'll likely expand this feature more -- thank you for the request 👍

[whyboris]

help wanted : if you have experience and time, could you see how to get the built application to extract images?

Branch face-detection (see PR: https://github.com/whyboris/Video-Hub-App/pull/342) -- when npm start the app extracts faces 👍 but when npm run electron:windows the app doesn't extract faces 👎

[whyboris]

Things I've tried:

• [x] Make sure the Node is the same version as what Electron uses ✅ -- seems like it may be required for rebuilding native dependencies step

[whyboris]

🎉 🎉 🎉 🎉 🎉 🎉 🎉 🎉 PROBLEM SOLVED 🎉 🎉 🎉 🎉 🎉 🎉 🎉 🎉 🎉

Turns out the weights folder needs to be copied correctly once the app installs (e.g. inside Program Files/Video Hub App 2/... 👍

I should have a build-able branch ready later today (see #342)

[whyboris]

Current intention for the future features, not all will be in #342:

1. is to extract every face for every filmstrip and store it in a new folder (with the same file name as the hash as the file).
2. add a 'faces' view that will simply show a filmstrip with all the faces found
3. create another 'auto-generated tag' view that shows only the two-word tags (hopefully mostly names), but next to each it will have a single photo
4. each of the photos can be replaced by dragging-dropping an image file.
5. this new view will also (like the auto-generated tags) allow to add more names to the list
6. the view will behave like the auto-generated tag view -- clicking will open all the videos that contain that 'name'

These are the initial features -- should be pretty easy to implement. Once it's set and released, I will consider others.

Before that's done -- I'm open to more suggestions / recommendation for change, etc.

Part of the reason I'm aiming for those 6 things is that they are easy from the code point of view (re-using same functionality a lot). We'll see how it goes.

[whyboris]

CONTINUED DISCUSSION OF THIS FEATURE HERE: #342

[whyboris]

Feature is further postponed until I release version 3.0.0 - see #456 🙆‍♂️

[whyboris]

This feature is not abandoned. I've been wanting to work on it for 3 years now, but after moving to another state, continuing a full-time developer job, and spending time on fixing up the house, I've just not had time to resume this feature yet 😓

But in 2024 I intend to find time for it 🤞