ml5-library
ml5-library copied to clipboard
ml5js
Add useful keypoint getters for Facemesh
See the keypoint map: https://github.com/tensorflow/tfjs-models/blob/master/facemesh/mesh_map.jpg
In the ml5.js Facemesh model, we can consider providing some functions that return an array of keypoints for useful facial landmark ranges like the left eye, right eye, mouth, etc. For example:
facemeshInstance.getLeftEye() // returns [<keypoint 22>, <keypoint 23>, <keypoint 24>, ...]
Facemesh provides these: https://github.com/tensorflow/tfjs-models/blob/master/facemesh/src/keypoints.ts. We may be better off updating our documentation or examples to tell people how to use these rather than building our our own functions.
Just for adding a few other ideas into the stew, we thought about adding some helper functions or properties like isMouthOpen or isLeftEyeOpen in which case we could help to handle some of the logic to run these calculations for our audience. Maybe worth considering in terms of usability?
I'd be happy to release facemesh for now as well and then do a followup PR for these additions :)
@joeyklee That would be really useful. Do you know how reliable the model is for isLeftEyeOpen/isRightEyeOpen?
Hi! @oluoluoxenfree -- Yes! I believe this ticket is still open ✨
I might defer to @shiffman for coordinating on this feature addition in relation to the other backlog of open PRS that might take priority, but I think having these helper methods would be a nice addition for ml5 users <3
Do you know how reliable the model is for isLeftEyeOpen/isRightEyeOpen?
- Hi @aspdev91 -- thanks for this question. Quite likely there might be some inaccuracies since we would have to calculate the distance between the keypoints that make the face landmarks for the eyes which means the "reliability" of our calculation depends on the precision of the face keypoints. It could be something we test out to see if it is useful as a starting point for beginners and then more advanced users can tweak or add their own algorithms? 🤷 🙈
Also noting that since it has been so long since we've updated this feature, a bunch of the tensorflow models have been reorganized -- e.g. facemesh I think is now https://github.com/tensorflow/tfjs-models/tree/master/face-landmarks-detection // just something we will need to address as a whole across ml5
@joeyklee I was going to start a whole new issue but I wondered what the current state/direction and leadership of the project was? Really interested in getting involved :)
Thanks @oluoluoxenfree ! Definitely a good question.
I've not been as involved as I'd like to be over the last couple years and SO much has changed w/ ML generally and tensorflow that it has been hard to catch up, let alone maintain this project.
My general feeling is that there's interest to pick things up again, but slowly. Ideally I'd love to review and merge as many of the the open PRs as a way to get back up to speed -- @lindapaiste did so much good work and would set future feature oriented work up in a nice way.
One thing that does come to mind now would be to do an audit to see how different API changes from some of the tensorflow pre-trained models have changed / are out of line w/ the currentl version of ml5. That would help us to roadmap what changes we'd need to do to start using the latest tensorflow version and the latest pre-trained models.
Is there any news at all @shiffman on how things have developed this semester at NYU?
maybe another way to approach it too is -- what kind of things would you like to work on @oluoluoxenfree ? Is there anything in particular that you'd find particularly exciting?
@joeyklee I'm new to ML but old to JS and one of the few people who isn't an aspiring polyglot I guess? I'm really excited by the idea of ML in JS, and by making the barriers to entry much lower.
Auditing the API sounds like a cool way to get an overview, but I'm open to most things! Very vague answer sorry ha ha.