Custom pose classification tutorial

[khanhlvg]

Goal:

• Write a tutorial to explain how to train a custom pose classifier (e.g. to detect yoga pose)

Technologies used:

• ML Kit's Pose Detection API for pose detection
• TF and TFLite to train a custom model that takes ML Kit's model as input, and outputs classification logits. A model with 2 Dense layers should be enough.
• Firebase for storing training data

Artifacts:

• Data collection app
  • Capture images and use Pose Detection API to detect body landmarks
  • Store the landmarks to Firestore, together with user input labels
• Model training notebook
  • Load the training dataset from Firestore
  • Build a Keras model that classify the pose (model inputs are the landmark from Pose Detection)
  • Convert the model to TFLite
• Demo app
  • Use Pose Detection API to detect landmark from the camera input, and run classification using the TFLite model
  • Can be the same app as the data collection app

[sayakpaul]

Summarizing the discussion between @khanhlvg and myself from today's meeting -

• We first need to start with a demo mobile application (preferably Android or iOS) to first gather data. Here's how the application could be utilized -
  • Let the user first decide which pose (yoga pose for example) they would want to make.
  • Under the pose received from the above step the application would start capturing the landmarks (using ML Kit's Pose Detection API) via continuous video feeds from the camera. This way, for a particular user-defined pose we will have a good amount of different landmarks.
  • Pack the data and upload it to Firestore.

After sufficient data collection, we can utilize the data in order to perform pose classification. So, the problem statement becomes - given a set of body landmarks predict the pose.

I am happy to work on the model training part as well as the model conversion part for the pose classification part. Also, I can work with a mobile developer to help them develop the application such that we gather the data in an expected way.

[Anil-matcha]

Hi @khanhlvg . I can help with building an android app to collect the data required and once it is done can help with training a model too. I am creating a basic repo for the android app here https://github.com/Anil-matcha/pose_detection_data_collector . Kindly let me know if you have any other information to discuss

[khanhlvg]

@Anil-matcha thank you for your interest! @sayakpaul is looking for someone who can develop the Android app for data collection so I think this can be a nice collaboration.

@sayakpaul WDYT?

[sayakpaul]

Yeah exactly. @Anil-matcha and I have discussed about it and we are game :D

[anilmaddala]

@Anil-matcha @sayakpaul I would like to jump in on the project, is there any thing either of you need help with? I can help with on device inference for Android.

[sayakpaul]

@anilmaddala you can collaborate @Anil-matcha on the Android part for now on. Once that is sorted out we can work on the next steps together i.e. collecting the data and building the downstream model and so on.

[sayakpaul]

Is there an update you, folks, would like to share? @anilmaddala @Anil-matcha

[anilmaddala]

@sayakpaul I am working on the data collection part of the app. I have integrated the MLKit apis to get the Pose points, need to implement the persistance part. Can we do on device training from pose classification instead of doing the training on Firebase?

[sayakpaul]

@anilmaddala thanks for the updates. Actually, after we collect the data, the plan is to train a shallow neural network externally with TensorFlow/Keras, convert it to TensorFlow Lite and then redeploy it.

But I am curious regarding the on-device model training part. Do you have a pipeline in mind?

[anilmaddala]

@sayakpaul we might have to still build a base model in cloud. I was thinking for users to test out the app on new poses, we can implement something like https://blog.tensorflow.org/2019/12/example-on-device-model-personalization.html for on-device training and inference.

[sayakpaul]

@anilmaddala so the idea here is after collecting the poses, we will build a shallow model, convert it to TFLite and deploy that TFLite model to the application to perform on-device inference.

Now coming to personalized training (as described in the blog post you mentioned), in order for that to work we still need a master model first which we could transfer learn for a particular user. Our initial shallow model can act as a master model here. Tagging @khanhlvg for any inputs he might have.

Let me know if there's any doubt in that regard.

[SunAriesCN]

Well, it this google-research/poem may help for it.