🐕 dog-pose-estimation

As a fun weekend project, I developed a pose estimation model with my dog using DeepLabCut. DLC is a free and open source toolkit that enables researchers to estimate the pose of a subject. DLC leverages transfer learning, which is the ability to take a fully trained neural network model — which was trained with a large supervised data set — and re-train it with a smaller set of images.

Using the DLC Python toolkit, I was able to collect and label ~100 images of my dog Coco to re-train an accurate pose estimation model without needing a GPU or extensive programming experience. The toolkit made it easy to extract videos taken on my iPhone, annotate frames, train a model with TensorFlow, analyze and refine the model, and generate labeled videos that track Coco's pose with markers and show an outline of her skeleton.

📖 Read: Nath*, Mathis* et al., 2019
📺 Watch: Mackenzie Mathis: DeepLabCut Tech Demo
💻 Try DLC: https://github.com/DeepLabCut/DeepLabCut

📝 Follow all of the steps below inside a Jupyter Notebook

🎬 Step 1 - Collect Video

The first step is to collect videos of your dog. I took both indoor and ourdoor videos of Coco from many different angles with my iPhone X. Because I eventually want to detect her position based on pose (standing, sitting and down-stay), I took videos of each position.

All of the videos used for this experiment are available to download on Google Drive.

Tip: Cameras should be placed such that the features you want to track are visible.

💻 Step 2 - Install DeepLabCut

For this experiment, I used my Ubuntu 20.04 desktop computer with an Intel(R) Core(TM) i7-10700K CPU @ 3.80GHz and 128GB of RAM (🚫 No GPU required 🚫). Only ~30GB of RAM was used during the training, so 128GB is overkill.

To set up my environment, I used the conda file for CPU provided by DeepLabCut to create an environment called DLC-CPU. This environment will use python=3.7 tensorflow=1.13.1 and wxPython<4.1.0. There is also a conda file for GPU, which uses tensorflow-gpu=1.13.1. See more detailed instructions in the DLC docs

1. Install Anaconda 3 if you don't have it already
2. Download DLC's conda file for CPU
3. Open terminal where you placed the file and run conda env create -f DLC-CPU.yaml

✏️ Step 3 - Create and Configure Project

Create the Project

deeplabcut.create_new_project('ProjectName','YourName', ['/Users/FullPath/OfVideo1.mov', '/Users/FullPath/OfVideo2.mov', '/Users/FullPath/OfVideo1.mov'],
              copy_videos=True, multianimal=False)

Define Body Parts

Edit the config.yaml to add body parts. These are the parts that I used for my dog:

bodyparts:
- nose
- leftear
- rightear
- collar
- spine1
- spine2
- spine3
- tailstart
- tailend
- frontleftleg
- frontleftpaw
- frontrightleg
- frontrightpaw
- backleftleg
- backleftpaw
- backrightleg
- backrightpaw

Set config_path

For the next steps, you will use the environment variable config_path.

config_path = '/Users/FullPath/DeepLabCut/yourprojectname/config.yaml'

🖌️ Step 4 - Label Video Frames

Select Frames to Label:

deeplabcut.extract_frames(config_path, mode='automatic', algo='kmeans', crop=False)

Label Frames

Use the DLC Labeling Toolbox to label frames. DLC will automatically select frames from the video(s) you import using kmeans quantization. It will also downsample the frames before annotation.

Build Skeleton

📁 Step 5 - Create Training Dataset

deeplabcut.create_training_dataset(config_path, augmenter_type='imgaug')

⚙️ Step 6 - Train the Model

deeplabcut.train_network(config_path, shuffle=1, trainingsetindex=0, gputouse=None, max_snapshots_to_keep=5, autotune=False, displayiters=100, saveiters=15000, maxiters=30000, allow_growth=True)

📊 Step 7 - Analyze Video with the Model

deeplabcut.analyze_videos(path_config_file,videofile_path, videotype='mov')

📽️ Step 8 - Create Labeled Videos

deeplabcut.create_labeled_video(path_config_file, videofile_path, videotype='mov', draw_skeleton=True)