NCams

NCams is a toolbox to use multiple cameras to track and reconstruct the kinematics of primate limbs. NCams leverages state-of-the-art machine learning approaches for image tracking (DeepLabCut) and musculoskeletal modeling (OpenSIM) and includes integration and processing software specifically developed for primate limb tracking.

NCams is installed as a Python module with several submodules that include camera calibration, estimation of relative camera positions, triangulation of the marker information from multiple cameras. The module runs on Windows or *nix.

Features

1. Intrinsic camera calibration - any number of cameras can be quickly calibrated with support for checkerboard or charucoboards.

2. Extrinsic calibration/camera pose estimation - multiple methods for calculating the camera extrinsics are available (one-shot, stereo-sequential, and common-point).

3. Multiple triangulation/3D reconstruction methods inculding processing/filtering.
4. Inverse kinematics...

Getting Started

This project's code is available on GitHub. The example raw data is available on Dropbox and raw images on Box.

Prerequisites

Software:

• Python 3+/Anaconda
• DeepLabCut - if using it for labeling markers. See below for installation recommendations.
  • NVIDIA drivers
  • CUDA 10.0
• Spinnaker Python module if using FLIR cameras for capture.
• OpenSim 4.0 to calculate inverse kinematics.
• Module dependencies are listed in the setup file.

If you have problems with installations, check out our installation tips.

There are no hardware prerequisites for the core functionality of the toolbox. If using DeepLabCut or the spinnaker_tools, however, then an NVIDIA videocard that supports CUDA or FLIR cameras are required respectively.

Installation

1. Download the repository or clone it using git: git clone https://github.com/CMGreenspon/NCams.git.
2. Open Terminal, Command Line, or the desired Anaconda environment (e.g. the one with DeepLabCut installed) in the project folder.
3. Run python setup.py install.

Example calibration, pose estimation, and triangulation

Intrinsic and Extrinsic Calibration contains example code for setting up creating an NCams_Config, creating a compatible Charucoboard, calibrating camera intrinsics individually or in bulk, and using either one-shot or stereo-sequential extrinsic calibration. Example data for calibration may be found at Box.

Triangulation and Plotting Example walks through triangulating markers based on multiple CSVs (generated by DLC, though conversions can be made), smoothing & filtering in both 2D and 3D, interactive 3D plots of markers over time, and exporting videos with or without skeletal frames.

Labeling and 3D marker reconstruction

The analysis goes over marking images with DeepLabCut, training a network, and triangulation of the marker data.

Analysis of multiple sessions follows the analysis example, but is specifically designed to handle multiple sessions of recordings from the same cameras.

Tips have suggestions on NCams/DeepLabCut use that can be useful.

Inverse kinematics

The guide describes the necessary tools and steps to obtain joint angles of a skeletal model that follow the measured markers using OpenSim (SimTK). We use a [publically available skeletal model](??? not yet) of the human right arm and hand that we modified and markered to fit our purposes.

Structure of the repository

• ncams -- imported module
• Examples -- examples and guides on using NCams
• documentation -- various documentation on NCams
• opensim_models -- OpenSim skeletal models that we use for inverse kinematics
• dlc_markers -- images with location of markers relative to bone segments and a DLC config with marker names and skeleton
• images -- demonstrational images and gifs

Authors

• Charles M. Greenspon
• Anton Sobinov
• Developed in Bensmaia Lab.

Keywords

multi-camera calibration, triangulation, inverse kinematics, 3d reconstruction