Camera-based object tracking for VR experiences

Project Objective: Tracking a ball in 3D physical space and reproducing it within a VR headset, such that ball games can be played.

Immersivity of Virtual Reality experiences is often enhanced when users are able to interact with objects within a virtual environment.

Typically the mode of interaction restricts users to handling non-existent objects, creating a sense of incompleteness whereby the user's tactile sense is not stimulated.

This project introduces a complete system, such that a physical object is tracked using a webcam and reproduced in the same precise location within a virtual environment, allowing users to manipulate the object whilst in virtual reality.

The system I have created consists of two components that communicate via UDP:

• A python 2.7.16 application that utilises OpenCV to handle webcam data and produce an output of positional information of an untethered object (a ball!)
• A C# virtual reality application that takes the positional data of said object and renders it into the same position relative to the user, such that it can be interacted with (Unity3D implementation)

Demo Videos

Handling the ball 🎾

Bowling experience 🎳

Created by:

George Lowe

Contact me here:

This project was initially produced as part of my Dissertation project whilst undertaking my BSc in Computer Science.