👋 Sign Translate

Sign Translate is a web-based application for real-time multilingual sign language translation.
Built for desktop and mobile, based on state-of-the-art client side models.

sign.mt

Contributing Guidelines · Submit an Issue

Key Features

(Hyperlinks to issues indicate the feature does not exist fully)

Spoken-to-signed language translation

┌─────────────────────┐
│Spoken Language Audio│                                        ┌─────────┐
└─────────┬───────────┘                            ┌──────────►│Human GAN│
          │                                        │           └─────────┘
          ▼                                        │
┌────────────────────┐     ┌───────────┐    ┌──────┴──────┐    ┌───────────────┐
│Spoken Language Text├────►│SignWriting├───►│Pose Sequence├───►│Skeleton Viewer│
└─────────┬──────────┘     └───────────┘    └──────┬──────┘    └───────────────┘
          │                      ▲                 │
          ▼                      │                 │           ┌────────────────┐
┌───────────────────────┐        │                 └──────────►│Avatar Animation│
│Language Identification├────────┘                             └────────────────┘
└───────────────────────┘

• Text / Microphone inputs
• Text-to-speech
• Spoken Language identification (Detect Language)
• Spoken language text to SignWriting translation
• SignWriting to pose sequence
• Text to pose sequence fallback (server side)
• Skeleton / Avatar / Human Pose Viewers
• Copy / share / download video

Signed-to-spoken language translation

┌──────────────────────────┐                                ┌────────────────────┐
│Upload Sign Language Video│                      ┌────────►│Spoken Language Text│
└──────────┬───────────────┘                      │         └──────────┬─────────┘
           │                                      │                    │
           │          ┌────────────┐       ┌──────┴────┐               │
           ├─────────►│Segmentation├──────►│SignWriting│               │
           │          └────────────┘       └───────────┘               │
           │                                                           ▼
┌──────────┴────────────────┐                               ┌─────────────────────┐
│Camera Sign Language Video │                               │Spoken Language Audio│
└───────────────────────────┘                               └─────────────────────┘

• Camera / File upload video inputs
• SignWriting hand shape and orientation estimation
• SignWriting facial features estimation
• Signed Language identification (Detect Language)
• Segmentation - TODO CREATE ISSUE
• Tokenization - TODO CREATE ISSUE
• SignWriting to spoken language translation
• Text-to-speech
• Copy / share translation

Development Setup

Prerequisites

• Install Node.js which includes Node Package Manager

Setting Up the Project

Install dependencies locally:

npm install

Run the application:

npm start

Test the application:

npm test

Want to Help?

Want to report a bug, contribute some code, or improve documentation? Excellent! Read up on our guidelines for contributing and then check out one of our issues labeled as help wanted or good first issue.

Find this useful? Give our repo a star :star: :arrow_up:.