Hackathon: Live audio stream translation

Real time voice translation utilizing OpenAI's Whisper for transcript and ChatCompletion for translation.

• Demo Video

Features

• From a dropdown, select your spoken language and the language you want to translate to
• Speech to text transcription & Translation - Once you begin recording, Whisper and GPT3.5 will attempt to translate each word you speak in real-time
• Summarize - Once you are done recording, stop the recording and you will be able to summarize the transcription
• Voice - buggy. Once you are done speaking, stopping the recording session will give you an option for Voice. Which will read the translated text out via the TTS feature

AI Models

• Whisper Open Source Model
• GPT3.5-Turbo through Azure
• ElevenLabs - VoiceAI

Running locally

Setup Backend

Requirements:

• Python version 3.9+ - Download
• FFmpeg
  • Mac: brew install ffmpeg
  • Windows: follow this README

Setup:

• Follow the Backend README for further steps

Run the websocket server:

• run cd backend
• run python3 main.py

Setup Frontend

Requirements:

• Yarn
• Node (v16+)

Setup:

• run cd frontend
• run yarn install

Run the WebApp:

• run yarn start

BUGS

• App is very much unstable and not consistent.
• Occasionally will skip words or phrases due to race condition
• Timeout issue with Azure GPT3.5 API calls sometimes

INFO - error_code=Timeout error_message='The operation was timeout.' error_param=None error_type=None message='OpenAI API error received' stream_error=False
Error during translate:  The operation was timeout. { "error": { "code": "Timeout", "message": "The operation was timeout." } } 408 {'error': {'code': 'Timeout', 'message': 'The operation was timeout.'}} {'Content-Length': '75', 'Content-Type': 'application/json', 'apim-request-id': 'a0cecbef-1a20-406e-850a-138d89755b6c', 'Strict-Transport-Security': 'max-age=31536000; includeSubDomains; preload', 'x-content-type-options': 'nosniff', 'x-ms-region': 'South Central US', 'Date': 'Tue, 28 Mar 2023 19:15:28 GMT'}

Reference

• Our Design board
• DeepL Translator - Helpful during our testing process to measure accuracy of translated text