This project is a multimodal translation framework and large language model capable of speech-to-speech, speech-to-text, and text-to-text translation across nearly 100 languages. It provides a real-time speech translation engine and a comprehensive toolkit for converting spoken audio between languages.
The system is distinguished by its ability to preserve the original speaker's tone, pace, and prosody during translation. It utilizes a specialized on-device inference toolkit that converts model checkpoints into C-based libraries, enabling low-latency execution on mobile and edge hardware without a Python runtime.
The framework covers a wide range of capabilities including automatic speech recognition, expressive speech synthesis, and real-time translation streaming. It also includes audio content moderation for toxicity detection and tools for multimodal translation evaluation and distributed model fine-tuning.
The project is implemented using Jupyter Notebooks.
