Awesome GitHub RepositoriesReal-Time Speech Processing

NeMo is a multimodal AI framework and toolkit designed for the development, training, and scaling of large language models, generative AI systems, and speech-based models. It functions as an automatic speech recognition toolkit, a text-to-speech engine, and a framework for building models that process and generate combinations of text, image, and audio data. The project serves as a conversational AI orchestrator capable of managing real-time, interruptible voice interactions. It provides specialized workflows for speech translation, converting spoken audio from one language into text or speec

Vosk is an offline speech-to-text engine and API that converts spoken audio into text locally on a device. It provides a cross-platform speech toolkit with language bindings for integrating voice recognition into server environments, Android, iOS, and Raspberry Pi. The project includes a speaker identification tool to distinguish between different voices and an acoustic model trainer for building custom neural network models. These training tools enable speech feature extraction and model accuracy evaluation to improve recognition for specialized domains. The system supports real-time audio

Omi is an open-source wearable AI platform that captures audio and screen data to provide real-time conversational assistance and memory. It integrates a wearable hardware development kit with a vector memory database and large language model capabilities to create a persistent digital record of user interactions. The platform is distinguished by its BLE audio streaming pipeline, which transmits raw audio from wearable hardware for real-time transcription and speaker identification. It utilizes a plugin-based agent tool framework that allows AI assistants to autonomously invoke custom functio

This project is a multimodal translation framework and large language model capable of speech-to-speech, speech-to-text, and text-to-text translation across nearly 100 languages. It provides a real-time speech translation engine and a comprehensive toolkit for converting spoken audio between languages. The system is distinguished by its ability to preserve the original speaker's tone, pace, and prosody during translation. It utilizes a specialized on-device inference toolkit that converts model checkpoints into C-based libraries, enabling low-latency execution on mobile and edge hardware with

Ten Framework is a multimodal large language model agent framework designed for building low-latency conversational agents. It integrates voice, text, and visual inputs in real time to facilitate human interaction. The project includes a real-time speech processing pipeline for streaming transcription, voice activity detection, and speaker diarization. It also features an avatar synchronization engine that coordinates character lip animations and visual outputs with synthesized speech. The framework covers edge AI deployment through containerized packaging and direct integration with embedde

WhisperLiveKit is a real-time speech-to-text server that transcribes streaming audio into text with ultra-low latency using Whisper models. It serves transcription capabilities through REST endpoints and WebSocket connections, enabling external applications to send audio and receive transcriptions as words are spoken, making it suitable for live captioning or voice interfaces. The project distinguishes itself by combining real-time transcription with speaker diarization, assigning transcribed words to individual speakers during live audio streams for meeting or interview transcripts. It also

RealtimeSTT is a local speech-to-text engine and real-time automatic speech recognition server. It utilizes transformer-based recognition and omnilingual pipelines to convert live audio streams into text, providing a WebSocket-based streaming API for raw PCM audio transmission. The project is distinguished by a dual-backend transcription pipeline that uses a lightweight engine for immediate partial suggestions and a heavier model for final high-accuracy results. It includes a wake word detection system to trigger recording and employs a shared-resource inference model to distribute heavy spee

jetson-inference is a set of libraries and tools for executing optimized deep learning models on embedded GPU hardware. Its primary purpose is to enable real-time computer vision and AI inference at the edge with low latency and high throughput. The project distinguishes itself through high-performance streaming analytics and the ability to execute concurrent AI pipelines on auto-grade silicon. It provides specialized support for multi-sensor stream processing, utilizing zero-copy data transport to load camera frames directly into GPU memory. The codebase covers a broad surface of capabiliti

Moonshine is a complete on-device voice interface toolkit that provides speech recognition, text-to-speech synthesis, phonetic processing, speaker diarization, and intent recognition, all running locally on edge hardware without any cloud dependency. It executes quantized neural networks for speech and language tasks directly on the device, enabling fully offline conversational AI capabilities. The toolkit distinguishes itself by orchestrating multi-turn spoken exchanges through a conversational flow manager that maintains context across interactions and manages branching dialog flows. It inc

This project is a Chinese automatic speech recognition framework and deep learning system designed to convert spoken Chinese audio into written text. It functions as a toolkit for training, evaluating, and deploying speech-to-text models, utilizing a specialized pinyin-to-text converter that transforms phonetic sequences into Chinese characters using a probability graph model. The system is distinguished by its deployment flexibility, offering a dockerized recognition server that provides transcription capabilities as a remote API. It supports high-performance streaming through a gRPC speech-

Silero VAD is a voice activity detection model and deep learning speech classifier designed to distinguish human speech from silence across diverse languages and noisy environments. It functions as a pre-trained neural network capable of identifying speech segments within both static audio recordings and real-time data streams. The project includes a language identification tool for classifying spoken languages and a framework for fine-tuning audio models. It provides utilities for optimizing detection thresholds using validation datasets and retraining the model with custom labeled audio to

MeloTTS is an open-source text-to-speech library that generates natural-sounding speech across six languages, with the ability to mix two languages within a single utterance. Its architecture combines a token-based text frontend with a language-agnostic acoustic model, enabling it to handle bilingual code-switching and produce streaming audio output in real time. The system is designed to run efficiently on standard CPU hardware without requiring a dedicated GPU, using a lightweight neural network for real-time inference. It supports English, Spanish, French, Chinese, Japanese, and Korean, an

This project is a framework for building local voice assistants and a real-time audio streaming server. It functions as a containerized inference engine and a multilingual speech pipeline that orchestrates speech-to-text, language models, and text-to-speech components to convert spoken input into spoken output. The system is distinguished by its use of WebSocket-based bidirectional streaming for low-latency interactions. It features a voice activity detection system that manages speech boundaries and handles user barge-in interruptions during assistant playback. It also supports custom voice

Kimi-Audio is a large language model audio foundation model designed to understand audio input and generate high-fidelity speech responses in real time. It functions as a unified system encompassing a text-to-speech synthesis engine and a speech-to-text transcription tool. The project enables real-time audio conversations through a multi-modal conversation loop and chunk-wise streaming detokenization to reduce playback latency. It provides controls over speech speed, accent, and emotional tone during conversational audio generation. The system covers audio intelligence capabilities, includin

Cheetah is an LLM technical interview assistant composed of a native macOS application and a browser extension. It provides real-time coding and answering suggestions during technical interviews by combining live audio transcription with web-based context extraction. The system functions as a real-time interview coach that converts spoken questions into text using on-device speech-to-text processing. It uses a browser-integrated DOM scraper to extract live code and console logs, allowing the AI to analyze the current coding state and generate technical solutions based on the specific environm

PocketSphinx is an offline speech recognition engine that converts raw audio from files or live microphone streams into written text without requiring a network connection. It functions as a speech-to-text library, a real-time transcription engine, and a voice command processor, capable of detecting and transcribing spoken commands from continuous audio streams with configurable acoustic and language models. The engine uses weighted finite-state transducers to represent acoustic, phonetic, and language models as a single search graph for efficient decoding. It employs fixed-point acoustic mod

Qwen2.5-Omni هو نموذج لغوي ضخم متعدد الوسائط وشامل مصمم لمعالجة وتوليد المحتوى عبر النصوص والصوت والرؤية والفيديو. يعمل كذكاء اصطناعي صوتي في الوقت الفعلي، مستخدماً بنية شاملة (End-to-end) للحفاظ على محادثات صوتية متزامنة مع استجابات ذات زمن انتقال منخفض. يركز المشروع على الكفاءة من خلال نماذج الحافة المكممة (Quantized edge models)، مما يسمح بالاستدلال المحلي على أجهزة الجوال والأجهزة ذات الموارد المحدودة. ويستخدم تكميم الأوزان بـ 4 بت، وتفريغ العمليات على وحدة المعالجة المركزية (CPU)، وتحميل الأوزان عند الطلب لتقليل متطلبات ذاكرة GPU. يدمج النظام مشفرات متخصصة لتحليل تدفقات البيانات متعددة الوسائط ويتميز بفك تشفير متدفق لتوليد الكلام في الوقت الفعلي. كما يتضمن إمكانيات لتخصيص صوت الكلام لتعديل الخصائص النغمية والجنسية للمخرجات الصوتية.

RealtimeTTS هو محرك تحويل النص إلى كلام (TTS) في الوقت الفعلي ومعالج تدفق مصمم لتحويل تدفقات النص أو الرموز إلى تشغيل صوتي بأقل زمن انتقال. يوفر واجهة برمجية لإدارة تدفقات الصوت، وتقدم التوليف، وتكامل محركات الكلام المحلية أو السحابية. يتضمن النظام أداة لاستنساخ الصوت العصبي تولد كلاماً اصطناعياً عن طريق استخراج الميزات الصوتية من عينات صوتية مرجعية. ويستخدم تجريداً قائماً على الموفر لتوجيه طلبات التوليف عبر نماذج عصبية مختلفة وواجهات برمجة تطبيقات سحابية. يغطي المشروع مجموعة من المجالات الوظيفية بما في ذلك التحكم في التشغيل غير المتزامن، وتوليف الصوت من تدفق الرموز، وسير عمل مزامنة الكلام التي تطلق استدعاءات (callbacks) على مستوى الكلمة أو الحرف. كما يدعم تصدير الصوت المُولّد مباشرة إلى ملفات WAV للتخزين الدائم.

RealtimeVoiceChat is a conversational AI gateway and voice chat interface that enables real-time spoken interactions with large language models. It functions as a streaming audio pipeline that integrates speech-to-text transcription and text-to-speech synthesis to facilitate natural voice conversations. The system utilizes WebSocket-based audio streaming and an asynchronous text-to-speech pipeline to maintain low latency. It includes specialized conversational controls such as silence-based turn detection and client-side interrupt handling, which stops active audio playback when new user spee